experiment with hypothesis normaliser via HPI extraction and the corresponding 'cleanup' normaliser

[karlicoss]

No description provided.

[karlicoss]

@seanbreckenridge tried a quick hacky objects normaliser for hypothesis -- worked great! I guess exposing a method to parse files like in your modules is cleaner, but kinda nice config hacking works as well.

It was suggesting to prune some extra files that weren't pruned before, so I realized currently I am pruning hypothesis via json module, so helped me identify some flaky fields and implement a 'cleanup' normaliser.

(probably won't merge like this, will play a bit more first with other HPI modules)

[karlicoss]

Also forcing a module to use just one input file is kinda relevant to this karlicoss/HPI#318 (comment)

[purarue]

Ah yeah, I tend to circumvent this whole problem of dealing with the config or having to specify the one file youre interested in parsing by using non-cached functions from the ..._export library or the _parse_ function in the HPI code, not actually using the public-facing HPI functions. As examples:

https://github.com/seanbreckenridge/bleanser/blob/c96195b7b32769db5f7f351224a77fac217da4c6/src/bleanser_sean/modules/activitywatch.py#L4

https://github.com/seanbreckenridge/bleanser/blob/c96195b7b32769db5f7f351224a77fac217da4c6/src/bleanser_sean/modules/discord.py#L27

https://github.com/seanbreckenridge/bleanser/blob/c96195b7b32769db5f7f351224a77fac217da4c6/src/bleanser_sean/modules/zsh.py#L4

[purarue]

This does however mean I have to specify my files in two places, I have them hardcoded in this helper script

I havent found that too annoying to maintain though, and its not always the case that the files I'm parsing for HPI are the same files I want to be running bleanser on (for example I have some partial exports that get combined as a sort of incremental export)

[karlicoss]

Yeah, generally it's nicer to expose anyway, although in some cases DAL takes multiple inputs and does data merging itself, e.g. here https://github.com/karlicoss/rexport/blob/61eb8d219064c9f80dfb92a756b6323276314460/src/rexport/dal.py#L152-L160

Also would be kinda cool if in the future bleanser modules could be almost completely agnostic of the module implementation, e.g. using magic similar to inferring module stats to detect data providers https://github.com/karlicoss/HPI/blob/fe26efaea849e9c2b0fb57a4cc75878a45c3f8bf/my/core/stats.py#L16-L23

[karlicoss]

Wrote another module for comparing twitter databases from Android app -- proves super useful since the db has so much garbage #39
I'll start merging them into hpi/ subdirectory just so code isn't lost and runs the CI, later can think of something better as we shape them