Pixel file format

[Mittmich]

Implemented pixel file format (completes the following trello card) based on the considerations here. The general ideas have been updated in the datastructures ipython notebook.

[cchlanger]

I will slightly update the structure of the test directory so it is easier to see which part of the code the test corresponds to.

[cchlanger]

Should we use the slightly more elegant way for fixtures described here?
https://stackoverflow.com/questions/46089480/pytest-fixtures-redefining-name-from-outer-scope-pylint
Then we do not have use :
# pylint: disable=redefined-outer-name

[cchlanger]

Looks good from my side, no major changes. Minor refactors and text edits have been performed. The fixture question can be addressed in the next PR on snipping.

[cchlanger]

Looks good from my side, no major changes. Minor refactors and text edits have been performed. The fixture question can be addressed in the next PR on snipping.

[dmitrymyl]

@Mittmich thank you for this PR, that's a lot of work and conceptual development! Generally, I'd approve this PR except for minor code-related issues, but I would like to discuss concepts in depth, especially relating to the organisation of the data folder and column names.

Some of the concepts I'd like to cover:

1. Many things are shared between Pixels and Contacts. Will it make sense to create a parental class with shared methods and then inherit Pixels and Contacts from it?
2. What's the purpose of ContactsParameters and PixelsParameters? They are convenient to store parameters and enforce data schemas (like dataclass), but they look quirky in FileManager.load_contacts -- usually people specify function parameters with kwargs. Seems a bit non-pythonic to me, but that is also not a big deal here.
3. Folder structure: I'd like to have a human-readable names of parquet files. It is good for visual inspection of files, developing alternative libraries in other languages and future-proofing against changes in json.dumps and hash. To reduce the complexity of nested folder structure, I propose that all tables within one folder are either Pixels or Contacts, have the same state of global parameters and differ only in contact orders and compositions of labels. This means that Folder doesn't contain all the data from a single experiment, but rather only one transformed view of it.
4. metadata_combi -- does it contain only sister labels or will it also contain other fragment-level features, like 5mC marks? If not, Shall we distinguish between "molecule" labels (sisterA/sisterB, DNA/RNA, etc) and "fragment" labels (5mC level, open/close chromatin etc.)?

Otherwise, I generally agree with proposed architecture! In my view, it will be great if we discuss the code and our future directions to build a roadmap based on the proposal for the file format and query engine 😉

[dmitrymyl]

I was thinking about why this search is O(N), when it could be O(1). Seems like we cannot hash all PixelParameters. But why? For example, for metadata_combi we could use tuple instead of list, since this collection seems to be immutable by design. Then PixelParameters can be hashed to be dict keys, so _load_metadata will return Dict[PixelParameters: path].

[cchlanger]

We change Metadata file keys are gloal parameters and values are filenames.

[Mittmich]

Json dump can't encode parameters as keys as objects are not allowed as keys. We would need to serialize the object to a string and write a custom implemntation for that. Since this search operation is trivial (length of parameters is likely going to be small), I would vote for leaving it as is.

[dmitrymyl]

Ok, let's leave it like this for now

[cchlanger]

Here is a quick summary of some more points to discuss today:

• Direct access to Contacts and Pixels or use Filemanager only?
• Why hashing?
• When to inherit?

[dmitrymyl]

One thing that I am thinking about after our discussion last week, is related to the flat file structure and global parameters. I would say that the first analytical task that researchers would like to do is to visualize all multi-way contacts (original, not expanded) in some genomic locus. Hence, there should be a way to pull the whole dataset from a Contacts or Pixels container. However, there are no guards implemented or planned that what a users pulls from the container represents the whole dataset. So, who should be responsible for the integrity of the data?

1. The user, who specifies which container file they generate from Fragments, and we provide a function to do that.
2. spoc, and we have to add info about the composition of the original dataset (contact orders and labelling layouts) into the container.

I think it should be the first case, but we have to be transparent about that. Yet it feels like this has to be hardcoded into some specific data representation ("this file comprises the whole original dataset") to guard users from their mistakes. Or has it, really?

I would expect a dichotomy in use cases: it's either a complete dataset of original multi-way contacts in a form of either Contacts or Pixels with original coordinates or different bin sizes, or it's expanded contacts of specified order (either doublets, triplets or quadruplets) with different bin sizes (and maybe original coordinates). To somehow specify that I would originally suggest the following strategy:

1. Hardcode a hierarchical structure for the original dataset of multi-way contacts (can be stored as flat, but virtually pops up as hierarchical).
2. Expanded contacts are stored in individual containers per contact order.

After our discussion, I would suggest another strategy: specifying "presets" of tables in a container. Say, we have a preset for "original contacts", "expanded triplets", "expanded 2-3-4-5-way contacts", etc. Each preset is generated with a specific method in spoc from Fragments and the name of the preset is stored in the file as well. Thus we hardcode some "flavours" of multi-way contact data. This info can be used for verification by user or by downstream programs.

Also, to have distinctions between original and expanded contacts, we should add another global parameter "expanded" with options "no", "combinatorically", "adjacently", "non-adjancently" (or whatever would be neat and grammatically correct).

What do you think about all that?

[Mittmich]

Hey!
I added all the changes that we discussed, with the exception of the switch of metadata keys and values. The reasoning is that json dump can't deal with complex python objects as keys and I would say that the small search speed benefit doesn't justify the implementation of a custom serialiser :)

[dmitrymyl]

Nice, thanks @Mittmich! Everything seems good. With regard to keys — let's leave it like this for now and if we encounter any performance issues, we will change it later.