Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EPIC: issues with large number of small cuts #1781

Open
3 of 6 tasks
abuts opened this issue Dec 3, 2024 · 0 comments
Open
3 of 6 tasks

EPIC: issues with large number of small cuts #1781

abuts opened this issue Dec 3, 2024 · 0 comments
Assignees
Labels
bug Something isn't working Epic

Comments

@abuts
Copy link
Member

abuts commented Dec 3, 2024

User was working with big number of small cuts and identified bunch of interconnected issues, which are the essence of this epic. They would probably separate into multiple smaller tickets

  1. Incorrect experiment combine. Some parts will be sorted in bug(s?) in combine #1778 but some are related to 2. here. bug(s?) in combine #1778 assumes that files with the same run_id should contain same IX_experiment but in fact this should be checked and case of different IX_experiments dealt with. At least throw error for the beginning.

  2. slow operations if loading/doing multiple small cuts and moving objects between containers. (hashes are recalculated at loagind and if there are a lot of small cuts loading). Claimed 7-fold deceleration. Solution -- Hashable class -- subclass of serializable. It would also allow fast comparison of hashable IX_experiments. Additional acceleration would be providing using mex-code for binning and transforming pixels into appropriate coordinate system.

  1. constant grows of unique_references container and its design. Design introduces additional implementation of singleton which is non-trivial to understand. Also adds all experiment's references to unique references container which for N small cuts containing M(n) references so N*M element grows for each operations with this bunch. Design and Grows -- separate but related issues.
@abuts abuts added Epic bug Something isn't working labels Dec 3, 2024
@abuts abuts changed the title issue with large number of small cuts issues with large number of small cuts Dec 3, 2024
@abuts abuts changed the title issues with large number of small cuts EPIC: issues with large number of small cuts Dec 11, 2024
@abuts abuts self-assigned this Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Epic
Projects
None yet
Development

No branches or pull requests

1 participant