Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slim derivatives #44

Open
arokem opened this issue Dec 20, 2024 · 0 comments
Open

Slim derivatives #44

arokem opened this issue Dec 20, 2024 · 0 comments

Comments

@arokem
Copy link
Contributor

arokem commented Dec 20, 2024

For some large-scale datasets (and just in general?) it may be good to think about ways to slim down the derivatives generated by pyAFQ. For example, we might want a minimal set of derivatives to be stored (e.g., export("just_profiles"), which only stores the final tract profiles csv and dumps everything en-route, especially things imported from other pipelines (to avoid duplications).

Another thing to consider is whether compressing some derivatives may be a good approach. For example, the html visualization can be compressed by a substantial amount (in a quick experiment from 29.3MB to 9.4MB in one case), so maybe zipping it up as a first pass may make sense under some conditions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant