Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

updates for compatibility with input zstacks #48

Merged
merged 27 commits into from
Oct 8, 2024
Merged

updates for compatibility with input zstacks #48

merged 27 commits into from
Oct 8, 2024

Conversation

akhanf
Copy link
Member

@akhanf akhanf commented Sep 12, 2024 •
edited
Loading

Blaze microscope now outputs data with one tif file per zstack. This adds compatibility for it.

  • Includes a number of other fixes and cleaning up too (removes unused rules/scripts). Tested locally and on gcs.
  • Some refactoring of global config params for resources (adds total_cores, total_mem_mb)
  • currently disables qc reports (was taking long for gcs runs - need to revisit that in another PR)

@akhanf
swap uncorr and corr in flatfield corr
5928f7c
@akhanf
remove fiji from workflow
1ff3be8
@akhanf
updates for compatibility with input zstacks
65adb78 
Blaze microscope now outputs data with one tif file per zstack.
This adds compatibility for it.

TODO: test full end-to-end
@akhanf
use pyvips for tiff reading on zstacks
1ebbb74 
todo: do the same for gcs version
@akhanf
WIP: compatibility for 1x1 blaze acquisitions (1 tile)
78fe03f 
- disable fieldfrac? skip stitching?
akhanf added 22 commits September 29, 2024 16:04
@akhanf
fix for tileconfig key zstack 0->0000
4d1b90e
@akhanf
update the tif_to_zarr_gcs for tifzstacks on google cloud
a2ce218
@akhanf
fix
2ff9fb4
@akhanf
fix
dbb8755
@akhanf
reduce threads for tif_to_zarr_gcs
feb5988
@akhanf
back to reading whole zstack tif at once
05498fe
@akhanf
hopefully final fix for reading tif zstacks on gcs
8739291 
downloads temporary file then uses pyvips to read zstack in a delayed
function -- should avoid memory issues
@akhanf
remove dask sync (was debugging), and set chunks
3b03fbd
@akhanf
had to rechunk, as setting chunk size in from_delayed isn't allowed
6079e6d
@akhanf
constrain number of threads
edea152
@akhanf
use threaded instead
7b91969
@akhanf
skip qc, adjust threads
5fa43fb
@akhanf
update threads
c9c4ba9
@akhanf
reduce stdout
e88ba56
@akhanf
update resources
f1f4ba6 
- cores_per_rule -> total_cores
- added total_mem_mb
- better calculation of resources for tif_to_zarr (mem-based calc of
threads)
- snakefmt
@akhanf
refactoring for zipstore and ome zarr outputs
7c190d8 
- fixes a bug (was writing to zarr to .snakemake_touch)
- hopefully cleans things up and makes more readable
@akhanf
revert chunking for flatfield corr output
52009b9
@akhanf
fix for non-zstacked tif_to_zarr_gcs
5224309 
avoids costly globs and getting metadata from first image
@akhanf
fix
3cafe41
@akhanf
cleanup and fixes
3799107
@akhanf
add dryrun datasets
3dcccd4
@akhanf
cleanout unused lib/dask_image.py
e240483
@akhanf akhanf marked this pull request as ready for review October 8, 2024 12:43
@akhanf akhanf merged commit ee43d44 into main Oct 8, 2024
4 checks passed
@akhanf akhanf deleted the tifzstacks branch October 8, 2024 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@akhanf