Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add CLI for benchmarking datasets on SAM models #728

Merged
merged 19 commits into from
Oct 14, 2024
Merged

Add CLI for benchmarking datasets on SAM models #728

merged 19 commits into from
Oct 14, 2024

Conversation

anwai98
Copy link
Contributor

@anwai98 anwai98 commented Oct 9, 2024

The prototype CLI for benchmarking microscopy datasets on SAM models is ready for the first round of review (tested on LIVECell and DeepBacs)

I'll work on testing it on 3d datasets and extend interactive segmentation functionality for 3d (using SAM)

@anwai98 anwai98 requested a review from constantinpape October 9, 2024 08:37
@anwai98 anwai98 mentioned this pull request Oct 9, 2024
Merged
constantinpape
constantinpape reviewed Oct 9, 2024
View reviewed changes
Copy link
Contributor

@constantinpape constantinpape left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall. Two minor points:

  • We can make the data extraction work easily for 2d and 3d, see comments.
  • We should define a csv output that everything is saved to for easy further analysis

micro_sam/evaluation/benchmark_datasets.py Outdated Show resolved Hide resolved
micro_sam/evaluation/benchmark_datasets.py Outdated Show resolved Hide resolved
micro_sam/evaluation/benchmark_datasets.py Outdated Show resolved Hide resolved
micro_sam/evaluation/benchmark_datasets.py Outdated Show resolved Hide resolved
micro_sam/evaluation/benchmark_datasets.py Outdated Show resolved Hide resolved
@anwai98
Copy link
Contributor Author

anwai98 commented Oct 9, 2024 •
edited
Loading

Okay, in principle the pipeline is ready now.

@constantinpape let me know how it looks now.

PS: There are a few major bulks I still need to take care of:

  • Run both 2d and 3d segmentation for volumetric data.
  • Allow user to flexibly choose between AMG / AIS.
  • (Optional) Remove the cached original inputs.
  • (Optional) Remove the cached preprocessed inputs.
  • (Optional) Remove the stored predictions.

@anwai98 anwai98 requested a review from constantinpape October 9, 2024 18:43
@constantinpape
Copy link
Contributor

  • Allow user to flexibly choose between AMG / AIS.

I think the best would be to enable running both AMG and AIS, and by default only run AIS.

constantinpape
constantinpape reviewed Oct 9, 2024
View reviewed changes
micro_sam/evaluation/benchmark_datasets.py Outdated Show resolved Hide resolved
micro_sam/evaluation/benchmark_datasets.py Outdated Show resolved Hide resolved
micro_sam/evaluation/benchmark_datasets.py Outdated Show resolved Hide resolved
micro_sam/evaluation/benchmark_datasets.py Outdated Show resolved Hide resolved
micro_sam/evaluation/benchmark_datasets.py Outdated Show resolved Hide resolved
@anwai98 anwai98 marked this pull request as ready for review October 10, 2024 13:23
@anwai98
Copy link
Contributor Author

anwai98 commented Oct 10, 2024

This PR is now ready (tested this on DeepBacs and Lucchi).

I am now working on finalizing all open-source datasets in the preprocessing part, and with that should be finally good to go from my side.

Resolves #719

@anwai98 anwai98 linked an issue Oct 10, 2024 that may be closed by this pull request
Add out of domain test datasets for benchmarking finetuned models #719
Closed
@anwai98 anwai98 removed a link to an issue Oct 10, 2024
Add out of domain test datasets for benchmarking finetuned models #719
Closed
@anwai98 anwai98 linked an issue Oct 10, 2024 that may be closed by this pull request
Add out of domain test datasets for benchmarking finetuned models #719
Closed
anwai98 and others added 2 commits October 10, 2024 18:04
constantinpape
constantinpape reviewed Oct 10, 2024
View reviewed changes
Copy link
Contributor

@constantinpape constantinpape left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should refactor the loading functionality for automatic segmentation, rest looks good from my end now.

test/test_automatic_segmentation.py Outdated Show resolved Hide resolved
micro_sam/automatic_segmentation.py Show resolved Hide resolved
micro_sam/evaluation/benchmark_datasets.py Show resolved Hide resolved
@anwai98
Copy link
Contributor Author

anwai98 commented Oct 14, 2024

Hi @constantinpape,

This PR is GTG from my side now. (I took care of the minor TODOs which I procrastinated since last week). Thanks!

@constantinpape constantinpape merged commit 766aa9b into dev Oct 14, 2024
6 checks passed
@constantinpape constantinpape deleted the add-benchmark-datasets branch October 14, 2024 17:35
@anwai98 anwai98 mentioned this pull request Oct 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@constantinpape constantinpape constantinpape approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add out of domain test datasets for benchmarking finetuned models
2 participants
@anwai98 @constantinpape