Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace stacking gradient search with resample_blocks variant #626

Merged
merged 6 commits into from
Oct 24, 2024

Conversation

mraspaud
Copy link
Member

@mraspaud mraspaud commented Oct 22, 2024

This PR replaces the stacking gradient search with resample blocks for the swath to area case.
It also adds support for area to swath in the gradient search.

Copy link

codecov bot commented Oct 22, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.96%. Comparing base (c4ce8ee) to head (4dd2948).
Report is 12 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #626      +/-   ##
==========================================
- Coverage   93.97%   93.96%   -0.01%     
==========================================
  Files          86       86              
  Lines       13848    13538     -310     
==========================================
- Hits        13013    12721     -292     
+ Misses        835      817      -18     
Flag Coverage Δ
unittests 93.96% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mraspaud mraspaud marked this pull request as ready for review October 22, 2024 09:43
@mraspaud mraspaud requested a review from pnuu October 22, 2024 09:43
@coveralls
Copy link

coveralls commented Oct 22, 2024

Coverage Status

coverage: 93.66% (-0.03%) from 93.686%
when pulling 4dd2948 on mraspaud:feature-remove-stacking-gradient
into 8da073e on pytroll:main.

@mraspaud mraspaud self-assigned this Oct 22, 2024
Copy link
Member

@pnuu pnuu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of initial comments. I'll do some real-life testing next.

I'm surprised you could get rid of such a huge chunk of code!

pyresample/gradient/__init__.py Outdated Show resolved Hide resolved
pyresample/gradient/__init__.py Outdated Show resolved Hide resolved
@mraspaud
Copy link
Member Author

For reference, main seems to have striping problems:
overview_20190405_081003_main
This branch doesn't:
overview_20190405_081003

@mraspaud
Copy link
Member Author

Also, preliminary test, resampling that metop scene to euron1 takes 5GB on main vs 3GB with this PR

@pnuu
Copy link
Member

pnuu commented Oct 24, 2024

In my tests the memory usage stayed the same, but the PR is much faster, processing time went from 57.6 s to 49.1 s.

main
Screenshot 2024-10-24 at 12-18-29 Bokeh Plot

@pnuu
Copy link
Member

pnuu commented Oct 24, 2024

Third try, maybe now the correct graph...

PR
Screenshot 2024-10-24 at 12-18-33 Bokeh Plot

Copy link
Member

@pnuu pnuu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Thanks for a) fixing #620 and b) getting rid of huge amounts of code!

@mraspaud mraspaud merged commit 69c3a65 into pytroll:main Oct 24, 2024
24 of 26 checks passed
@mraspaud mraspaud deleted the feature-remove-stacking-gradient branch October 24, 2024 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Gradient search resampling swath data gives transposed results
3 participants