Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduced percentile normalization for synthesis challenge metrics #700

Merged
merged 6 commits into from
Aug 2, 2023

Conversation

FelixSteinbauer
Copy link
Contributor

Proposed Changes

  • Added __percentile_clip function that normalizes input tensors based on percentiles and clips values that are blow/above. This is relevant as we notices that the image data we are working with has few but very strong outliers. These outliers are removed and the metric computation (especially metrics like PSNR which depend on the value range) gets more robust/reliable.
  • Also, as we now know for sure that all values will be between 0 and 1 we can fix the value range for PSNR to 1.0 (as soon as the other pull request which introduces the data_range parameter is approved. This is why there is a "#TODO" comment in the code)

Note: I did test the __percentile_clip() function.

Checklist

  • I have read the CONTRIBUTING guide.
  • My PR is based from the current GaNDLF master .
  • Non-breaking change (does not break existing functionality): provide as many details as possible for any breaking change.
  • Function/class source code documentation added/updated.
  • Code has been blacked for style consistency.
  • If applicable, version information has been updated in GANDLF/version.py.
  • If adding a git submodule, add to list of exceptions for black styling in pyproject.toml file.
  • Usage documentation has been updated, if appropriate.
  • Tests added or modified to cover the changes; if coverage is reduced, please give explanation.
  • If customized dependency installation is required (i.e., a separate pip install step is needed for PR to be functional), please ensure it is reflected in all the files that control the CI, namely: python-test.yml, and all docker files [1,2,3].

@github-actions
Copy link
Contributor

github-actions bot commented Jul 29, 2023

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@codecov
Copy link

codecov bot commented Jul 29, 2023

Codecov Report

Merging #700 (28fc7ea) into master (44e6748) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

❗ Current head 28fc7ea differs from pull request most recent head 3c31ab8. Consider uploading reports for the commit 3c31ab8 to get more accurate results

@@            Coverage Diff             @@
##           master     #700      +/-   ##
==========================================
- Coverage   94.70%   94.69%   -0.01%     
==========================================
  Files         117      117              
  Lines        8208     8207       -1     
==========================================
- Hits         7773     7772       -1     
  Misses        435      435              
Flag Coverage Δ
unittests 94.69% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
GANDLF/cli/generate_metrics.py 100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Collaborator

@sarthakpati sarthakpati left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor changes to increase code coverage. Looks good otherwise. 👍🏽

GANDLF/cli/generate_metrics.py Outdated Show resolved Hide resolved
GANDLF/cli/generate_metrics.py Outdated Show resolved Hide resolved
@FelixSteinbauer
Copy link
Contributor Author

@sarthakpati This PR seems to have the same issue with the testing pipeline I reported for our other PR

@sarthakpati
Copy link
Collaborator

Apologies, but due to an unforeseen issue, OpenFL tests are failing [ref]. Please hang on tight while we are coordinating a fix, and we will re-initiate the failing run automatically.

sarthakpati
sarthakpati previously approved these changes Aug 1, 2023
Copy link
Collaborator

@sarthakpati sarthakpati left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Collaborator

@sarthakpati sarthakpati left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Edit - mistakenly requested changes.

@sarthakpati sarthakpati self-requested a review August 2, 2023 12:42
Copy link
Collaborator

@sarthakpati sarthakpati left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sarthakpati sarthakpati merged commit 329378b into mlcommons:master Aug 2, 2023
14 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Aug 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants