Models: fix preprocessing transforms #1166
Conversation
github-actions
bot
added
models
torchgeo/models/resnet.py
Outdated
| 818.86747235, | ||
| ] | ||
| ), | ||
| mean=torch.tensor([590.23569706, 614.21682446, 429.9430203]), |
There was a problem hiding this comment.
We only have weights for RGB images
torchgeo/models/resnet.py
Outdated
| ] | ||
| ), | ||
| mean=torch.tensor([590.23569706, 614.21682446, 429.9430203]), | ||
| std=2 * torch.tensor([675.88746967, 582.87945694, 572.41639287]), |
There was a problem hiding this comment.
SeCo and SSL4EO use 2x the std dev for normalization:
There was a problem hiding this comment.
https://github.com/ServiceNow/seasonal-contrast/blob/8285173ec205b64bc3e53b880344dd6c3f79fa7a/datasets/bigearthnet_dataset.py#L111 Is this also different from the Kornia Normalize method, as they use some min max operation and also clip to range [0, 255]? Or is this something like percentiles used to normalize? Never seen this tbh. I guess my question is does the difference matter?
There was a problem hiding this comment.
True, this is actually quite different. They substract (mean - 2 * std), not mean, and they divide by (4 * std), not std. I have no idea why they scale by 255, seems odd unless you're trying to plot something. And we obviously aren't clamping (again, only useful for plotting).
EDIT: oh, is it because they need to load it as a PIL image? Maybe that removes the 255 when converting to a Tensor.
There was a problem hiding this comment.
Not sure how they're even using PIL here since it only supports RGB, not MSI.
There was a problem hiding this comment.
(I guess that's why SeCo only had RGB weights?)
There was a problem hiding this comment.
Ah yes, they divide by 10K * 255 for S1 but not for S2?
There was a problem hiding this comment.
The more I look at these, the less sure I am that they are correct. May want to reach out to the authors.
There was a problem hiding this comment.
Once we figure this out, we should also fix the SeCO datamodule I recently added. According to @wangyi111, it actually doesn't matter that much what the exact normalization is.
There was a problem hiding this comment.
Opened an issue to get to the bottom of this. I actually think we're using the wrong normalization (this was only used for BigEarthNet, not for any other dataset). My guess is the SeCo normalization, but we'll see if we get a response. If I don't hear anything back, let's just assume SeCo normalization.
adamjstewart
force-pushed
the
fixes/model-transforms
branch
from
858cc65 to
397265a
Compare
* Models: fix preprocessing transforms * Fix normalization of SeCo std dev * black * Fix SeCo transforms * Add comment explaining source of transforms
* Models: fix preprocessing transforms * Fix normalization of SeCo std dev * black * Fix SeCo transforms * Add comment explaining source of transforms
Our pre-trained models come with preprocessing transforms that should be used to normalize and reshape images used with them. However, these transforms don't actually run. This PR fixes them and adds tests to ensure that all transforms actually work.