CCC: Continuously Changing Corruptions

[oripress]

Task Description

CCC is a dataset used to benchmark models over time. The dataset is based on a generalization of common corruptions used in ImageNet-C, contains 462 million corrupted images in its development set, and pre-computed evaluation sets for evaluating continual adaptation algorithms over the course of one to ten million images to adapt to.

Dataset Creation

Dataset creation is done by computing noise trajectories that maintain a constant baseline accuracy.

Expected Insights/Relevance

Using our dataset, we can rigorously test classification methods that continuously adapt to changing environments.

License

MIT LIcense

[stes]

FYI, I updated the PR to use the shift happens PR template.

[stes]

Hi @oripress, thanks a lot for your contribution. We are working towards the first release of the benchmark and for this, released our contribution guide with the aim of unifying the implementation of all contributions.

Please have a look, the required changes should be relatively small and are mostly concerned with placing the files within the package, and adding documentation.

From reviewing your PR, the main required changes are:

• Place the files into their own package inside the tasks/ folder
• Add a README and optionally citation info. You can re-use the content from your PR description.
• We are also looking into the best ways to perform unit or integration tests and would be happy to get some feedback on how this would be most effective for your benchmark. Would it be possible to add a "dry-run/demo" option to the benchmark, and e.g. run it only on a small subset of the data? No immediate action/update is required right now, we are first gathering opinions from all authors on this.

Thanks a lot for your contribution, and please ping me if you have any questions. If you have technical questions, also feel free to ping @kotekjedi who is happy to help with implementation details.

[stes]

@zimmerrol do you have ideas how we want to handle requirements for subpackages?

[zimmerrol]

I propose to add them to the requirements files and add comments in that file indicating which tasks need these requirements.

[stes]

Hi!

This is an automated report for the documentation coverage of your PR. To provide good user experience for the benchmark, we aim to cover most functions with doc strings, using the Google docstring format.

We would greatly appreciate your help in assuring that docstring coverage is sufficiently high for your contribution.

The current coverage is:

Name	Total	Miss	Cover	Cover%
models/torchvision.py	4	0	4	100%
tasks/ccc.py	3	2	1	33%
tasks/ccc_imagenet_c.py	26	24	2	8%
tasks/ccc_utils.py	16	15	1	6%
------------------------------	-----------	----------	-----------	------------
TOTAL	49	41	8	16.3%
RESULT: FAILED (minimum: 80.0%, actual: 16.3%)

You can reproduce this result by running

interrogate -v --ignore-nested-functions --ignore-private --ignore-semiprivate --ignore-init-method

on the files you added to your PR.

Outdated.