Question about large dataset

[aryabhata-archive]

Hi. Interesting results with CIFAR-10. However, it is hard to deal with the larger dataset. The object "Causing the issue". I'm unable to use full dataset (74K training, 25K validation). How can we solve this?

[senwu]

Hi @aryabhata-archive,

Thanks for your interest in our work!

Can you provide me more information about your task?

Here are some hints about running your own dataset with our framework:

To train on the larger dataset or other image datasets in general in our framework, you need to do the following things:

1. Create a new dataset for your data to load the dataset here: https://github.com/SenWu/dauphin/tree/master/dauphin/image/datasets
2. Update the data.py to accept your dataset here: https://github.com/SenWu/dauphin/blob/master/dauphin/image/data.py
3. Select your the transforms and policy you want to use here: https://github.com/SenWu/dauphin/blob/master/dauphin/image/augment_policy.py, all available transforms can be found here: https://github.com/SenWu/dauphin/tree/master/dauphin/image/transforms

Then you good to go.

Thanks,
Sen

[senwu]

Close this for now, discussion in another issue #1.