imagenet

[PushparajaMurugan]

Hi, How do we train imagenet data?

[senwu]

Hi @PushparajaMurugan,

Thanks for your interest in our work!

To train on Imagenet dataset or other image datasets in general in our framework, you need to do the following things:

1. Create a new dataset for your data to load the dataset here: https://github.com/SenWu/dauphin/tree/master/dauphin/image/datasets
2. Update the data.py to accept your dataset here: https://github.com/SenWu/dauphin/blob/master/dauphin/image/data.py
3. Select your the transforms and policy you want to use here: https://github.com/SenWu/dauphin/blob/master/dauphin/image/augment_policy.py, all available transforms can be found here: https://github.com/SenWu/dauphin/tree/master/dauphin/image/transforms

Then you good to go.

Thanks,
Sen

[PushparajaMurugan]

Hi @senwu . Thank you for your response. Your results on CIFAR-10 is a really impressive one. I have tried to create the imagenet dataset object based on CIFARDataset. But, I'm facing issue with appending the data (as given in below). It came up with an error.

if index is None:
for i, (x, y) in enumerate(dataset):
X_dict["image_name"].append(f"{name}{split}{i}")
X_dict["image"].append(x)
Y_dict["labels"].append(y)

[aryabhata-archive]

@PushparajaMurugan., me too. Can you share the code?

[PushparajaMurugan]

@aryabhata-archive , I didn't make much modification. I have jus replaced the train_dataset , and test_dataset in get_dataloader based on "https://github.com/tgilewicz/uniformaugment/blob/master/UniformAugment/data.py". I don't see this need any more modification. I think.

[senwu]

Hi @PushparajaMurugan,

Can you share more information about your error? Maybe the snapshot of your error?

Your understanding is correct, you don't need to modify too much code to run your dataset. If you want to use the exact setting for transformations, you only need the modify data.py and have your own dataset.

Thanks,
Sen

[PushparajaMurugan]

Hi @senwu ,

This is the error I got. It didn't generate much information about the error. It just says that "1302863 Killed". May be, It is like a memory error? I'm not sure though.

Thanks,
Raja

[senwu]

Hi @PushparajaMurugan,

The log you shared above didn't provide any useful information. Can you directly run the cmd and share me all logs?

Sen

[PushparajaMurugan]

Hi @senwu ,
Even the log file doesn't have the information. But I heavily suspect that this is because of the appending the data in dataset object which may exhaust the RAM memory. Could you pls share the information about the machine you have used for training?

cmd.txt
config.txt
emmental.log

[senwu]

For our CIFAR exps, we use 12G GPU memory Titan X and 128G CPU memory.

From the log, I cannot see any useful information. I would suggest you to use pdb to check what cause the issue. I don't think CPU memory will be the issue since we don't load huge data into CPU. For the dataset like ImageNet, we load from disk for every batch.

[PushparajaMurugan]

hi @senwu, If that is the case, the problem is with my side, in "ImageNet dataloader function". Can I make a pull request, if that is ok? So that you can have a look at the code?

Thanks
Raja

[senwu]

Hi @PushparajaMurugan,

Of course. Any contribution is welcome!

Sen

[PushparajaMurugan]

Hi @senwu , have a look at the pull request. Do let me know if anything needs to be changed.

Thanks,
Rj.