Okayish text-to-image generation results on COCO

[kobiso]

DALLE on COCO

• I trained DALLE + VQGAN (Taming transformers) with COCO dataset and got okayish generation results.
• Note that the model is not trained on Deepspeed DDP, but trained on horovod DDP.
  • dalle-pytorch version 0.7.2 is used.
• Pretrained model is available: https://github.com/kobiso/DALLE-reproduction

Main results

Text to image generation and re-ranking by CLIP (from training text)

Click here to see more results

Generate rest of image based on the given cropped image (from training text)

Click here to see more results

Text to image generation with custom text

Click here to see more results

Model spec

VAE

• Pretrained VQGAN

DALLE

• dim = 256
• text_seq_len = 80
• depth = 32
• heads = 8
• dim_head = 64
• reversible = 0
• attn_types = (full,sparse)

Optimization

• Optimizer: Adam
• Learning rate: start with 0.00045 and perform ReduceLROnPlateau
  • Learning rate has to be scaled by the number of gpus: 0.00045 * gpus
• Apply loss weighting
• Gradient clipping is necessary, unless NaN will occur
• Batch size: 64 * 8 (gpus)
  • I used horovod for multi-gpus training
• The model is not converged, so it could be better when it converged.

Opinion

• It seems generation from the training text is okay, but generation from custom text is poor.
• When CUB is only dealing with birds, COCO has more kinds of objects with complicated backgrounds.
  • So, training a generalized DALLE requires much more number of datasets.

[mhh0318]

Cool, how about your bpe code book? you train another one with COCO or just use the same as your CUB200 experiments?

[mhh0318]

Got the new one from your project! Awesome

[afiaka87]

In the image mask example - did you use an image from the training set as the mask? Or from elsewhere? @kobiso

[kobiso]

@afiaka87 I used images from the training sets :)

[JeniaJitsev]

Looks not bad at all, congratulations on that! Can you give us an idea how long training took until the observed state, with 8 GPUs that you are using?

[kobiso]

The model is not fully converged one (I accidently stopped training in the middle).
So, it took 3 days and 5 hours with 8 GPUs.
When we wait the model to converge, it should be around a week i guess.

[GuoxingY]

The results look great! Could you share the loss during the training?

[kobiso]

Sure :)

[GuoxingY]

Thanks so much ! Did you use the VQGAN pretrained on Imagenet and train transformer on COCO with split 2017 ? I tried to train on COCO with split 2014 with your config but did not get the results as good as yours.