lucidrains /DALLE-pytorch

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

★ 5,628⑂ 637👁 91⑃ 128● 277▤ 180
PythonMITCreated 2021-01-05
下载:1.6.61.6.6 ↓1.6.62023-05-24
+0
日增长
5,628
stars
637
forks
91
watchers
128
prs
277
issues
180
releases

README

DALL-E in Pytorch


  
         
  
  
  Released DALLE Models
  Web-Hostable DALLE Checkpoints

  Yannic Kilcher's video

Implementation / replication of DALL-E (paper), OpenAI's Text to Image Transformer, in Pytorch.  It will also contain CLIP for ranking the generations.

---



Quick Start

Deep Daze or Big Sleep are great alternatives!

For generating video and audio, please see NÜWA

 Appreciation
  
This library could not have been possible without the contributions of janEbert, Clay, robvanvolt, Romain Beaumont, and Alexander! 🙏

 Status


- Hannu has managed to train a small 6 layer DALL-E on a dataset of just 2000 landscape images! (2048 visual tokens)



- Kobiso, a research engineer from Naver, has trained on the CUB200 dataset here, using full and deepspeed sparse attention



- (3/15/21) afiaka87 has managed one epoch using a reversible DALL-E and the dVaE here

- TheodoreGalanos has trained on 150k layouts with the following results

  
  

- Rom1504 has trained on 50k fashion images with captions with a really small DALL-E (2 layers) for just 24 hours with the following results



- afiaka87 trained for 6 epochs on the same dataset as before thanks to the efficient 16k VQGAN with the following discussion"results


  
   

  
Thanks to the amazing "mega b6696" you can generate from this checkpoint in colab - 

  


- (5/2/21) First 1.3B DALL-E from 🇷🇺 has been trained and released to the public! 🎉

- (4/8/22) Moving onwards to DALLE-2!

 Install

bash
$ pip install dalle-pytorch


 Usage

Train VAE

python
import torch
from dalle_pytorch import DiscreteVAE

vae = DiscreteVAE(
    image_size = 256,
    num_layers = 3,            number of downsamples - ex. 256 / (2  3) = (32 x 32 feature map)
    num_tokens = 8192,         number of visual tokens. in the paper, they used 8192, but could be smaller for downsized projects
    codebook_dim = 512,        codebook dimension
    hidden_dim = 64,           hidden dimension
    num_resnet_blocks = 1,     number of resnet blocks
    temperature = 0.9,         gumbel softmax temperature, the lower this is, the harder the discretization
    straight_through = False,  straight-through for gumbel softmax. unclear if it is better one way or the other
)

images = torch.randn(4, 3, 256, 256)

loss = vae(images, return_loss = True)
loss.backward()

 train with a lot of data to learn a good codebook


Train DALL-E with pretrained VAE from above

python
import torch
from dalle_pytorch import DiscreteVAE, DALLE

vae = DiscreteVAE(
    image_size = 256,
    num_layers = 3,
    num_tokens = 8192,
    codebook_dim = 1024,
    hidden_dim = 64,
    num_resnet_blocks = 1,
    temperature = 0.9
)

dalle = DALLE(
    dim = 1024,
    vae = vae,                   automatically infer (1) image sequence length and (2) number of image tokens
    num_text_tokens = 10000,     vocab size for text
    text_seq_len = 256,          text sequence length
    depth = 12,                  should aim to be 64
    heads = 16, 

Top issues

Discussions

Latest release

1.6.6 1.6.6 2023-05-24

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON