lucidrains /DALLE-pytorch
Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch
★ 5,628⑂ 637👁 91⑃ 128● 277▤ 180
TA的项目
+0
日增长
5,628
stars
637
forks
91
watchers
128
prs
277
issues
180
releases
README
DALL-E in Pytorch
Released DALLE Models
Web-Hostable DALLE Checkpoints
Yannic Kilcher's video
Implementation / replication of DALL-E (paper), OpenAI's Text to Image Transformer, in Pytorch. It will also contain CLIP for ranking the generations.
---
Quick Start
Deep Daze or Big Sleep are great alternatives!
For generating video and audio, please see NÜWA
Appreciation
This library could not have been possible without the contributions of janEbert, Clay, robvanvolt, Romain Beaumont, and Alexander! 🙏
Status
- Hannu has managed to train a small 6 layer DALL-E on a dataset of just 2000 landscape images! (2048 visual tokens)
- Kobiso, a research engineer from Naver, has trained on the CUB200 dataset here, using full and deepspeed sparse attention
- (3/15/21) afiaka87 has managed one epoch using a reversible DALL-E and the dVaE here
- TheodoreGalanos has trained on 150k layouts with the following results
- Rom1504 has trained on 50k fashion images with captions with a really small DALL-E (2 layers) for just 24 hours with the following results
- afiaka87 trained for 6 epochs on the same dataset as before thanks to the efficient 16k VQGAN with the following discussion"results
Thanks to the amazing "mega b6696" you can generate from this checkpoint in colab -
- (5/2/21) First 1.3B DALL-E from 🇷🇺 has been trained and released to the public! 🎉
- (4/8/22) Moving onwards to DALLE-2!
Install
bash
$ pip install dalle-pytorch
Usage
Train VAE
python
import torch
from dalle_pytorch import DiscreteVAE
vae = DiscreteVAE(
image_size = 256,
num_layers = 3, number of downsamples - ex. 256 / (2 3) = (32 x 32 feature map)
num_tokens = 8192, number of visual tokens. in the paper, they used 8192, but could be smaller for downsized projects
codebook_dim = 512, codebook dimension
hidden_dim = 64, hidden dimension
num_resnet_blocks = 1, number of resnet blocks
temperature = 0.9, gumbel softmax temperature, the lower this is, the harder the discretization
straight_through = False, straight-through for gumbel softmax. unclear if it is better one way or the other
)
images = torch.randn(4, 3, 256, 256)
loss = vae(images, return_loss = True)
loss.backward()
train with a lot of data to learn a good codebook
Train DALL-E with pretrained VAE from above
python
import torch
from dalle_pytorch import DiscreteVAE, DALLE
vae = DiscreteVAE(
image_size = 256,
num_layers = 3,
num_tokens = 8192,
codebook_dim = 1024,
hidden_dim = 64,
num_resnet_blocks = 1,
temperature = 0.9
)
dalle = DALLE(
dim = 1024,
vae = vae, automatically infer (1) image sequence length and (2) number of image tokens
num_text_tokens = 10000, vocab size for text
text_seq_len = 256, text sequence length
depth = 12, should aim to be 64
heads = 16, Top issues
#10Results57 comments · 2021-01-07#137Reproducing DALL-E using DeepSpeed32 comments · 2021-03-29#74OpenAIVae implementation27 comments · 2021-03-09#61Typical training results17 comments · 2021-03-01#301CI / Tests for DALLE-pytorch16 comments · 2021-06-13#192Allowing for custom trained VQGAN during DALLE training14 comments · 2021-04-13#170"adamw" optimizer + weight decay = poor generations12 comments · 2021-04-06#99Kl Loss correction12 comments · 2021-03-17
Discussions
#460Image Reconstruction Using DALLE-Pytorch for Higher Resolutions0 comments · 2024-12-10#437Dall-E training with low ressources (cuda memory error)0 comments · 2022-07-06#429Applying DALLE to Protein Imaging0 comments · 2022-05-27#423I am trainning the network on 8 million Logos0 comments · 2022-04-02#415is it better to have many captions or fewer detailed captions?0 comments · 2022-02-10#404.0 comments · 2021-12-31#396new 32 layer model2 comments · 2021-12-01#392How do I rank DALLE-pytorch output by using CLIP?0 comments · 2021-11-30