lucidrains /x-transformers

A concise but complete full-attention transformer with a set of promising experimental features from various papers

★ 5,961⑂ 532👁 53⑃ 144● 281▤ 606
PythonMITCreated 2020-10-24
下载:2.16.12.16.1 ↓2.16.12026-02-12
+0
日增长
5,961
stars
532
forks
53
watchers
144
prs
281
issues
606
releases

README

x-transformers



A concise but fully-featured transformer, complete with a set of promising experimental features from various papers.

 Install

bash
$ pip install x-transformers


 Usage

Full encoder / decoder

python
import torch
from x_transformers import XTransformer

model = XTransformer(
    dim = 512,
    enc_num_tokens = 256,
    enc_depth = 6,
    enc_heads = 8,
    enc_max_seq_len = 1024,
    dec_num_tokens = 256,
    dec_depth = 6,
    dec_heads = 8,
    dec_max_seq_len = 1024,
    tie_token_emb = True       tie embeddings of encoder and decoder
)

src = torch.randint(0, 256, (1, 1024))
src_mask = torch.ones_like(src).bool()
tgt = torch.randint(0, 256, (1, 1024))

loss = model(src, tgt, mask = src_mask)  (1, 1024, 512)
loss.backward()


Decoder-only (GPT-like)

python
import torch
from x_transformers import TransformerWrapper, Decoder

model = TransformerWrapper(
    num_tokens = 20000,
    max_seq_len = 1024,
    attn_layers = Decoder(
        dim = 512,
        depth = 12,
        heads = 8
    )
).cuda()

x = torch.randint(0, 256, (1, 1024)).cuda()

model(x)  (1, 1024, 20000)


GPT3 would be approximately the following (but you wouldn't be able to run it anyways)

python

gpt3 = TransformerWrapper(
    num_tokens = 50000,
    max_seq_len = 2048,
    attn_layers = Decoder(
        dim = 12288,
        depth = 96,
        heads = 96,
        attn_dim_head = 128
    )
).cuda()


Encoder-only (BERT-like)

python
import torch
from x_transformers import TransformerWrapper, Encoder

model = TransformerWrapper(
    num_tokens = 20000,
    max_seq_len = 1024,
    attn_layers = Encoder(
        dim = 512,
        depth = 12,
        heads = 8
    )
).cuda()

x = torch.randint(0, 256, (1, 1024)).cuda()
mask = torch.ones_like(x).bool()

model(x, mask = mask)  (1, 1024, 20000)


State of the art image classification (SimpleViT)

python
import torch
from x_transformers import ViTransformerWrapper, Encoder

model = ViTransformerWrapper(
    image_size = 256,
    patch_size = 32,
    num_classes = 1000,
    attn_layers = Encoder(
        dim = 512,
        depth = 6,
        heads = 8,
    )
)

img = torch.randn(1, 3, 256, 256)
model(img)  (1, 1000)


Image - caption

python
import torch
from x_transformers import ViTransformerWrapper, TransformerWrapper, Encoder, Decoder

encoder = ViTransformerWrapper(
    image_size = 256,
    patch_size = 32,
    attn_layers = Encoder(
        dim = 512,
        depth = 6,
        heads = 8
    )
)

decoder = TransformerWrapper(
    num_tokens = 20000,
    max_seq_len = 1024,
    attn_layers = Decoder(
        dim = 512,
        depth = 6,
        heads = 8,
        cross_attend = True
    )
)

img = torch.randn(1, 3, 256, 256)
caption = torch.randint(0, 20000, (1, 1024))

encoded = encoder(img, return_embeddings = True)
decoder(caption, context = encoded)  (1, 1024, 20000)


PaLI, state of the art language-vision model

python
import torch
from x_transformers import ViTransformerWrapper, XTransformer, Encod

Top issues

Latest release

2.16.1 2.16.1 2026-02-12
What's Changed
 Support efficient flash attention for packed sequences using flash-attn-2.0 by @muthissar in https://github.com/lucidrains/x-transformers/pull/350

 New Contributors
 @muthissar made their first contribution in https://github.com/lucidrains/x-transformers/pull/350

Full Changelog: https://github.com/lucidrains/x-transformers/compare/2.16.0...2.16.1

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON