lucidrains /x-transformers
A concise but complete full-attention transformer with a set of promising experimental features from various papers
★ 5,961⑂ 532👁 53⑃ 144● 281▤ 606
TA的项目
+0
日增长
5,961
stars
532
forks
53
watchers
144
prs
281
issues
606
releases
README
x-transformers
A concise but fully-featured transformer, complete with a set of promising experimental features from various papers.
Install
bash
$ pip install x-transformers
Usage
Full encoder / decoder
python
import torch
from x_transformers import XTransformer
model = XTransformer(
dim = 512,
enc_num_tokens = 256,
enc_depth = 6,
enc_heads = 8,
enc_max_seq_len = 1024,
dec_num_tokens = 256,
dec_depth = 6,
dec_heads = 8,
dec_max_seq_len = 1024,
tie_token_emb = True tie embeddings of encoder and decoder
)
src = torch.randint(0, 256, (1, 1024))
src_mask = torch.ones_like(src).bool()
tgt = torch.randint(0, 256, (1, 1024))
loss = model(src, tgt, mask = src_mask) (1, 1024, 512)
loss.backward()
Decoder-only (GPT-like)
python
import torch
from x_transformers import TransformerWrapper, Decoder
model = TransformerWrapper(
num_tokens = 20000,
max_seq_len = 1024,
attn_layers = Decoder(
dim = 512,
depth = 12,
heads = 8
)
).cuda()
x = torch.randint(0, 256, (1, 1024)).cuda()
model(x) (1, 1024, 20000)
GPT3 would be approximately the following (but you wouldn't be able to run it anyways)
python
gpt3 = TransformerWrapper(
num_tokens = 50000,
max_seq_len = 2048,
attn_layers = Decoder(
dim = 12288,
depth = 96,
heads = 96,
attn_dim_head = 128
)
).cuda()
Encoder-only (BERT-like)
python
import torch
from x_transformers import TransformerWrapper, Encoder
model = TransformerWrapper(
num_tokens = 20000,
max_seq_len = 1024,
attn_layers = Encoder(
dim = 512,
depth = 12,
heads = 8
)
).cuda()
x = torch.randint(0, 256, (1, 1024)).cuda()
mask = torch.ones_like(x).bool()
model(x, mask = mask) (1, 1024, 20000)
State of the art image classification (SimpleViT)
python
import torch
from x_transformers import ViTransformerWrapper, Encoder
model = ViTransformerWrapper(
image_size = 256,
patch_size = 32,
num_classes = 1000,
attn_layers = Encoder(
dim = 512,
depth = 6,
heads = 8,
)
)
img = torch.randn(1, 3, 256, 256)
model(img) (1, 1000)
Image - caption
python
import torch
from x_transformers import ViTransformerWrapper, TransformerWrapper, Encoder, Decoder
encoder = ViTransformerWrapper(
image_size = 256,
patch_size = 32,
attn_layers = Encoder(
dim = 512,
depth = 6,
heads = 8
)
)
decoder = TransformerWrapper(
num_tokens = 20000,
max_seq_len = 1024,
attn_layers = Decoder(
dim = 512,
depth = 6,
heads = 8,
cross_attend = True
)
)
img = torch.randn(1, 3, 256, 256)
caption = torch.randint(0, 20000, (1, 1024))
encoded = encoder(img, return_embeddings = True)
decoder(caption, context = encoded) (1, 1024, 20000)
PaLI, state of the art language-vision model
python
import torch
from x_transformers import ViTransformerWrapper, XTransformer, EncodTop issues
#159How to use the rope scaling with x_transformers ?21 comments · 2023-06-29#166Feature request: support return_mems in ContinuousTransformerWrapper16 comments · 2023-07-17#212ONNX export failed14 comments · 2023-11-27#21Any tips for speeding up generation?14 comments · 2021-03-11#143RuntimeError: No available kernel. Aborting execution.12 comments · 2023-05-12#292Support for a mask during autoregressive generation with Key-Value Caching10 comments · 2024-11-12#332Question: `prepend_embeds` & `prepend_mask` + KV caching8 comments · 2025-06-26#70Absolute positional embedding present with ALiBi relative positional encoding.7 comments · 2021-11-28
Latest release
2.16.1 2.16.1 2026-02-12What's Changed Support efficient flash attention for packed sequences using flash-attn-2.0 by @muthissar in https://github.com/lucidrains/x-transformers/pull/350 New Contributors @muthissar made their first contribution in https://github.com/lucidrains/x-transformers/pull/350 Full Changelog: https://github.com/lucidrains/x-transformers/compare/2.16.0...2.16.1