Dao-AILab /flash-attention

Fast and memory-efficient exact attention

★ 25,299⑂ 3,231👁 144⑃ 1,240● 1,860▤ 131
PythonBSD-3-ClauseCreated 2022-05-19
下载:v2.8.3.post2v2.8.3.post2 ↓22026-10-13
+6
日增长
+54
周增长
+152
月增长
25,299
stars
3,231
forks
144
watchers
1,240
prs
1,860
issues
131
releases

README

FlashAttention
This repository provides the official implementation of FlashAttention and
FlashAttention-2 from the
following papers.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness  
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré  
Paper: https://arxiv.org/abs/2205.14135  
IEEE Spectrum article about our submission to the MLPerf 2.0 benchmark using FlashAttention.
!FlashAttention

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning  
Tri Dao

Paper: https://tridao.me/publications/flash2/flash2.pdf

!FlashAttention-2


 Usage

We've been very happy to see FlashAttention being widely adopted in such a short
time after its release. This page
contains a partial list of places where FlashAttention is being used.

FlashAttention and FlashAttention-2 are free to use and modify (see LICENSE).
Please cite and credit FlashAttention if you use it.


 FlashAttention-3 beta release
FlashAttention-3 is optimized for Hopper GPUs (e.g. H100). 

Blogpost: https://tridao.me/blog/2024/flash3/

Paper: https://tridao.me/publications/flash3/flash3.pdf

!FlashAttention-3 speedup on H100 80GB SXM5 with FP16

This is a beta release for testing / benchmarking before we integrate that with
the rest of the repo.

Currently released:
- FP16 / BF16 forward and backward, FP8 forward

Requirements: H100 / H800 GPU, CUDA = 12.3.

We highly recommend CUDA 12.8 for best performance.

To install:
sh
cd hopper
python setup.py install

To run the test:
sh
export PYTHONPATH=$PWD
pytest -q -s test_flash_attn.py

Once the package is installed, you can import it as follows:
python
from flash_attn_3 import flash_attn_interface
flash_attn_interface.flash_attn_func()


To install using uv, in your pyproject.toml:

toml
[project]
dependencies = [
    "flash-attn-3"
]

[tool.uv]
no-build-isolation = true

[tool.uv.sources]
flash-attn-3 = { git = "https://github.com/Dao-AILab/flash-attention", subdirectory = "hopper" }


 FlashAttention-4 (CuTeDSL)

FlashAttention-4 is written in CuTeDSL and optimized for Hopper and Blackwell GPUs (e.g. H100, B200).

To install:
sh
pip install flash-attn-4


If you're on CUDA 13, we recommend installing with the cu13 extra for best performance:
sh
pip install "flash-attn-4[cu13]"


Once installed, you can use it as follows:
python
from flash_attn.cute import flash_attn_func

out = flash_attn_func(q, k, v, causal=True)


 Installation and features
Requirements:
- CUDA toolkit or ROCm toolkit
- PyTorch 2.2 and above.
- packaging Python package (pip install packaging)
- psutil Python package (pip install psutil)
- ninja Python package (pip install ninja) 
- Linux. Might work for Windows starting v2.3.2 (we've seen a few positive reports) but Windows compilation still requires more testing. If you have ideas on how to set up prebuilt CUDA wheels for Windows, please reach out via Github issue.

\ Make sure that ninja is installed and that it works correctly (e.g. ninja
--version then echo 

Top issues

Discussions

Latest release

v2.8.3.post2 v2.8.3.post2 2026-10-13
Maintenance release of FlashAttention 2 on top of v2.8.3.post1. No kernel feature changes.

- Backport 2622: torch.load(..., weights_only=True) in flash_attn/utils/distributed.py and flash_attn/models (CVE-2026-31253 / GHSA-7g5w-pq96-8c5w). 2974
- Backport 2702: safe tarfile extraction in hopper/setup.py (CVE-2026-62239 / GHSA-w9r7-4gwr-j959). 2974
- Backport 2260: fix int32 overflow in the FA2 backward dq_accum offsets, which caused an illegal memory access or NaN gradients when batch  round128(seqlen_q)  nheads  headdim exceeded 2^31 (2987). 2994
- setup.py: request cu13 prebuilt wheels for 

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON