← Blog · 01 Nov 2026

FlashAttention, Part 1: Making softmax safe

Before we make attention fast, let's make sure its numbers behave.

Attention looks pretty compact on paper: multiply some matrices, apply softmax, multiply again. But behind that little equation are two questions: where do all the intermediate numbers live, and can we actually represent them safely?

Let's take this one part at a time. This post covers the attention setup and making softmax safe, including the three-pass algorithm and a worked example. Online softmax gets its own post next, and the block-matrix diagrams come in Part 3.

Before we start. The prerequisites for the series are basic linear algebra (matrix multiplication and transposes), high-school calculus (derivatives), a basic understanding of attention, and patience. We won't need the derivatives in this first part, but we'll come back to them for the backward pass.

The attention equation, without skipping the heads

First, the scaled dot-product attention formula:

Attention⁡(Q,K,V)=softmax⁡ ⁣(QKTdk)V.\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^{\mathsf T}}{\sqrt{d_k}}\right)V.

For head ii, apply the learned query, key, and value projections:

head⁡i=Attention⁡(QWiQ,KWiK,VWiV).\operatorname{head}_i=\operatorname{Attention}(QW_i^Q,KW_i^K,VW_i^V).

Then put the hh heads together and apply the output projection:

MultiHead⁡(Q,K,V)=Concat⁡(head⁡1,head⁡2,…,head⁡h)WO.\operatorname{MultiHead}(Q,K,V)=\operatorname{Concat}(\operatorname{head}_1,\operatorname{head}_2,\ldots,\operatorname{head}_h)W^O.

That's the multi-head setup. For the rest of the post, we only need to follow one head.

An important detail: from here on, QQ, KK, and VV are already projected through their respective WQW^Q, WKW^K, and WVW^V. We aren't applying those projections again inside the calculations below.

For the self-attention example, take Q,K,V∈RN×dQ,K,V\in\mathbb{R}^{N\times d}: NN tokens and dd features per head. We use the same value dimension dd to keep the shapes simple. We'll leave masks and dropout out of this first explanation.

Keep an eye on the matrix dimensions

To focus on the matrix shapes, let's temporarily suppress the 1/d1/\sqrt d factor and write

S=QKT∈RN×N,P=softmax⁡(S)∈RN×N,O=PV∈RN×d.S=QK^{\mathsf T}\in\mathbb{R}^{N\times N},\qquad P=\operatorname{softmax}(S)\in\mathbb{R}^{N\times N},\qquad O=PV\in\mathbb{R}^{N\times d}.

For actual scaled attention, replace SS by QKT/dQK^{\mathsf T}/\sqrt d. The shapes and the softmax argument below stay the same.

Here are the two shape checks written out:

Q⏟N×d  KT⏟d×N=S⏟N×N,P⏟N×N  V⏟N×d=O⏟N×d.\underbrace{Q}_{N\times d}\;\underbrace{K^{\mathsf T}}_{d\times N}=\underbrace{S}_{N\times N},\qquad \underbrace{P}_{N\times N}\;\underbrace{V}_{N\times d}=\underbrace{O}_{N\times d}.

The score in row ii, column jj is qiTkjq_i^{\mathsf T}k_j, where qiq_i and kjk_j are the query and key vectors viewed as columns. Softmax turns each row of scores into weights over all the keys.

q₁ᵀk1q₁ᵀk2q₁ᵀk3q₁ᵀk4q₁ᵀk5⋮⋮⋮⋮⋮q₅ᵀk1q₅ᵀk2q₅ᵀk3q₅ᵀk4q₅ᵀk50.10.050.50.150.2⋮⋮⋮⋮⋮0.30.10.350.20.05S: scoresP: weightsrow-wisesoftmax5 × 55 × 50.1+0.05+0.5+0.15+0.2 = 10.3+0.1+0.35+0.2+0.05 = 1
A 5×5 example with two illustrative probability rows. The dot products are symbolic; these numbers illustrate normalized rows, not a calculation from specified QQ and KK.

Notice what hasn't changed: softmax takes a 5×55\times5 matrix and returns a 5×55\times5 matrix. It changes the entries, not the dimensions. Each row now sums to one.

Why memory matters here

The inputs and output have NdNd entries each. But SS and PP each have N2N^2 entries. Those intermediate matrices grow quickly as we increase the sequence length.

For this computation, there are two important places to keep data:

HBM / DRAMGlobal memoryRoom for the big tensorsShared memoryMuch smaller; on chipCompute on pieces hereload a piececopy results back
The goal is to avoid unnecessary HBM traffic for the attention intermediates, not to eliminate all HBM access. Inputs still have to be read and outputs written.

The key idea is to calculate parts of the matrix multiplication using the smaller workspace, then copy the needed results back. We'll develop the actual block calculations in Part 3. First, let's make the softmax itself safe.

Where softmax gets into trouble

Given x∈RNx\in\mathbb{R}^N, softmax is

softmax⁡(x)i=exi∑j=1Nexj.\operatorname{softmax}(x)_i=\frac{e^{x_i}}{\sum_{j=1}^{N}e^{x_j}}.

Exponentiate each number, then divide by the sum. Straightforward—until the vector contains large values and those exponentials get too large to represent.

A floating-point warning. Formats such as float32 and float16 have a limited range. The problem here is specifically overflow in the intermediate exponentials: they may be unrepresentable even though the final probabilities are perfectly reasonable.

input teᵗ(0, 1)t ≤ 0 means eᵗ ≤ 1positive inputs:rapid growth
The curve shows why shifting the input helps. If we keep the exponent nonpositive, the exponential won't exceed one.

Let's sneak a constant into the exponent

We can multiply the numerator and denominator by the same positive constant cc without changing the ratio. Here's the full chain:

exi∑j=1Nexj=c exic∑j=1Nexj=c exi∑j=1Nc exj=elog⁡cexi∑j=1Nelog⁡cexj=exi+log⁡c∑j=1Nexj+log⁡c=exi−κ∑j=1Nexj−κ,κ=−log⁡c.\begin{aligned} \frac{e^{x_i}}{\sum_{j=1}^{N}e^{x_j}} &=\frac{c\,e^{x_i}}{c\sum_{j=1}^{N}e^{x_j}}\\ &=\frac{c\,e^{x_i}}{\sum_{j=1}^{N}c\,e^{x_j}}\\ &=\frac{e^{\log c}e^{x_i}}{\sum_{j=1}^{N}e^{\log c}e^{x_j}}\\ &=\frac{e^{x_i+\log c}}{\sum_{j=1}^{N}e^{x_j+\log c}}\\ &=\frac{e^{x_i-\kappa}}{\sum_{j=1}^{N}e^{x_j-\kappa}}, \qquad \kappa=-\log c. \end{aligned}

Same softmax, different exponent. We use κ\kappa for this constant so it doesn't get confused with the key matrix KK.

The shift is ours to choose. To prevent any positive exponent, we need xj−κ≤0x_j-\kappa\leq0 for every jj. Equivalently, κ\kappa must be at least as large as the maximum input. So the smallest shift that keeps every exponent nonpositive is:

κ=xmax⁡=max⁡1≤j≤Nxj.\kappa=x_{\max}=\max_{1\leq j\leq N}x_j.

With this choice, the largest input becomes zero, so its exponential is exactly one. Every other exponent is also nonpositive, which keeps every exponential at most one and the normalizer at least one. A larger shift would avoid overflow too, but would push every exponential lower than necessary. That gives us

softmax⁡(x)i=exi−xmax⁡∑j=1Nexj−xmax⁡.\boxed{\operatorname{softmax}(x)_i=\frac{e^{x_i-x_{\max}}}{\sum_{j=1}^{N}e^{x_j-x_{\max}}}.}

This avoids exponential overflow and guarantees a nonzero denominator. Very small contributions can still underflow or be rounded away, so we're not claiming perfect floating-point arithmetic.

The trick: subtract the maximum before exponentiating. The probabilities don't change. The size of the intermediate exponentials does.

What does the safe algorithm actually do?

For an N×NN\times N score matrix, do the following for each row. Each row is a vector with NN values.

Pass through one row Work Input reads
1. Find the maximum O(N)O(N) O(N)O(N)
2. Calculate the normalization factor O(N)O(N) O(N)O(N)
3. Normalize every element O(N)O(N) O(N)O(N)

Each pass costs O(N)O(N) per row, so all three passes together are still O(N)O(N) per row, or O(N2)O(N^2) for softmax over the whole matrix. The last pass also writes NN output values. These are logical input reads; if a row stays on chip, a new pass needn't mean another read from HBM.

The indexed version makes the order explicit:

m0=−∞,mi=max⁡(mi−1,xi),i=1,…,N,ℓ0=0,ℓj=ℓj−1+exj−mN,j=1,…,N,yk=exk−mNℓN,k=1,…,N.\begin{aligned} m_0&=-\infty, & m_i&=\max(m_{i-1},x_i), & i&=1,\ldots,N,\\ \ell_0&=0, & \ell_j&=\ell_{j-1}+e^{x_j-m_N}, & j&=1,\ldots,N,\\ & & y_k&=\frac{e^{x_k-m_N}}{\ell_N}, & k&=1,\ldots,N. \end{aligned}

First finish finding mNm_N, then use that final maximum for all the exponentials. The output is called yy here so we don't confuse the original inputs with their probabilities.

And here's the same three-pass algorithm in pseudocode:

maximum = -infinity
for value in values:
    maximum = max(maximum, value)

normalizer = 0
for value in values:
    normalizer += exp(value - maximum)

for index in range(length(values)):
    output[index] = exp(values[index] - maximum) / normalizer

Let's walk through the example: [3,2,5,1][3,2,5,1]

Step 1: find the maximum. For x=[3,2,5,1]x=[3,2,5,1], we have xmax⁡=5x_{\max}=5.

originalsubtract 53−22−3501−4

Step 2: calculate the normalization factor.

ℓ=e3−5+e2−5+e5−5+e1−5=e−2+e−3+e0+e−4≈1.20344.\begin{aligned} \ell&=e^{3-5}+e^{2-5}+e^{5-5}+e^{1-5}\\ &=e^{-2}+e^{-3}+e^0+e^{-4}\approx1.20344. \end{aligned}

Step 3: divide each shifted exponential by the same total.

y1=e3−5ℓ,y2=e2−5ℓ,y3=e5−5ℓ,y4=e1−5ℓ.y_1=\frac{e^{3-5}}{\ell},\qquad y_2=\frac{e^{2-5}}{\ell},\qquad y_3=\frac{e^{5-5}}{\ell},\qquad y_4=\frac{e^{1-5}}{\ell}.

Numerically,

softmax⁡(x)≈[0.11246,  0.04137,  0.83095,  0.01522].\operatorname{softmax}(x)\approx[0.11246,\;0.04137,\;0.83095,\;0.01522].

The biggest input gets the biggest weight. All four weights sum to one. And none of the exponentials needed to be larger than one.

That's Part 1: safe before fast

We now have the attention shapes, the memory motivation, the full constant-shift derivation, and a safe softmax we can compute by hand.

Next: can we do fewer than three passes? That's Part 2. We'll try calculating the maximum and normalizer together, see exactly where the obvious attempt breaks, and repair it without starting over.