Why Transformers Look the Way They Do: Deriving Q, K, and V

A detailed reconstruction of the Transformer architecture published in Towards Data Science argues that queries, keys, and values are not arbitrary design choices but near-inevitable consequences of a short chain of engineering constraints. For practitioners building or evaluating attention-based systems—including those exploring self-hosted deployments of large models—understanding why the architecture has the shape it does is more durable knowledge than any analogy about tokens asking questions.

The analysis traces a single thread from 2014 RNN-era attention through the 2017 "Attention Is All You Need" formulation by Vaswani et al., arriving at the full scaled dot-product attention equation by resolving concrete problems one at a time. The payoff is a mechanistic account that also explains two underappreciated components: why the mixing matrix W₀ exists, and why the feedforward MLP block is itself a key-value store.

Fixed Memory Is the Root Problem, Not an Analogy

Standard RNNs, including the LSTM introduced by Hochreiter and Schmidhuber in 1997, write each new input into a fixed-size state vector, forcing earlier information to be overwritten as the sequence grows. Bahdanau et al.'s 2015 paper resolved this by retaining the full history of RNN states as an expanding memory, adding direct connections from every past state to every future one. The cost was sequential computation: producing the Nth output still required N ordered steps, preventing GPU parallelism. Vaswani et al.'s 2017 insight was to discard the left-to-right recurrent connections entirely and rely solely on those direct cross-time connections. In a two-layer non-recurrent model, the total compute steps stay fixed at 2 regardless of sequence length, while the equivalent recurrent model's steps grow linearly.

Queries Arise from a Symmetry Problem, Not Semantics

Once recurrence is removed, the network needs dynamic rather than fixed weights—because sequence length varies between two tokens and two thousand. If the weight connecting input xᵢ to output unit Oⱼ is made a function of xᵢ alone, then xᵢ produces the same weight for every Oⱼ it feeds into, meaning adjacent output units compute nearly identical transformations. This weight symmetry defeats the flexibility of the architecture. Breaking it requires making each weight a function of both the source unit's value and a term unique to each destination unit. Since each output unit Oⱼ is responsible via a skip connection for additively modifying its corresponding input xⱼ, that input xⱼ is the natural symmetry-breaking second argument. The two arguments to the weight-generating function are exactly the key and the query before any projection is applied—no semantic framing required.

Cache Size Pressure Produces Wₖ, Wq, and W₀

Once the attention function is chosen to be a dot product (preferred over the tanh-based formulation of Bahdanau et al. because tanh produces only one high-output region versus two for the product, reducing unit independence), the reusable matrix-vector products must be cached for efficiency. Under concrete example conditions—sequence length 5,000, 50 layers, 20 value matrices V, cached vector dimension 1,000, and 2 bytes per float—the key-value cache totals 10 GB. Projecting into a reduced dimension r = 200 cuts that to 2 GB, a 5× reduction, but requires the projection matrices Wₖ and Wq to operate in that lower-dimensional space. The same logic applies to value vectors: keeping V as a down-projection matrix and introducing a separate mixing matrix W₀ to up-project back to the full model dimension of 1,000 keeps value-vector cache sizes tractable. Absorbing W₀ into V would force V to live in the full 1,000-dimensional space and eliminate the cache benefit.

The MLP Block Is a Second Key-Value Store

Geva et al.'s 2021 EMNLP paper demonstrated that the feedforward block functions as a key-value memory independent of attention. The incoming weights of each intermediate MLP unit act as key vectors that fire when the input matches a learned pattern; the outgoing weights act as value vectors that are then written to the residual stream. Geva et al. identified concrete examples of such patterns in a trained model, including sensitivity to sentences ending in the word "substitutes" and sentences expressing a "part of" relation. The second weight matrix W₂ exists precisely to allow each intermediate unit to write an arbitrary vector into the residual stream rather than being coupled to a single coordinate of it—and to down-project from a larger intermediate feature space back to the smaller model dimension, since more MLP units than model dimensions are desirable for expressivity.

The broader implication of this derivation is architectural rather than historical. Transformers' quadratic attention scaling is not an incidental flaw—it is baked into the direct all-to-all connectivity that replaced recurrence. Dao et al.'s 2022 FlashAttention work (NeurIPS, pp. 16344–16359) showed that even sparse attention mechanisms fail to beat full quadratic attention in wall-clock time because GPU memory bandwidth, not raw FLOP count, becomes the bottleneck. For teams evaluating emerging architectures like GLM's open-weights models or studying memory management approaches such as TencentDB's agent memory systems, the derivation here clarifies which Transformer constraints are fundamental and which are amenable to replacement—a prerequisite for building anything that could succeed it.