Attention.

Answer. For each query, compute compatibility scores with permitted keys, normalize those scores, and return a weighted sum of the associated value vectors. In standard softmax attention without dropout, the coefficients are nonnegative and sum to one. The output is a value mixture, not a key or token index.

Follow-up. Must all attention normalize to one? No. That is the contract of normalized softmax attention; other gating or attention operators can use different rules.

Trap. “Focus on important words” describes an intuition, not the computation.

Answer. Q, K, V are input-dependent activations. WQ, WK, WV are learned parameters that map hidden states into matching and payload spaces. A query describes what a position seeks; keys describe candidates for matching; values carry information to mix. The same projection parameters are normally shared across sequence positions within a layer.

Follow-up. Why separate keys and values? Features that identify useful information need not be the features you want to return.

Trap. Calling Q, K, V the learned weights confuses activations with parameters.

Answer. Self-attention means all three come from the same source sequence, not that they are identical. Different projections of X produce different roles: Q=XWQ, K=XWK, V=XWV. In deeper layers X already contains contextual information from earlier layers.

Follow-up. What if WQ=WK? Unmasked logits become a symmetric Gram matrix, but row-softmax weights need not be symmetric. A larger-norm other key can score above the query’s own key.

Trap. Tied Q/K does not guarantee maximal self-attention.

Answer. A plain average ignores the query and mixes all values equally. A learned content-dependent mixture can favor different sources for different inputs. Soft weights also allow gradients to flow through scores. Hard argmax selection is discontinuous and has zero derivative almost everywhere with respect to scores.

Follow-up. Can hard retrieval still be learned? Yes, using surrogate losses, relaxations, or separate retrieval training. Ordinary backpropagation through the discrete selection itself is not enough.

Trap. A differentiable mixture is not guaranteed to be the best retrieval system for every scale.

Answer. q·k = ||q|| ||k|| cos(theta), so it combines angular alignment and vector norms. Norms can carry learned signal or cause overly large scores. Calling the score similarity does not mean it is cosine similarity or a calibrated relevance probability.

Follow-up. How does cosine attention differ? Normalize Q/K norms and typically introduce a temperature or scale to control score spread.

Trap. A larger score can come from a larger norm rather than a better directional match.

Answer. Additive attention uses a learned function such as uᵀtanh(Wq q + Wk k + b). Bilinear attention scores qᵀWk. Dot product uses qᵀk, often after separate projections. Dot products map naturally to efficient matrix multiplication; additive scoring offers a different parameterization and can reconcile different source widths.

Follow-up. Is additive always more expressive in practice? No unconditional ordering follows without specifying width, parameter budget, and scoring family.

Trap. Hardware efficiency and parameterization matter more than a blanket ranking of scoring functions.

Answer. For scores shaped [B,H,nq,nk], softmax runs along nk, the last axis. Each query/head receives a separate distribution over permitted keys. Columns generally do not sum to one. Normalizing queries instead produces a different operator while preserving plausible tensor shapes.

Follow-up. What test catches the error? Use rectangular nq≠nk and assert each valid row sums to one before dropout; compare a scalar reference.

Trap. Shape checks alone cannot detect the wrong softmax axis.

Answer. Under independent zero-mean unit-variance query/key coordinates, q·k is a sum of dk products with variance dk. Dividing by sqrt(dk) gives variance one, limiting initial score spread and reducing softmax saturation. This uses per-head key width, not sequence length.

Follow-up. Why not divide by dk? Under the same assumptions, variance becomes 1/dk and increasingly flat scores are encouraged.

Trap. The initialization argument is not a permanent guarantee after learning.

Answer. For tau>0, softmax(s/tau) sharpens as tau decreases and flattens as tau increases. As tau approaches zero, a unique maximum gets all mass; tied maxima split mass equally. As tau approaches infinity, all finite permitted scores get uniform mass.

Follow-up. Can you change tau at inference? Yes mechanically, but it changes the learned function and needs quality evaluation. Attention temperature is separate from vocabulary sampling temperature.

Trap. Masked keys remain excluded; the uniform limit is over permitted finite keys.

Answer. Adding or subtracting a common row constant cancels between numerator and denominator. Subtracting the maximum makes the largest exponent one and all others at most one, preventing overflow from large positive logits. It preserves the mathematical probabilities.

Follow-up. Does it fix an all-masked row? No. If every score is negative infinity, maximum subtraction is undefined and there is no distribution over an empty support.

Trap. Stability tricks do not replace an explicit empty-support policy.

Answer. In standard next-token training, query position i can read keys j≤i, including itself. The hidden state at i predicts token i+1 because the targets are shifted. Mask future logits before softmax. Causality is about the prediction factorization, not making every attention matrix strictly lower triangular.

Follow-up. What if labels are unshifted? The model can learn to copy the visible current token, so a causal mask alone does not prevent that leakage.

Trap. Diagonal permission and target shifting must be reasoned about together.

Answer. A transparent all-negative-infinity softmax is undefined. For a real query, fail fast unless the application defines another semantic result. For a padded query, exclude it or compute a safe fallback and zero its output before downstream use, then exclude its loss. Backend behavior may differ.

Follow-up. Why not use a finite negative sentinel? Equal finite sentinels produce a uniform distribution over forbidden keys, not tiny probabilities or an empty result.

Trap. Zeroing only the loss may not prevent intermediate NaNs from contaminating gradients.

Answer. Key-padding masks block nonexistent pad positions. Packed-sequence masks prevent one example from reading another example’s tokens in the same tensor. Combine permitted relations with logical AND, along with causality if needed. Padded query outputs or losses require separate handling.

Follow-up. Can position resets replace segment masks? No. Reused position indices do not stop attention across packed examples.

Trap. Specify whether Boolean True means allowed or blocked before combining masks.

Answer. Cross-attention takes Q from one sequence and K/V from another. Its output length equals query length, while K/V share source length. A decoder can query encoder states, or text can query visual features. Self-attention uses one source sequence for all roles.

Follow-up. Is cross-attention causal? Not inherently. Permissions depend on the task; a fixed fully observed encoder source is usually visible in full except padding.

Trap. Self/cross describes data sources; causal/bidirectional describes visibility.

Answer. For ordinary softmax attention before dropout, each output row is a convex combination of allowed value rows. The claim concerns projected values, not raw embeddings. Dropout can break the row-sum constraint; output projection and residual addition change the space/result to which the claim applies.

Follow-up. What about signed weights? A signed combination need not be convex; it is affine only if the weights also sum to one.

Trap. Softmax does not produce negative weights merely because of rounding.

Answer. It resembles normalized kernel regression: o(q)=sum_j w(q,k_j)v_j / sum_j w(q,k_j), with softmax affinity w=exp(q·k/sqrt(dk)). Neural attention learns both matching features and payloads. This view helps derive feature-map alternatives.

Follow-up. Does that imply a symmetric positive-semidefinite token affinity matrix? Not generally: Q and K can use different projections and masks can be directional.

Trap. The similarity-function meaning of kernel differs from a GPU compute kernel.

Answer. Dense soft attention mixes available values inside a model layer. External retrieval commonly selects documents or passages from a much larger store and returns them as context or structured results. Retrieval can be learned, but discrete top-k selection is not the same differentiable weighted sum.

Follow-up. Can a mixture equal one original value without one-hot weights? Yes, for example when all values are equal.

Trap. Do not claim every possible source contributes: masks, sparsity, and numerical underflow matter.

Answer. Weights describe one layer/head’s mixing coefficients. Output effects also depend on values, output projections, residual paths, later layers, and the target prediction. Distinct attention patterns can produce the same output. High mass is not a calibrated probability that a token caused the decision.

Follow-up. What should you use instead? Combine attribution with controlled perturbations and behavior tests, stating baseline and causal assumptions. Integrated gradients is not automatically causal identification.

Trap. A plausible heatmap or generated rationale is evidence to inspect, not proof.

Answer. No. Recurrent encoder-decoder models used attention to access multiple source hidden states instead of relying only on a final compressed state. Bahdanau-style additive alignment and Luong-style scoring predate Transformers. The Transformer made attention the primary token-mixing mechanism rather than recurrence.

Follow-up. What bottleneck did it address? The decoder no longer had to recover every source detail from one fixed final encoder vector.

Trap. Attention is a mechanism; Transformer is an architecture built around it.

Answer. Teacher forcing supplies every training input token in advance. A causal mask lets each position compute without reading future content, so positions can be processed together within each layer. During ordinary generation the next input token is unknown until the previous prediction is selected.

Follow-up. What does caching remove? Repeated computation of previous K/V and causal prefix states, not the dependency on the newly generated token. Speculation can verify proposals in parallel.

Trap. The generation dependency is not proof that attention itself is inherently sequential.

Answer. Q has shape [nq,dk], K [nk,dk], and V [nk,dv]. QKᵀ is [nq,nk]; row-softmax preserves that shape; multiplying by V gives [nq,dv]. Query and key feature widths agree; key and value sequence lengths agree.

Follow-up. Can dk differ from dv? Yes mathematically. A fast implementation may impose narrower supported shape constraints.

Trap. Neither equal query/key length nor equal key/value width is required by the basic equation.

Answer. For a_i=exp(s_i)/sum_j exp(s_j), differentiation gives da_i/ds_j=a_i(delta_ij−a_j). Therefore J=diag(a)−aaᵀ. Off-diagonal terms describe competition between logits. Row sums are zero because a uniform logit shift changes nothing.

Follow-up. What happens near one-hot? The entries become small, so score gradients can shrink. At temperature tau, differentiation with respect to unscaled scores adds 1/tau.

Trap. a_i(1−a_i) gives only the diagonal, not the full derivative.

Answer. Let G=dL/dO and A=softmax(QKᵀ/sqrt(dk)). Then dV=AᵀG, E=GVᵀ, and D=A elementwise-multiplied by (E−row_sum(A*E)). Finally dQ=DK/sqrt(dk), dK=DᵀQ/sqrt(dk). D is the gradient with respect to scaled logits.

Follow-up. How do input gradients combine in self-attention? Add dQ WQᵀ, dK WKᵀ, and dV WVᵀ because X feeds all three branches.

Trap. Do not omit the scale or treat the softmax Jacobian as elementwise.

Answer. Maximize sum_j a_j s_j + tau H(a) over the probability simplex, where H(a)=−sum_j a_j log a_j and tau>0. Lagrange stationarity yields s_j−tau(log a_j+1)+lambda=0, so a_j is proportional to exp(s_j/tau). Normalizing gives softmax.

Follow-up. How do masks enter? Restrict the simplex to permitted keys. An empty feasible support does not define a distribution.

Trap. Entropy reward has a positive coefficient; confusing its sign gives the wrong optimizer.

Answer. For row permutation P, Q′=PQ, K′=PK, V′=PV. Scores transform as PSPᵀ and row-softmax gives A′=PAPᵀ. Therefore O′=A′V′=PAV=PO. This requires shared token projections and no fixed order-dependent information.

Follow-up. When is the output invariant instead? A symmetric pooling operation such as mean can make a sequence-level readout invariant.

Trap. A fixed causal mask or fixed positions generally breaks arbitrary token-permutation equivariance.

Answer. For fixed queries, jointly permuting K and V preserves their pairings and leaves the weighted sum unchanged. Corresponding mask and score-bias columns must be permuted too. This differs from self-attention equivariance, where queries also move.

Follow-up. What if only K is permuted? Keys then select the wrong payload associations, changing the function in general.

Trap. Order-insensitive source aggregation requires all relevant source metadata to transform consistently.

Answer. No. rank(QKᵀ)≤dk, but row-wise softmax is nonlinear. Q=K=[[0],[1]] gives rank-one logits [[0,0],[0,1]], while probabilities [[1/2,1/2],[1/(1+e),e/(1+e)]] have nonzero determinant and rank two.

Follow-up. Does full rank imply arbitrary attention expressivity? No. Low-dimensional parameterization still constrains which logit relationships are possible.

Trap. Do not move softmax through a low-rank factorization as if it were linear.

Answer. The unmasked score matrix QQᵀ is symmetric. Row normalization uses different denominators for different rows, so softmax(QQᵀ) need not be symmetric. Causal or asymmetric masks can also remove score symmetry.

Follow-up. Must self-score be largest? No. If q=[1] and another key=[2], q·k_other=2 exceeds q·q=1. Equal normalized vector norms change that comparison.

Trap. Symmetric logits and symmetric probabilities are different properties.

Answer. Nothing changes in softmax: exp(s_j+c) shares a common factor exp(c), which cancels. The output stays the same for fixed values. This gives stable max-subtraction and a null direction of the softmax Jacobian.

Follow-up. What if the constant is added to one column across all queries? That favors one key in each row and generally changes outputs.

Trap. Row-wise shift invariance is not invariance to arbitrary elementwise shifts.

Answer. For invertible R, replacing Q by QR and K by K R^(-T) preserves QKᵀ. A simple case scales Q by c and K by 1/c. This non-uniqueness shows that individual matching coordinates are not uniquely identifiable from scores.

Follow-up. What if both are scaled by c? Scores scale by c², changing softmax sharpness unless c²=1 or the scores are degenerate.

Trap. Preserving a product does not mean transformed parameters have identical optimization or regularization behavior.

Answer. With normalized weights and no attention dropout, every output is that same value vector regardless of the score distribution. Q/K can change without changing O. This holds with masks as long as at least one value remains permitted.

Follow-up. What happens to score gradients for a loss depending only on that mixture? They cancel because the mixture is independent of scores; gradients to values can still flow.

Trap. Different attention maps do not necessarily mean different outputs.

Answer. Its softmax weight is one, and the output equals its value. The normalized distribution is constant with respect to its finite score, so score gradients for that row are zero. The value path still receives gradients.

Follow-up. Is this the same as all keys masked? No. One-key support defines a valid distribution; empty support does not.

Trap. A zero score gradient can be mathematically correct, not a failed implementation.

Answer. Duplicating one pair adds another identical exponential term to the softmax denominator and doubles that pair’s combined unnormalized mass. Unless the surrounding configuration makes it irrelevant, the output changes. Duplicating every pair equally preserves the mixture under content-only attention.

Follow-up. How can this matter in context design? Repeated evidence can attract more aggregate mass without being more reliable. Position encodings can further distinguish duplicates.

Trap. Attention is not automatically invariant to repeated copies of one source.

Answer. Associativity applies to matrix multiplication, but softmax is a nonlinear row normalization inserted between products. Removing it changes both weighting and normalization. Feature-map linear attention creates a different or approximate affinity that can be evaluated associatively with a separate denominator.

Follow-up. When can reassociation be exact? For the unnormalized linear operator (QKᵀ)V, assuming identical arithmetic idealizations; finite precision can still differ.

Trap. An algebraic speedup must preserve every intervening operation, not just matrix dimensions.

Answer. Use small float64 Q/K/V and a scalar loss such as sum(O*G). Perturb each input coordinate by ±epsilon and compare the centered difference to the analytic gradient. Disable dropout and avoid highly saturated logits. Test masks with nonempty support.

Follow-up. Why both absolute and relative tolerances? Relative error is unstable near zero gradients; absolute error alone may hide large proportional errors.

Trap. A finite forward output does not establish correct gradients.

Answer. It needs control of coordinate-product means, variances, and cross-coordinate covariances. Independence and zero-mean unit-variance coordinates are convenient sufficient conditions. Shared or correlated learned projections can violate them, particularly self-scores with Q/K tied.

Follow-up. What is the general warning? Var(sum z_i)=sum Var(z_i)+2 sum_{i<j} Cov(z_i,z_j). Covariances need not vanish after learning.

Trap. LayerNorm normalizes token features but does not prove query-key products are independent.

Answer. Yes. Standard inverted dropout samples entries of A, zeros some, and scales survivors by 1/(1−p). A particular row need not sum to one, so its output need not be a convex combination. Conditional on A and V, the expected output matches AV.

Follow-up. Does evaluation keep that randomness? A module dropout layer disables it in eval; functional SDPA requires dropout_p=0 explicitly.

Trap. Expectation-preserving is not sample-by-sample normalization-preserving.

Answer. A direct permitted edge only makes information accessible. Learned logits, value compression, competing distractors, limited head width, later processing, and training distribution determine whether it is used. Long paths are not required for dense attention, but useful retrieval is not guaranteed.

Follow-up. What experiment separates access from use? Fix evidence and move it across context depth while controlling token budget and distractors; evaluate the actual answer.

Trap. Graph reachability is not an accuracy guarantee.

Answer. Neither per-position operator should casually be called translation invariant. Convolutions are translation equivariant under suitable boundary assumptions. Content-only attention is permutation equivariant, which includes compatible shifts/permutations. Relative-position attention can preserve shift relations when masks and boundaries transform consistently.

Follow-up. What produces invariance? An appropriate symmetric global readout can remove output-position dependence.

Trap. Conflating invariance with equivariance is the error; boundaries and positional information matter.

Answer. Max subtraction prevents exponential overflow for finite scores, but QKᵀ can overflow before softmax, tiny probabilities can underflow, reductions can accumulate rounding error, and all-masked rows remain undefined. Large or nonfinite values can also corrupt AV.

Follow-up. How do you respond? Check score/value ranges, use appropriate accumulation precision, validate inputs/masks, and compare to a high-precision reference.

Trap. “Numerically stable softmax” does not mean the entire attention block is numerically safe.

Answer. Each head computes a separate query-dependent distribution and value mixture. This allows different feature subspaces to use different routing patterns at the same position. One wide head uses one distribution across its value coordinates. Learned output projection mixes the concatenated results.

Follow-up. Do heads correspond to human concepts? Sometimes interpretable patterns emerge, but fixed roles are not guaranteed and heads may be redundant.

Trap. Multi-head structure is not simply an ensemble of independent complete models.

Answer. At residual width d with combined query/key/value widths each d, the three input projections contribute 3d² weights and the output projection d², totaling 4d² without biases. Splitting into H heads of width d/H does not multiply that count by H.

Follow-up. What changes the calculation? Different key/value widths, GQA KV projections, biases, or a different output width.

Trap. State whether total width or per-head width is held fixed when comparing head counts.

Answer. Project X[B,T,d] into Q/K/V, reshape each into [B,T,H,dh], then transpose to [B,H,T,dh]. Compute scores [B,H,T,T] and mixtures [B,H,T,dh]. Transpose back before merging H and dh, then apply WO.

Follow-up. Why can view fail after transpose? Strides may be noncontiguous. reshape can copy; contiguous().view makes the storage requirement explicit.

Trap. Reshape alone does not permute the head and sequence axes.

Answer. Concatenation preserves separate head outputs; WO maps that combined feature vector to the desired residual width and mixes information across heads. It is a learned linear operation, not another attention distribution.

Follow-up. Can WO be omitted? It defines a valid variant if dimensions fit, but changes expressivity and parameterization relative to standard MHA.

Trap. The post-projection output need not stay inside the original values’ convex hull.

Answer. No. With fixed d, increasing H reduces dh. More independent mixtures trade against narrower per-head matching/payload spaces. Leading projection and pairwise matmul costs can stay similar, while score storage, softmax overhead, and hardware efficiency change.

Follow-up. How would you choose H? Compare quality and real workload latency at controlled width, parameter budget, training setup, and context length.

Trap. Same leading-order arithmetic is not identical runtime or an unconditional expressivity improvement.

Answer. Attention mixes information across positions. The FFN applies a learned nonlinear feature transformation independently to each token using shared weights. A common form expands width, applies an activation, and projects back; gated variants use multiplicative feature interactions.

Follow-up. Is FFN optional mathematically? A model without it is possible, but it is a different architecture and often has different capacity and training behavior.

Trap. Do not describe attention and FFN as interchangeable because both contain matrices.

Answer. Pre-norm uses X+Sublayer(Norm(X)); post-norm uses Norm(X+Sublayer(X)). Pre-norm preserves a direct residual gradient path and often makes deep optimization easier. The original Transformer used post-LayerNorm. Neither arrangement universally wins on every quality/optimization metric.

Follow-up. Why do residuals matter? They let sublayers learn refinements around an identity path and improve optimization; they are not a mathematical prerequisite for defining attention.

Trap. Easier training is not proof of universally better final accuracy.

Answer. LayerNorm normalizes each token across feature dimensions using mean and variance, typically with learned affine parameters. RMSNorm scales by root mean square without subtracting the mean. Attention softmax normalizes candidate-key logits into a per-query distribution. These operations act on different axes and serve different purposes.

Follow-up. Does either feature norm guarantee stable QK scores? No. Learned projections and correlations still matter.

Trap. Do not substitute feature normalization for key-axis softmax or its scaling argument.

Answer. An encoder commonly uses bidirectional self-attention over observed input. A decoder-only language model uses causal self-attention to model a next-token factorization. An encoder-decoder adds target-side causal self-attention plus cross-attention to encoded source states. Objectives and visibility must match intended use.

Follow-up. Can an encoder become a decoder just by changing an inference mask? That defines a new computation but does not establish training for the desired conditional distribution.

Trap. Architecture, training objective, and generation procedure are separate but coupled choices.

Answer. Teacher forcing supplies the true previous target tokens during training; free-running generation supplies the model’s previous selected tokens. Both can use the same causal attention rule, but the input distributions differ when generated history contains errors.

Follow-up. Does a causal mask eliminate that distribution shift? No. It prevents future access, not exposure to model-generated prefixes.

Trap. Mask correctness and robustness to accumulated generation errors are different issues.

Answer. Learned positions are a trainable lookup table with a finite supported index range. Sinusoidal vectors are generated from fixed frequencies and can be evaluated at any index. Both are often added to token embeddings, introducing absolute-index information through projected content/position interactions.

Follow-up. Does a defined sinusoid imply reliable extrapolation? No. The model may not use unfamiliar positions or distances well without suitable training.

Trap. Mathematical addressability and task-level usable context are not the same.

Answer. For column vectors, q_rot=R(mθ)q and k_rot=R(nθ)k. Then q_rotᵀk_rot=qᵀR(mθ)ᵀR(nθ)k=qᵀR((n−m)θ)k. Repeat over coordinate pairs with different frequencies. The explicit rotation depends on relative offset.

Follow-up. Does the whole model become shift invariant? Not necessarily. Content hidden states, masks, and other positional signals can change. The identity holds with underlying q/k fixed.

Trap. A transpose changes the sign of the first rotation; do not reverse the relative offset accidentally.

Answer. In a common RoPE decoder, store each key after applying its intended logical position once; values are usually unrotated. New queries/keys use continued logical positions, not local chunk indices reset to zero. Preserve checkpoint-specific frequencies, pairing, and scaling.

Follow-up. What if a sliding cache evicts tokens? Physical storage offsets can change while logical positions continue. Keep those notions separate.

Trap. Applying rotation twice or resetting positions silently breaks cache parity.

Answer. It adds a negative distance-dependent logit bias with a different fixed slope per head. For causal key n≤query m, a common form subtracts slope_h*(m−n). This imposes recency preference without input position vectors.

Follow-up. Does it guarantee retrieval beyond training length? No. The formula extrapolates, but quality remains model/task-dependent and the recency penalty can suppress distant evidence.

Trap. The exact original slope schedule depends on head count; a single eight-head example is not the general formula.

Answer. New distances and phase combinations can be unfamiliar; learned retrieval behavior may not extrapolate. Position interpolation compresses the target index range, changing local angular resolution. Base or frequency-aware scaling changes that trade-off differently. Serving must match the trained configuration.

Follow-up. How do you validate an extension? Measure short-context regressions, evidence-depth curves, long-document reasoning, and cache parity, not just successful allocation or perplexity.

Trap. Context extension is not a universally safe configuration-only optimization.

Answer. Image patches are projected to token features, position information is added or incorporated, and attention mixes spatial tokens. With simple nonoverlapping P×P patches, an H×W divisible image yields (H/P)(W/P) patches before special tokens and merging.

Follow-up. What happens when both image dimensions double at fixed P? Token count quadruples; dense pairwise attention work grows about sixteenfold at fixed width.

Trap. Actual multimodal tokenizers can resize, merge, crop, or compress patches, so inspect the real pipeline.

Answer. One or more learned queries attend over a sequence of input features to produce a fixed-size set of summaries. Query count sets output length independently of input length. Without positional information, jointly permuting source features preserves each query output.

Follow-up. Why multiple queries? They can learn different summary mixtures, though diversity is not guaranteed without suitable training signals.

Trap. A learned query vector is a parameter; its projected/contextual activation still depends on architecture.

Answer. They share the broad idea of input-dependent weighting, but differ in axis, score function, and permission structure. Graph attention often aggregates allowed neighbors; channel attention may gate feature channels; spatial attention may weight image locations. Not every named attention module computes softmax(QKᵀ)V.

Follow-up. What should you clarify first? Which entities are queries/candidates, what is normalized, and whether information is gated or aggregated.

Trap. The word attention alone does not specify the operator.

Answer. Separate heads have separate normalizations and can choose different mixtures for different feature subspaces. Combining all head features before one softmax instead produces one shared distribution based on the combined score. Nonlinearity prevents generally moving the split through softmax.

Follow-up. Can parameters make them equivalent in special cases? Yes, for example identical head distributions with compatible value/output mappings. That does not establish general equivalence.

Trap. A matched parameter count does not imply identical function classes.

Answer. Attention softmax distributes mass over context keys and creates hidden representations. A language-model head projects final hidden states to vocabulary logits; its loss or sampling softmax distributes mass over possible next tokens. They have different dimensions, meanings, and temperatures.

Follow-up. Where does MoE fit? Often in the feed-forward sublayer, routing tokens to experts rather than routing between token positions.

Trap. Attention probability is not the probability of the generated word or the truth of a source.

Answer. Specify Q[B,Hq,nq,dk], K[B,Hkv,nk,dk], V[B,Hkv,nk,dv], dtype/device, masks, head-group mapping, dropout, and output [B,Hq,nq,dv]. Validate sequence/feature compatibility and nonempty permitted support or a documented empty-row policy.

Follow-up. Why not validate only final shape? Wrong softmax axes and head mappings can return exactly the expected shape while changing semantics.

Trap. A correct-looking output shape is necessary, not sufficient.

Answer. Functional scaled_dot_product_attention uses True for allowed entries. nn.MultiheadAttention Boolean attention and key-padding masks use True for blocked entries. Reusing the same Boolean tensor can therefore invert visibility while all shapes remain valid.

Follow-up. What test catches this? A tiny example with one allowed key and unmistakably different values; the output must equal the allowed value.

Trap. Mask conventions are API-specific, not an industry-wide rule.

Answer. Functional SDPA applies dropout according to dropout_p, independent of a parent module’s training flag. Pass dropout_p=self.p if self.training else 0.0. A separate nn.Dropout module handles its own evaluation state, but a functional call does not infer it.

Follow-up. Is zero dropout sufficient for bitwise reproducibility? Not always. GPU kernels and reduction order can introduce nondeterminism or numeric differences.

Trap. Evaluation mode and gradient disabling are different controls.

Answer. With p previous tokens and m new queries, query row i is at p+i and may read key j≤p+i. Build [m,p+m] permissions from these absolute positions. The checked PyTorch non-square is_causal flag uses upper-left alignment, which is wrong for this appended-chunk layout.

Follow-up. Can one-token decode omit the causal mask? Yes if K/V contain only valid past/current keys, no future or unused slots, and padding is handled.

Trap. A rectangular triangle is meaningless until query/key positions are defined.

Answer. Do not feed an all-negative-infinity row into a naive softmax and hope the loss mask fixes it. Skip that row or temporarily supply a safe support and zero the query output before downstream use; exclude its loss. Real empty-support queries should fail unless another explicit semantic policy exists.

Follow-up. Why not post-hoc multiply NaNs by zero? IEEE arithmetic makes NaN*0 remain NaN, and backward can still propagate invalid values.

Trap. Finite sentinels manufacture distributions over forbidden keys.

Answer. Combine causality with a same-segment permission relation and valid-key masks. Position resets or separator tokens alone do not enforce independence. Loss shifting must also respect example boundaries, so the last token of one example does not predict the first of another unintentionally.

Follow-up. What is a strong test? Change every token in one packed segment and verify outputs and losses of another independent segment remain unchanged.

Trap. A causal mask alone still exposes earlier packed examples.

Answer. Transposing a tensor changes strides, often making it noncontiguous. view requires a compatible memory layout; reshape may allocate a copy. In attention, transpose heads back into [B,T,H,dh] before merging. Copies can be correct but affect memory and latency.

Follow-up. How do you check? Inspect strides/contiguity and profile allocations, then compare semantic outputs with distinct per-head values.

Trap. Fixing a view error by arbitrary reshaping can preserve dimensions while scrambling head/token meaning.

Answer. Use float64 for small numerical gradient checks. For low-precision inputs, accumulate scores/softmax and often AV in float32, while preserving float64 inputs rather than unconditionally downcasting. Check actual autocast behavior so an explicit float cast is not silently undone by operator policy.

Follow-up. How do production kernels differ? They may use mixed internal precision and reordered reductions. Compare with dtype-appropriate tolerances and task-level regression tests.

Trap. Exact mathematical attention does not require bitwise-identical floating-point outputs.

Answer. Test shape and row sums, masked probabilities, causal future perturbation, padding/segment isolation, full-versus-cached parity, head mapping, and gradients. Include rectangular lengths, length one, uneven chunks, and invalid inputs. Use a transparent independent reference rather than comparing two wrappers around the same bug.

Follow-up. What test targets positional bugs? Compare full and incremental execution with nonzero cache offset and the actual rotary configuration.

Trap. Random square tensors alone provide weak coverage.

Answer. Inference does not need training graphs for previous tokens; retaining them wastes memory. An inference-only cache can store detached K/V under no_grad or inference_mode. During training, detaching reused history would truncate gradients and change optimization, so the interface should reject accidental cache use.

Follow-up. Does eval() detach anything? No. It changes behaviors such as dropout, not autograd recording.

Trap. A serving optimization can be a training correctness bug.

Answer. Map each query head to its intended KV group. For Hq=4,Hkv=2 and contiguous groups, indices are [0,0,1,1]. repeat_interleave on the head axis matches that pattern; whole-axis repeat yields [0,1,0,1]. A native kernel can reuse KV without materializing duplicates.

Follow-up. Is either mapping inherently impossible? No, but training and inference must agree on the architecture’s mapping.

Trap. Identical KV heads in tests hide mapping mistakes.

Answer. Multiplying probabilities by a mask removes mass without renormalizing. Allowed weights then sum below one and output scale depends on invalid scores. Mask logits before normalization. Post-mask renormalization is mathematically equivalent for nonempty support in exact arithmetic but can be less stable.

Follow-up. What diagnostic input helps? Append masked keys with huge positive scores and distinctive values; valid outputs should remain unchanged.

Trap. A zero forbidden weight alone does not prove the remaining distribution is correct.

Answer. Backend selection depends on installed version, device, dtype, head dimensions, strides, masks, dropout, and other constraints. A math or memory-efficient fallback can preserve correctness but have different performance. Inspect backend diagnostics and kernel traces.

Follow-up. How do you benchmark fairly? Warm up, synchronize timing appropriately, fix shapes/dtypes, and separate end-to-end overhead from the kernel itself.

Trap. Function names are not evidence of which GPU kernel ran.

Answer. Compare outputs and gradients to a small high-precision reference with identical masks and no stochastic dropout. Check Q/K/V separately and test nontrivial rectangular shapes. For dropout-enabled comparisons, matching random masks or comparing appropriate statistics is necessary.

Follow-up. What if outputs agree but gradients do not? Inspect backward recomputation, softmax scaling, mask derivatives, and dtype conversions rather than accepting forward-only parity.

Trap. Forward parity does not validate training correctness.

Answer. A cache depends on token IDs, positions, model weights/version, adapters, modality representations, and other state that affects hidden representations. Reusing it after a mismatch can silently change predictions. Sharing physical blocks across requests also needs authorization and isolation guarantees.

Follow-up. Does a matching raw prompt string suffice? No. Tokenization or model configuration may differ even for identical text.

Trap. Cache identity is broader than a string hash.

Answer. At batch B, length n, width d, four d×d projections cost 8Bnd² FLOPs when multiply-add counts as two. QKᵀ and AV together cost 4Bn²d at total head width d. This excludes softmax, normalization, FFN, and backward.

Follow-up. What about causality? A kernel that skips masked pairs reduces the quadratic constant; computing then masking the full matrix does not.

Trap. Different FLOP conventions can differ by two; state yours.

Answer. Prefill processes many known queries and has large matrix multiplications plus quadratic pairwise attention. Cached one-token decode computes one new query set and reads a growing history, with per-layer attention work O(td) plus projections O(d²). Decode often becomes memory-bandwidth sensitive, but workload and hardware decide.

Follow-up. Why does batch size matter? Batching changes arithmetic intensity, weight reuse, cache volume, and latency/queueing trade-offs.

Trap. Not every prefill is compute-bound or every decode bandwidth-bound.

Answer. For 32 layers, 8 KV heads, head width 128, 8192 tokens, batch one, and two-byte elements: 2*32*8*128*8192*2 = 1,073,741,824 bytes = 1 GiB. The factor two counts keys and values.

Follow-up. What if query heads are 32? GQA still stores 8 KV heads. Same-width MHA with 32 KV heads would store 4 GiB.

Trap. Use binary GiB and stored KV heads, not an unlabeled GB estimate.

Answer. It omits model weights, optimizer state if training, intermediate activations, temporary workspaces, allocator reservations/fragmentation, cache metadata, alignment, and quantization overhead. Prefix sharing or layer-specific representations can alter physical storage.

Follow-up. How do you choose concurrency? Budget complete peak memory with headroom and measure the serving allocator, rather than divide all GPU memory by logical KV bytes.

Trap. A payload-only maximum is not a safe production concurrency setting.

Answer. MHA stores per-query-head K/V, MQA shares one KV head, and GQA shares K/V within groups. Fewer KV heads reduce K/V projection width and cache payload, while query heads still compute separate distributions. Quality and kernel efficiency are empirical.

Follow-up. Can you convert an MHA checkpoint by reshaping? Not losslessly in general. Averaging/grouping heads changes parameters and often needs uptraining or retraining.

Trap. Reducing KV heads does not proportionally remove all query-head attention arithmetic.

Answer. Multi-head latent attention caches a compressed latent representation from which attention-relevant content is derived, often with separate rotary-position state. Suitable projections can be absorbed into query/output computations to avoid naively expanding every cached key/value at decode. Details depend on the architecture.

Follow-up. How do you calculate its memory? Count the actual latent and positional cache tensors per token/layer, not the standard Hkv*dh formula with an invented head count.

Trap. MLA is not just another spelling of MQA or GQA.

Answer. It tiles exact dense attention to reduce HBM reads/writes and avoid materializing the full score/probability matrix. Online softmax combines block statistics, and backward can recompute intermediates instead of storing them all. Pairwise arithmetic remains quadratic in sequence length.

Follow-up. Is it approximate? It computes the same mathematical operator, with possible floating-point differences from operation order.

Trap. Linear intermediate-memory scaling is not linear arithmetic complexity.

Answer. Maintain maximum m, sum l, and numerator u. For new block scores s, let m_new=max(m,max(s)), alpha=exp(m−m_new), and e=exp(s−m_new). Update l=alpha*l+sum(e), u=alpha*u+sum(e*v); normalize u/l after all blocks.

Follow-up. What about an all-masked tile? Skip it or handle zero prior mass explicitly; do not compute negative infinity minus negative infinity. Entirely empty rows still need policy.

Trap. Averaging independently normalized block outputs gives the wrong weights.

Answer. It manages logical KV blocks using noncontiguous physical storage, improving allocation utilization and enabling controlled sharing. It does not by itself change which logical keys each query attends to or reduce the number of logical KV elements.

Follow-up. Can it coexist with fused exact attention? Yes in compatible serving implementations, because cache layout and attention-kernel data movement solve related but distinct problems.

Trap. Paging is not sparse attention.

Answer. Cache quantization directly perturbs context-specific keys and values. Key error changes scores and softmax weights; value error changes payloads. Effects can compound with context length, outliers, grouping, and task sensitivity. Weight quantization changes a different source of computation.

Follow-up. How would you evaluate? Sweep context length and evidence depth, compare severe-error slices, and account for scales, residual precision, dequantization cost, and memory savings.

Trap. Nominal four-bit payload is not automatically a fourfold end-to-end speedup.

Answer. It admits and removes requests as they progress, improving utilization compared with waiting for a fixed batch to finish. Requests can have different cache lengths and prefill/decode phases, making scheduling, ragged layouts, and memory management important.

Follow-up. What is the trade-off? Larger batches can improve throughput while increasing per-request latency and cache pressure. Measure queueing and p99, not only aggregate tokens per second.

Trap. More batching is not universally better for interactive latency.

Answer. Breaking a long prompt into chunks can reduce peak workspaces and let a scheduler interleave decode work, improving responsiveness under some workloads. Each chunk must attend to the permitted prior prefix and its own causal portion using correct positions.

Follow-up. Does it eliminate quadratic prompt attention? No. Exact attention over all prompt pairs still performs the required total pairwise work, though scheduling and memory peaks change.

Trap. Chunking execution is not the same as truncating the attention window.

Answer. A draft proposes several tokens, and the target evaluates their conditional probabilities in a causal verification pass. Each candidate position sees only the accepted prefix and earlier proposed tokens. An appropriate acceptance/correction procedure can preserve the target sampling distribution.

Follow-up. Why does cache rollback matter? Rejected suffix states must not remain as accepted history. Track logical length and any paged blocks consistently.

Trap. Parallel verification does not permit later draft tokens to leak into earlier conditional scores.

Answer. Reuse requires identical token prefix and compatible model, adapter, position, and modality state. Reference-count shared blocks and apply copy-on-write or equivalent rules for divergence. Enforce tenant/data isolation and consider cache-hit side channels.

Follow-up. Can two prompts sharing a suffix share causal suffix K/V? Not generally: suffix representations depend on their different preceding contexts.

Trap. Matching visible token text somewhere in a prompt is insufficient.

Answer. Tensor parallelism partitions model features or heads across devices and typically communicates projected/output results. Context parallelism partitions sequence positions and requires exchanging K/V or partial attention statistics so queries see their permitted context. Both trade device capacity for communication.

Follow-up. Where does ring attention fit? It circulates KV blocks and combines exact block statistics while overlapping communication and compute where possible.

Trap. Distributing dense attention does not reduce its global pairwise arithmetic order.

Answer. Block sizes, network bandwidth/latency, GPU compute, overlap, topology, sequence length, and balancing determine whether communication is hidden. Longer contexts may provide enough computation per block to amortize transfers; small workloads can be dominated by communication.

Follow-up. How do you validate correctness? Compare global outputs/gradients to a nonpartitioned reference, including causal and boundary masks.

Trap. Near-infinite context in a paper title is not unlimited practical memory or free communication.

Answer. At fixed width/batch, projection work doubles and dense pairwise attention work quadruples. Logical KV memory doubles. A naive stored attention matrix quadruples; fused attention can avoid storing that matrix. Actual runtime depends on which term and hardware bottleneck dominate.

Follow-up. What about one-token decode at twice the retained history? Its pairwise attention work and logical KV reads approximately double, with projections unchanged.

Trap. A single quadratic slogan cannot describe every phase or memory category.

Answer. Sparse indexing, irregular memory access, small matrix blocks, launch overhead, low hardware occupancy, and unsupported layouts can outweigh arithmetic savings. Dense GEMMs exploit highly optimized hardware. Structured block sparsity is often easier to accelerate than arbitrary fine-grained sparsity.

Follow-up. What should you report? End-to-end latency and memory at matched task quality, plus achieved kernel behavior and workload shapes.

Trap. Fewer mathematical operations does not guarantee faster execution.

Answer. Report hardware, software versions, dtype, head dimensions/counts, batch and sequence distributions, mask/cache layout, warmup, synchronization/timing method, selected kernel, peak memory, and correctness tolerance. Separate prefill, decode, and end-to-end request latency.

Follow-up. Why include quality? Approximate/sparse/quantized alternatives can gain speed by changing the model function; compare at an acceptable quality level.

Trap. A best-case square-tensor microbenchmark is not a serving SLA.

Answer. Decompose peak memory into weights, KV cache, prefill activations/workspaces, allocator reserves, and fragmentation. Check whether returning attention matrices forces quadratic storage, whether caches duplicate, and whether a kernel falls back. Then choose a targeted intervention with quality tests.

Follow-up. Why can OOM occur only during prefill? Its activation/workspace peak can exceed the steady decode footprint even when the final KV cache fits.

Trap. A logical memory formula is a starting model, not a full allocator trace.

Answer. It restricts each query to a local neighborhood, reducing pairwise work to roughly O(nwd) for window width w. Distant tokens cannot interact directly in one local layer. Multiple layers propagate information farther, but useful recall remains a learned behavior.

Follow-up. What is the reach bound for L plain causal local layers? Roughly L(w−1) positions backward when w includes the current token, absent global links or dilation.

Trap. Receptive field and effective retrieval quality are different.

Answer. Global tokens can read many positions and be read by local tokens, shortening communication paths across a sparse graph. They may form a bottleneck because large amounts of information must pass through limited representations. Patterns vary in whether global reads/writes are symmetric.

Follow-up. What evaluation catches the bottleneck? Multi-evidence tasks requiring several distant facts, not only one isolated needle.

Trap. A short graph path does not guarantee enough information capacity.

Answer. Replace or approximate affinity by phi(q)ᵀphi(k) with feature width r. Associativity lets you precompute sum phi(k)vᵀ and a normalization vector. Complexity is linear in n when r and value width are treated as fixed; causal recurrence maintains those statistics.

Follow-up. What can grow instead? Feature width needed for approximation quality or state capacity, plus numerical costs.

Trap. It is not generally exact ordinary softmax attention with unchanged finite-dimensional features.

Answer. Maintain S_t=S_(t−1)+phi(k_t)v_tᵀ and z_t=z_(t−1)+phi(k_t). Then o_t=phi(q_t)ᵀS_t / (phi(q_t)ᵀz_t), with an appropriate denominator policy. State size is r*dv+r per head rather than retaining every token.

Follow-up. What is lost? A fixed-size aggregate can make exact token-specific retrieval harder and may suffer interference; the trade-off depends on the feature map and model.

Trap. Constant history-state size is not unlimited exact memory.

Answer. State-space or gated recurrent layers summarize history in evolving state according to their update rules. Softmax attention keeps addressable token-level keys/values and performs query-dependent retrieval. Hybrid models combine mechanisms, often retaining attention in selected layers for retrieval benefits.

Follow-up. Are all recurrent alternatives linear attention? No. Their state transitions, gates, and normalization can be fundamentally different.

Trap. Compare actual equations and empirical task behavior rather than treating all efficient sequence mixers as equivalent.

Answer. Some trained models allocate substantial attention mass to early or special positions even when semantic relevance seems low. Dropping these positions can destabilize certain windowed inference setups. Keeping a small initial prefix plus recent history has empirical support for some models.

Follow-up. Does that preserve all old knowledge? No. Evicted middle tokens are not directly retrievable just because sink tokens remain. See the StreamingLLM source in the bibliography.

Trap. Sinks are an observed mechanism in studied models, not a universal cache-eviction law.

Answer. A studied induction pattern continues a repeated sequence: after seeing A followed by B earlier, a later A can lead attention toward the earlier continuation B. Compositions of heads can implement matching and copying circuits. This offers mechanistic evidence for some forms of in-context pattern completion.

Follow-up. Does it explain all few-shot reasoning? No. Findings in studied architectures do not establish a complete theory of in-context learning in every large model.

Trap. An interpretable circuit is not a universal explanation of model intelligence.

Answer. An architecture may represent a desired routing function in principle yet fail to learn it with available data, optimization, initialization, or width. Saturated scores, redundant heads, bottlenecks, and weak supervision can prevent practical use of theoretical capacity.

Follow-up. How would you investigate? Combine controlled synthetic tasks with real-task ablations, learning curves, and head/score diagnostics; isolate optimization from insufficient information.

Trap. Existence of a parameter setting does not prove SGD will find it.

Answer. If removing a head preserves a measured metric, that head may be redundant under the tested distribution or compensated by other paths. It does not prove it is useless on all inputs. Pruning before retraining and pruning with recovery training answer different questions.

Follow-up. What controls matter? Repeated seeds, rare/error slices, context lengths, ablation location, and retraining budget.

Trap. Low average attention entropy or an attractive heatmap is not a pruning criterion by itself.

Answer. Sharp distributions can reflect accurate retrieval, unwanted saturation, positional shortcuts, or collapsed heads. Diffuse distributions can correctly average multiple relevant values. Entropy must be interpreted with task behavior, value content, and head/layer role.

Follow-up. What metric would you pair with entropy? Targeted retrieval/causal interventions and output-quality slices, not entropy alone.

Trap. Optimizing a diagnostic statistic can damage the actual task.

Answer. Normalizing Q/K can reduce sensitivity to vector norms and stabilize score magnitudes; a learned scale or temperature can restore adjustable sharpness. Depending on the method, it changes the scoring function and its optimization dynamics, not merely an implementation detail.

Follow-up. Does it replace all normalization? No. Residual-stream norms, key-axis softmax, and score scaling have different roles and axes.

Trap. Name the exact normalization rule before claiming equivalence to cosine or LayerNorm.

Answer. Some models use relevant information less effectively when it appears in the middle of a long context than near its ends. Measure task accuracy across evidence insertion depth while controlling total length, distractors, wording, and answer leakage.

Follow-up. Why not test one needle only? Exact copying can miss multi-hop reasoning, conflicting evidence, and aggregate tasks. Include representative application behavior and uncertainty.

Trap. An advertised context window is not measured usable recall.

Answer. Interpolation rescales position indices, compressing angular separation for all affected frequencies. Changing the base changes the frequency spectrum nonuniformly, typically affecting slower frequency bands more. Frequency-aware methods add other choices. Each changes the trained positional geometry.

Follow-up. Which should you choose? Start with the checkpoint’s supported extension recipe and evaluate short/long-context quality; there is no model-independent winner.

Trap. A successful load at a larger context length is not evidence of quality preservation.

Answer. In exact arithmetic with finite unmasked logits, softmax weights are positive, though they may be tiny. Explicit masks, sparse normalization, hard routing, or numerical underflow can create zero weights. Even tiny weights can matter if values have large magnitude.

Follow-up. Can the output ignore a token despite nonzero attention? Yes, value/output projections or downstream cancellation can make its contribution irrelevant to a target prediction.

Trap. Attention weight alone is not the size of the final causal contribution.

Answer. Control model/training budget, data, sequence distribution, hardware/software, and evaluation tasks. Report quality-versus-time/memory curves, not isolated asymptotic claims. Identify whether the operator is exact, approximate, or architecturally different, and include long-range retrieval and short-context regressions.

Follow-up. Why compare at matched quality too? A cheaper operator with worse task quality may need a larger model or more training, reversing the systems advantage.

Trap. An apples-to-apples FLOP count alone is not an apples-to-apples product comparison.

Answer. First identify evidence needs, latency/cost limits, and severe-error constraints. Compare joint token attention with text-to-vision/page cross-attention and controlled evidence compression. Preserve tiny-text/OCR and source references needed for policy decisions; do not optimize only average accuracy.

Follow-up. How do you diagnose a miss? Determine whether evidence was omitted, was present but unused, or was interpreted under the wrong policy rule.

Trap. A high-weight patch or rationale is not verified policy evidence.

Answer. Measure frame sampling, patch count, resolution/crops, token merging, and temporal redundancy. At 448×448 with 14×14 patches, simple patchification yields 1024 tokens/frame before merging. Ten frames produce 10,240 visual tokens, potentially dominating dense interactions.

Follow-up. What is the quality risk? Aggressive sampling can miss brief defects; spatial reduction can erase small text. Evaluate event duration and text-size slices.

Trap. Token count estimates must reflect actual preprocessing, not a model family’s marketing name.

Answer. Sixteen is the arithmetic payload-only upper bound, not a safe recommendation. Check uneven context lengths, growth during generation, allocator blocks, metadata, peak workspace, and headroom. Set admission control from measured complete memory and latency behavior.

Follow-up. How would you raise capacity? Evaluate compatible GQA/MLA models, KV quantization, prefix sharing, and storage management with quality/isolation gates.

Trap. Never convert a memory upper bound directly into an SLA-backed concurrency target.

Answer. Confirm the kernel actually runs, then separate prefill, decode, queueing, transfers, and other model operations. Check shape/dtype/mask changes, fallback paths, warmup, memory copies, and batch policy. Compare to a pinned baseline with correctness parity.

Follow-up. What if kernel time improved but end-to-end worsened? Look for changed batching/queueing, extra conversions, and bottlenecks shifted to other stages.

Trap. A local speedup does not guarantee a request-level speedup.

Answer. Reproduce on a fixed dataset and compare full-precision versus quantized outputs with identical prompts and model weights. Slice by context depth/length, OCR size, language, rare labels, and key/value outliers. Inspect score and output deviations without treating attention maps as ground truth.

Follow-up. What is a sensible rollback gate? A pre-agreed severe-error tolerance with uncertainty, plus latency/memory targets, not an unquantified claim of negligible quality loss.

Trap. Average agreement can hide rare but important regressions.

Answer. No. It controls which token positions are visible, not which text has authority. A hostile instruction inside visible content can still influence the model. Keep trusted instructions separate, treat retrieved material as evidence, constrain outputs/tools, and test injection attacks.

Follow-up. Would masking all retrieved text solve it? It also removes required evidence. The problem needs trust-aware system design, not a blanket attention prohibition.

Trap. Visibility, salience, and authority are different concepts.

Answer. Use controlled data splits, a matched training/fine-tuning budget, comparable token budgets, and multiple seeds where feasible. Evaluate task accuracy and rare severe errors with uncertainty, plus throughput, latency, and memory. Include short and long inputs and deployment-relevant distributions.

Follow-up. What about attention-head visualizations? Use them to form hypotheses, then test behavior or interventions. They are not acceptance criteria.

Trap. A single attractive benchmark or head map is insufficient evidence.

Answer. Fixed encoder outputs can have their projected cross-attention K/V cached per decoder layer where applicable. Decoder self-attention K/V grows with generated tokens. Cache visual or page encoder features only when inputs and preprocessing/model versions match.

Follow-up. Can cross-attention K/V be shared across decoder layers automatically? No. Different layers can have different projection weights, so derived K/V differ unless the architecture explicitly shares them.

Trap. Cache reuse must follow the computation graph, not just matching tensor dimensions.

Answer. Choose based on where necessary evidence lives and what interactions the task needs. Retrieval can reduce irrelevant tokens but may omit distributed or subtle evidence. Larger context retains more information at higher cost and still needs usable-context validation. Truncation is simplest but risks systematic omission.

Follow-up. What experiment decides? Measure evidence coverage and end-task quality separately, then compare latency/cost at acceptable severe-error rates.

Trap. More context cannot recover evidence that preprocessing never supplied.

Answer. Start with correctness parity and phase-specific profiling. For unchanged dense-model semantics, use compatible exact fused kernels and remove avoidable copies/materialized attention matrices. If KV capacity/bandwidth remains limiting, evaluate cache layout and supported lower-memory model/cache options behind quality gates.

Follow-up. What makes the recommendation senior-level? It names the bottleneck, quantified assumptions, chosen trade-off, falsifiable tests, and rollback criteria rather than listing acronyms.

Trap. Do not propose an architectural replacement before measuring the existing bottleneck.