Calculator Inputs
How to Use This Calculator
- Enter rectangular query, key, and value matrices.
- Match query and key feature counts exactly.
- Match key and value row counts exactly.
- Select a head count dividing every feature width.
- Choose scaling, temperature, masking, clipping, and dropout options.
- Add readable labels for clearer heatmap interpretation.
- Submit the form and inspect results above.
- Compare every head using the displayed selector.
- Download matrices or save the attention heatmap.
Understanding Attention Matrix Calculations
Attention Foundations
Attention matrices describe how strongly each query relates to available keys. Every row usually represents one query token or feature. Every column represents a key receiving measurable attention weight.
Raw attention scores come from dot products between query and key vectors. Larger dot products indicate stronger directional similarity within the embedding space. Negative scores can still receive probability after softmax normalization.
Formula Used
Scaled scores: Z = clip(S ÷ T) + M
Attention: A = softmax(Z)
Context: C = AV
Standard divisor: d = √dk
Scaled dot-product attention divides scores by the square root of dimension. This scaling prevents large vector dimensions from creating extreme logits. Stable logits produce smoother gradients during model training and evaluation.
Softmax converts adjusted scores into nonnegative probabilities across the chosen axis. Row normalization is standard because each query distributes attention across keys. Column normalization supports specialized analysis but changes probability interpretation.
Masks and Numerical Controls
Causal masks prevent queries from viewing keys located later in sequences. Custom masks can block padding, unavailable features, or forbidden relationships. Masked positions receive zero probability after the normalization step.
Temperature controls how sharply probabilities concentrate around the strongest scores. Lower temperatures create focused patterns with larger peaks. Higher temperatures spread attention across more keys and relationships.
Logit clipping limits extreme scores before probabilities are calculated. It can reveal patterns hidden by one dominant relationship. Excessive clipping may flatten meaningful distinctions between closely related tokens.
Multi-Head Diagnostics
Multi-head attention separates feature dimensions into independent representation subspaces. Different heads can capture syntax, position, identity, or semantic similarity. Averaging heads gives a useful summary but hides specialized behavior.
Entropy measures uncertainty within each attention distribution. Low entropy suggests concentrated attention on few keys. High entropy indicates broader allocation across several plausible relationships.
Peak weight identifies the strongest single connection within a head. Sparsity measures how many probabilities approach zero after processing. Diagonal mass reveals self-focused behavior when query and key lengths match.
Context and Input Preparation
Context vectors combine value rows using calculated attention probabilities. Each output row summarizes information selected for its corresponding query. Concatenated head contexts preserve separate learned perspectives for later layers.
Begin with rectangular numeric matrices using consistent row widths. Match key and value row counts before submitting calculations. Choose head counts dividing query, key, and value feature dimensions.
Reading the Graphs
Add labels to make heatmaps easier to interpret. Apply causal or custom masks only when relationships require restrictions. Compare scores, probabilities, entropy, and generated context outputs together.
Strong heatmap bands reveal positional or semantic relationships. Isolated peaks may indicate decisive evidence or dominance. Uniform regions can signal uncertainty, saturation, or insufficient distinction.
Frequently Asked Questions
1. What does an attention matrix represent?
It shows relationships between query and key positions. Each cell contains one calculated attention weight. Larger values indicate stronger influence during context construction.
2. Why divide scores by square root dimension?
Large dimensions can produce excessive dot-product magnitudes. Scaling keeps logits within a more stable range. Softmax then avoids becoming unnecessarily sharp or saturated.
3. What dimensions must match?
Query and key feature counts must match. Key and value row counts must also match. Selected heads must divide both feature widths evenly.
4. What does temperature change?
Temperature changes probability concentration before softmax normalization. Lower values sharpen strong relationships significantly. Higher values distribute attention more evenly across keys.
5. When should causal masking be used?
Use causal masking for left-to-right sequence generation. Future key positions become unavailable to earlier queries. This prevents information leakage from later sequence positions.
6. What does attention entropy indicate?
Entropy estimates uncertainty inside each attention distribution. Lower entropy means attention is more concentrated. Higher entropy suggests several keys receive meaningful probability.
7. Why compare separate attention heads?
Different heads inspect different feature subspaces. One head may emphasize position or identity. Another head may emphasize broader semantic relationships.
8. What does attention dropout simulate?
Dropout removes selected attention links during simulation. This tests sensitivity to missing connections. Seeded randomness makes repeated comparisons reproducible.
9. How are context vectors calculated?
Attention probabilities weight corresponding value rows. Weighted rows are summed for every query. Multiple head outputs are concatenated into one context matrix.