Attention Heatmap Results
Results appear here after calculation.
Top Attention Pairs
| Rank | Query | Key | Weight |
|---|
Token Summary
| Query | Top Key | Max | Entropy |
|---|
Selected Cell or Token Details
Processed Matrix
Formula Used
Row normalization divides each attention value by its row sum. This makes each query distribution total one. Percent mode then multiplies normalized values by one hundred.
Attention entropy measures how broadly each query distributes attention. Lower entropy indicates focused attention across fewer key tokens. Higher entropy indicates a more distributed attention pattern.
Difference modes compare corresponding cells between two matrices. Absolute difference removes direction while preserving change magnitude. Percentage change uses the comparison matrix as baseline.
How to Use
- Paste an attention matrix or load an example.
- Add matching query and key token labels.
- Select normalization, transformation, masking, and filtering settings.
- Choose colours, annotations, dimensions, and comparison options.
- Select Calculate Heatmap and inspect the generated analysis.
- Export the graph, matrix, statistics, or complete report.
Example Data
| Query | [CLS] | The | cat | sat | [SEP] |
|---|---|---|---|---|---|
| [CLS] | 0.40 | 0.20 | 0.15 | 0.15 | 0.10 |
| The | 0.10 | 0.35 | 0.30 | 0.15 | 0.10 |
| cat | 0.05 | 0.20 | 0.45 | 0.20 | 0.10 |
| sat | 0.05 | 0.10 | 0.30 | 0.45 | 0.10 |
| [SEP] | 0.10 | 0.10 | 0.15 | 0.20 | 0.45 |
Understanding Attention Heatmaps
Rows usually represent queries and columns represent keys. Darker cells commonly indicate stronger token relationships. Interpretation depends on the chosen colour scale.
Self-attention compares tokens within the same sequence. Cross-attention links decoder queries with encoder key tokens. Causal attention hides future positions during generation.
Attention weights are useful diagnostic signals, not complete explanations. High attention does not always prove causal importance. Compare multiple heads, layers, and attribution methods.
Frequently Asked Questions
What does each heatmap cell represent?
Each cell shows one query token's weight for one key token.
Should every attention row sum to one?
Transformer attention commonly uses row softmax, making rows total one.
What is attention entropy?
Entropy measures whether attention is concentrated or broadly distributed.
Can I compare two attention heads?
Yes. Paste both matrices and select a difference output.
What does a causal mask do?
It prevents each position from viewing later sequence positions.
How are padding tokens detected?
Labels matching PAD patterns are removed or masked automatically.
Can I average several heads?
Yes. Provide JSON matrices and choose an aggregation method.
Why can attention contain negative values?
Raw scores or difference matrices can contain negative values.
Does attention explain model decisions completely?
No. Combine attention with gradients, ablations, and other attribution methods.