Diagnostics
- Generate or import a policy to see diagnostics.
Policy matrix
No policy calculated yet.
Calculated state results
| State | Row | Column | Selected action | Action probability | Maximum Q-value | State value | Reward | Entropy | Confidence | Status | Previous action | Changed |
|---|
- Choose grid dimensions and the available action set.
- Select a policy rule, exploration rate, and temperature.
- Build the grid, then edit rewards, values, statuses, Q-values, and probabilities.
- Select the heatmap metric and display options.
- Calculate the policy to view arrows, diagnostics, tables, and summary metrics.
- Snapshot a baseline to compare action changes and agreement.
- Use evolution controls to apply repeated policy-improvement steps.
- Export the report, chart, CSV, JSON, or PDF.
| State | Reward | Status | Q-values: Up, Right, Down, Left |
|---|---|---|---|
| S0 | -0.04 | Start | 0.20, 0.65, 0.10, 0.05 |
| S3 | 1.00 | Goal | 0, 0, 0, 0 |
| S5 | 0 | Obstacle | 0, 0, 0, 0 |
| S7 | -1.00 | Terminal | 0, 0, 0, 0 |
What does a policy heatmap show?
It shows the preferred action for each environment state. Colors can represent Q-values, rewards, entropy, confidence, values, visits, or policy changes.
How are greedy actions selected?
The calculator selects actions having the largest Q-value. Ties can show every optimal action, choose the first, or choose randomly.
What is policy confidence?
Confidence is the largest action probability within a state. Higher confidence indicates a more decisive policy distribution.
What does high policy entropy mean?
High entropy means action probabilities are broadly distributed. The policy is more uncertain or exploratory in that state.
Can stochastic policies be displayed?
Yes. Enter action probabilities or use epsilon-greedy and softmax conversion. Secondary actions can appear above a probability threshold.
How does the baseline comparison work?
A baseline stores earlier selected actions. The calculator then reports changed states, agreement, and a policy difference heatmap.
What does the evolution step calculate?
It applies a one-step reward and discounted-value update. This demonstrates repeated policy improvement within the configured grid.
How are obstacles handled?
Obstacle cells are excluded from policy calculations and rendering values. Movement into obstacles follows the selected boundary behavior.
Can I import policies from another tool?
Yes. Import JSON or CSV containing states, rewards, values, visits, statuses, Q-values, and action probabilities.
Which reinforcement-learning methods are supported?
The visualization accepts policies from Q-learning, SARSA, policy iteration, value iteration, Monte Carlo, actor-critic, and custom methods.