Calculated Results
Results appear here after calculation.
Greedy Policy
| State | Best action | Best Q | Gap |
|---|
Interpretation
Comparison Summary
| State | Current policy | Comparison policy | Changed |
|---|
Bellman Update Result
Formula Used
The Q-learning update combines immediate reward with the best discounted future action value. The temporal-difference error measures the difference between the target and current estimate. Repeated updates gradually move the estimate toward the target.
Q(s,a) ← Q(s,a) + α[r + γ max Q(s′,a′) − Q(s,a)] TD error = r + γ max Q(s′,a′) − Q(s,a)
SARSA uses the next action actually selected instead of the maximum action value. Terminal transitions use no discounted future value. This distinction changes exploration behavior during learning.
How to Use
- Enter state, action, Q-value, and optional episode rows.
- Select a plot, policy method, episode, and display settings.
- Optionally provide comparison data and Bellman update inputs.
- Choose grid mode when states represent grid cells.
- Calculate, inspect the policy, then export desired results.
Example Data
| State | Action | Q-value | Episode | Meaning |
|---|---|---|---|---|
| S1 | Right | 2.80 | 1 | Estimated return for moving right. |
| S2 | Up | 3.40 | 1 | Highest action value at state S2. |
| S3 | Down | 2.20 | 1 | Tied with another optimal action. |
Frequently Asked Questions
What is a Q-value?
A Q-value estimates expected discounted return after taking one action. It depends on both state and action. Larger values usually indicate stronger choices.
How is the greedy policy found?
The calculator selects the largest Q-value for each state. Equal maximum values create tied actions. Ties are displayed together.
Why can Q-values be negative?
Negative values can represent penalties, costs, or undesirable future outcomes. Their scale depends on rewards. Comparisons remain meaningful within one environment.
What does the policy gap mean?
The gap compares the best and second-best action values. A small gap suggests uncertainty. A large gap suggests a clearer choice.
What does epsilon-greedy mean?
Epsilon-greedy usually selects the best action. It occasionally explores alternatives. Epsilon controls that exploration probability.
How does softmax policy selection work?
Softmax converts Q-values into action probabilities. Temperature controls probability sharpness. Lower temperatures favor the highest value.
What does convergence show?
Convergence plots track values across episodes. Stable lines indicate limited recent change. Stability alone does not guarantee optimality.
When should SARSA be used?
SARSA learns from the action actually chosen. It reflects exploratory behavior during updates. This can produce safer learned policies.
Can different agents be compared?
Yes, paste a second compatible dataset. The calculator compares values and policies. Changed optimal actions are clearly marked.