Single Q-Value Update
Batch Transition Calculator
Use action labels to identify maximum next actions.| State | Action | Current Q | Reward | Next state | Next Q-values | α | γ | Terminal | Updated Q | TD error |
|---|
Editable Q-Table Update
Episode Update Simulator
Formula Used
Qnew(s,a) = Qold(s,a) + α[r + γ max Q(s′,a′) − Qold(s,a)]
TD target = r + γ max Q(s′,a′)
TD error = TD target − Qold(s,a)
How to Use
Enter the current value, reward, learning rate, and discount factor. Add all available next-state action values. Select terminal status when an episode ends there.
Calculate the update and inspect every intermediate quantity. Compare the old value with the new estimate. Use batch mode for several transitions together.
Edit the Q-table for state-action experiments. Apply transitions sequentially to observe value propagation. Export results for later model analysis work.
Example Data
| Scenario | Qold | Reward | α | γ | Next Q-values | Terminal |
|---|---|---|---|---|---|---|
| Grid-world step | 2.5 | 5 | 0.1 | 0.9 | 3.2, 4.0, 2.8 | No |
| Penalty transition | 1.5 | -4 | 0.25 | 0.8 | 1.2, 0.5, -0.2 | No |
| Goal state | 3 | 10 | 0.5 | 0.95 | 0 | Yes |
| Optimistic estimate | 8 | 1 | 0.15 | 0.9 | 4, 5, 3 | No |
Interpretation Guide
A positive TD error raises the selected action value. A negative TD error lowers an optimistic estimate. Zero error leaves the value unchanged.
Larger learning rates produce faster value movement. Smaller rates create smoother and slower adaptation. Discounting controls attention to future rewards.
Terminal transitions normally remove future-state value. Reward clipping can stabilize unusually large reward scales. Ties reveal equally valued next actions.
Common Mistakes
- Using the chosen next action instead of the maximum next action.
- Applying a future Q-value after a terminal transition.
- Entering learning rates or discount factors outside zero and one.
- Mixing normalized and unnormalized rewards without checking scale.
- Updating the wrong state-action cell inside a Q-table.
- Confusing Q-learning with SARSA's on-policy next-action update.
Merged Result Text
Calculate a single update, batch transitions, or a Q-table change to populate this reusable summary.
Frequently Asked Questions
What does Q-learning update?
It updates one state-action value using a reward. The update also considers the best next action. Repeated updates estimate an optimal action policy.
Why use the maximum next Q-value?
Q-learning is an off-policy control method. It learns from the best estimated future action. Behavior may still explore different actions during training.
What happens when alpha equals one?
The old estimate receives no direct weight. The new value becomes the current TD target. Such updates can react strongly to noisy samples.
What happens when gamma equals zero?
The agent ignores all estimated future rewards. Only the immediate reward shapes the target. This creates completely myopic value updates.
How are terminal states handled?
Terminal states usually have no future action value. Their next-state contribution therefore becomes zero. A custom terminal value remains available here.
Can rewards be negative?
Yes, penalties are commonly represented using negative rewards. They can reduce the selected action value. The calculator accepts floating-point negative numbers.
What is reward clipping?
Reward clipping limits rewards to a chosen interval. It can reduce sensitivity to extreme magnitudes. Clipping also changes the effective learning signal.
How does Q-learning differ from SARSA?
Q-learning uses the maximum next action value. SARSA uses the action actually selected next. Their targets can differ during exploratory behavior.
Do Q-values always converge?
Convergence requires suitable exploration and learning conditions. Every state-action pair needs adequate repeated visits. Function approximation can introduce additional instability risks.