Configure The Grid Model
Formula Used
V(s) = maxa Σs′ P(s′|s,a)[R(s,a,s′) + γV(s′)]
The value represents the best expected discounted return. Transition probabilities include intended and slipped movements. Terminal states return their defined reward without additional continuation.
How To Use This Calculator
- Set grid dimensions and the discount factor.
- Enter terminal states, obstacles, and custom rewards.
- Choose movement, boundary, initialization, and update settings.
- Select heatmap labels and the preferred color scale.
- Press the button and review results above the form.
Value Function Heatmap Guide
Understanding Value Function Heatmaps
Value functions estimate future return from every reachable environment state. Heatmaps convert those estimates into clear spatial patterns for analysis. Bright regions usually indicate states with stronger long-term opportunities.
Why Grid Models Matter
Grid environments simplify reinforcement learning without removing important decision dynamics. Each cell represents a state, reward, obstacle, or terminal outcome. That structure makes policy behavior easier to inspect and explain.
Core Bellman Update
The calculator applies Bellman optimality updates across every valid state. Each action considers immediate rewards and discounted future state values. The best expected action becomes the state's updated value.
Discount Factor Effects
Gamma controls how strongly future rewards influence current state estimates. Low gamma values favor immediate gains and shorter decision paths. High gamma values preserve distant rewards across longer planning horizons.
Transition Noise
Real agents may not execute every selected action perfectly. Slip probability distributes movement across unintended alternative actions. More noise can flatten values and change preferred routes.
Rewards And Terminals
Living rewards shape movement before the agent reaches a terminal. Positive rewards encourage exploration, while penalties encourage faster completion. Terminal rewards define final outcomes and stop future accumulation.
Obstacles And Boundaries
Obstacles remove states and redirect attempted movement toward existing positions. Fixed boundaries also keep agents inside the defined grid. Wrapped boundaries instead connect opposite edges for continuous movement.
Convergence Diagnostics
Residual change measures the largest update between consecutive value sweeps. Small residuals indicate that additional updates offer limited improvement. The iteration count reveals how quickly settings reach stable estimates.
Comparing Update Modes
Synchronous updates use values from the previous complete iteration. In-place updates reuse newly calculated values during the same sweep. Both approaches converge under common discounted finite-state conditions.
Practical Experiment Design
Start with simple rewards before adding noise and custom states. Change one parameter per experiment to preserve meaningful comparisons. Record convergence metrics alongside visible policy changes for interpretation.
Interpreting Negative Values
Negative values do not automatically indicate a failed learning configuration. They may reflect unavoidable costs before reaching useful terminal rewards. Compare relative values and chosen actions before judging performance.
Using Custom Rewards
State-specific rewards model hazards, bonuses, resources, or temporary objectives. Place them carefully because repeated visits can strongly dominate long-term return. Discounting and movement noise determine how widely their influence spreads.
Policy Verification
Inspect arrows near obstacles, corners, and competing terminal states carefully. Unexpected directions often reveal reward formatting or transition assumptions. Recheck coordinates whenever the visual policy contradicts intended behavior.
Reliable Conclusions
A heatmap summarizes one model under one selected parameter configuration. It does not replace testing with trajectories and sampled episodes. Repeat experiments before selecting a final reinforcement learning design.
Frequently Asked Questions
What does a value function represent?
It estimates expected discounted return from a particular state. Higher values usually indicate stronger long-term outcomes. Rewards and transition probabilities determine every estimate.
What is the discount factor?
Gamma controls the importance of future rewards. Smaller values emphasize immediate outcomes. Larger values preserve distant consequences during planning.
How does slip probability work?
Slip models uncertain action execution. Probability moves from the chosen action toward alternatives. Higher slip values create less predictable policies.
Why are obstacle cells blank?
Obstacle cells are excluded from value updates. Attempted movement into obstacles leaves the agent stationary. Blank heatmap cells distinguish blocked locations clearly.
What causes slow convergence?
High gamma values can slow value propagation. Tight tolerances also require more update sweeps. Large grids and noisy transitions add further computation.
Which update mode should I choose?
Synchronous updates are easier to compare mathematically. In-place updates often converge using fewer sweeps. Both should approach similar discounted solutions.
Can state values be negative?
Yes, penalties can produce negative expected returns. Negative values may still define the best available path. Compare states relatively before changing rewards.
What do policy arrows show?
Each arrow shows the highest-valued action for one state. Ties use the first matching action. Noise remains included during action evaluation.
How should I validate the result?
Check rewards, coordinates, and boundary behavior first. Inspect arrows around terminals and obstacles next. Compare repeated runs before finalizing any important policy design.