Configure The Exploration Schedule
Choose a decay model, training horizon, and comparison schedule.
Formula Used
The calculator clamps every epsilon value between configured limits.
Linear: ε(t) = max(εmin, ε0 − (ε0 − εmin)t / D)
Exponential: ε(t) = εmin + (ε0 − εmin)e^(−kt)
Inverse time: ε(t) = εmin + (ε0 − εmin) / (1 + kt)
Cosine: ε(t) = εmin + 0.5(ε0 − εmin)(1 + cos(πt / D))
Polynomial: ε(t) = εmin + (ε0 − εmin)(1 − t / D)^p
Staircase: ε(t) = max(εmin, ε0f^floor(t / S))
Here, ε0 is starting epsilon. εmin is the minimum rate. Values D, k, p, f, and S control decay behavior.
How To Use This Calculator
- Enter starting and minimum epsilon values.
- Select a primary exploration schedule.
- Add a comparison schedule when needed.
- Set episodes, warmup, and decay controls.
- Choose a threshold and action estimate.
- Press Calculate Curve to generate results.
- Review metrics, milestones, and chart behavior.
- Export CSV values or save the chart.
Understanding Exploration Rate Curves
Exploration And Exploitation
Exploration controls how often an agent tests unfamiliar actions during training. Higher rates encourage discovery, while lower rates favor learned rewards. A planned curve balances curiosity with reliable exploitation over time.
The starting rate usually begins near one for broad action sampling. The minimum rate preserves occasional discovery after learning becomes more stable. Episodes define how long the selected schedule remains active.
Linear And Exponential Decay
Linear decay reduces exploration by an equal amount each episode. This pattern is predictable, transparent, and easy to explain. It works well when learning progress follows a steady pace.
Exponential decay changes quickly early, then slows near the minimum. It suits agents gaining useful policies during early training. Careful rates prevent exploration from disappearing before difficult states appear.
Long Tail Schedules
Inverse time decay falls sharply, yet keeps a long exploration tail. The tail supports continued discovery across extended or changing environments. This schedule often helps when rewards arrive late or sparsely.
Cosine decay creates a smooth transition with gentle endpoints. Its gradual shape avoids abrupt behavioral changes during important training phases. The curve can support stable tuning in sensitive learning systems.
Shape Controls
Polynomial decay adds a power value controlling the curve shape. Larger powers retain exploration longer before a stronger final decline. Smaller powers create faster reductions during the early episodes.
Staircase decay holds rates, then reduces them at fixed intervals. Stable plateaus allow performance measurement before each exploration adjustment. However, large drops may temporarily disturb learned action patterns.
Warmup And Measurement
Warmup episodes keep the starting rate unchanged during initial experience collection. This feature helps populate replay memory with varied transitions. Excessive warmup can waste episodes after useful behavior emerges.
Area under the curve summarizes total exploration exposure across training. Average epsilon shows the typical random-action probability for all episodes. Threshold timing reveals when the agent becomes mostly exploitative.
Comparing Curves
Compare several schedules before launching expensive reinforcement learning experiments. Inspect early, middle, and final rates for unexpected behavior. Exported values can support reports, reviews, and reproducible configuration files.
No single exploration curve fits every environment or reward structure. Test curves against learning stability, return quality, and state coverage. Strong decisions combine visual evidence with repeated controlled training runs.
Evaluation Discipline
Track random actions separately from greedy actions during evaluation. Reward trends should improve as exploration gradually declines. State coverage can expose premature convergence.
Use several random seeds for fair schedule comparisons. Record environment changes and training budgets. Reliable exploration planning strengthens both performance and reproducibility results.
Frequently Asked Questions
What does epsilon represent?
Epsilon represents the probability of selecting a random action. A larger value increases exploration. A smaller value favors the best known action.
Which decay schedule is best?
No schedule is universally best. Linear decay offers predictable control. Exponential and inverse schedules suit faster early learning or longer exploration tails.
Why keep a minimum exploration rate?
A minimum rate preserves occasional discovery after convergence. It helps agents react to rare states. It also supports adaptation within changing environments.
What does the decay duration control?
Decay duration controls when bounded schedules reach minimum epsilon. Larger durations preserve exploration longer. Smaller durations shift behavior toward exploitation earlier.
How is curve area interpreted?
Curve area summarizes accumulated epsilon across episodes. Larger areas indicate more exploration exposure. Compare areas only when training horizons match.
Why add warmup episodes?
Warmup keeps exploration high during early experience gathering. This can diversify replay memory. Too much warmup may delay policy improvement.
What is the target threshold?
The target threshold marks a meaningful epsilon level. The result reports its first reached episode. Missing results mean the curve never crossed it.
Are expected action counts exact?
They are mathematical expectations, not guaranteed counts. Actual random choices vary by sampling. Longer runs usually approach the expected proportions.
Can comparison curves share parameters?
Yes. Both curves use identical limits and control values. Shared parameters make schedule shape differences easier to inspect.