Simulation Results
Algorithm comparison
| Rank | Algorithm | Total reward | Average reward | Cumulative regret | Average regret | Best-arm rate | Exploration | Exploitation | Reward SD | 95% CI |
|---|
Cumulative reward
Cumulative regret
Average reward
Arm selection frequency
Estimated-value convergence
Exploration versus exploitation
Arm statistics
| Algorithm | Arm | Selections | Selection % | Estimated value | True expected reward | Total observed reward |
|---|
Trial-by-trial decision log
The table shows the first 500 logged decisions from the leading simulation run.
| Trial | Algorithm | Selected arm | Reward | Cumulative reward | Cumulative regret | Mode | Selection explanation |
|---|
Export and utility tools
Formula used
Qt+1(a) = Qt(a) + [Rt − Qt(a)] / Nt(a)
Choose a random arm with probability ε. Otherwise choose argmax Q(a).
UCB(a) = Q(a) + c × √(ln t / N(a))
Regret(T) = Σt=1T[μ*(t) − μAₜ(t)]
P(a) = exp(Q(a)/τ) / Σ exp(Q(b)/τ)
Draw θa ~ Beta(αa, βa) and select the largest draw.
How to use this calculator
- Choose a stationary, changing, or adversarial environment.
- Configure every arm and its reward distribution.
- Select one or more bandit algorithms.
- Adjust exploration, confidence, prior, and update parameters.
- Run simulations and compare rewards, regret, and convergence.
- Export tables, charts, settings, or a printable report.
Example data
| Arm | Distribution | Parameter 1 | Parameter 2 | Interpretation |
|---|---|---|---|---|
| Arm A | Bernoulli | 0.30 | 0 | Thirty percent success probability. |
| Arm B | Bernoulli | 0.50 | 0 | Fifty percent success probability. |
| Arm C | Bernoulli | 0.70 | 0 | Seventy percent success probability. |
Algorithm guidance
Epsilon-greedy is simple and dependable. It explores randomly at a controlled rate. Decay helps reduce unnecessary exploration later.
UCB methods favor uncertain arms systematically. They work well with stationary rewards. Their confidence coefficient controls optimism.
Thompson Sampling uses probability distributions over arm quality. It often balances exploration efficiently. Binary rewards fit Beta priors naturally.
EXP3 targets adversarial or highly unstable rewards. It updates action weights using importance-weighted feedback. Gamma controls its exploration intensity.
Frequently asked questions
What is a multi-armed bandit problem?
It models repeated choices among uncertain actions. Each action provides an unknown reward. The goal is maximizing long-term reward.
What is exploration versus exploitation?
Exploration tests less-known arms. Exploitation chooses the strongest current estimate. Effective policies balance both behaviors.
What does cumulative regret measure?
Regret compares chosen rewards with optimal expected rewards. Lower regret indicates better decisions. It is central for policy comparison.
When should epsilon decay?
Decay is useful when rewards remain stable. It reduces random choices over time. Non-stationary problems may need continued exploration.
How does UCB choose an arm?
UCB combines estimated reward with uncertainty. Rarely selected arms receive larger bonuses. This encourages structured exploration.
Why use Thompson Sampling?
It samples plausible arm qualities from posteriors. Uncertain promising arms get natural opportunities. Performance is often strong.
Can rewards be continuous?
Yes, normal, uniform, and Poisson rewards are supported. Estimates update using observed numerical rewards. Thompson values are normalized internally.
What is a non-stationary bandit?
Its reward behavior changes over time. Older observations may become less relevant. Constant learning rates can adapt faster.
Why run multiple simulations?
Single runs contain random variation. Repeated runs produce more stable averages. Confidence intervals become more meaningful.
Which strategy is always best?
No strategy wins in every environment. Reward stability and noise matter greatly. Compare several algorithms using identical seeds.