Average Reward Curve Calculator

Formula Used

How to Use This Calculator

Enter one reward run per line. Use equal environments and consistent evaluation rules. Then select smoothing, uncertainty, target, and display settings.

Press the calculation button to build the curve. Review metrics before interpreting individual reward spikes. Export the table when further analysis is required.

Understanding Average Reward Curves

Average reward curves summarize how an agent performs across repeated episodes. Each point represents the mean reward from one or several runs. Rising curves usually indicate improving decisions and stronger learned behavior.

These curves help compare policies under identical environments and reward rules. They also expose instability, regressions, and unusually noisy training phases. Reliable comparisons require equal episode counts and consistent evaluation settings.

Reading Raw and Smoothed Rewards

Raw rewards preserve every spike, drop, and temporary learning failure. However, dense fluctuations can hide the broader direction of progress. Smoothing reveals trends while retaining the original series for context.

A simple moving average uses an equally weighted rolling window. Larger windows create steadier curves but delay visible turning points. Smaller windows respond quickly but preserve more random variation.

An exponential moving average gives greater weight to recent observations. Its alpha value controls responsiveness to new rewards. Higher alpha values react faster and produce less historical smoothing.

Comparing Multiple Training Runs

Multiple runs provide stronger evidence than one lucky training sequence. The calculator averages aligned episodes across all supplied runs. Confidence bands estimate uncertainty around each episode's mean reward.

Narrow bands suggest repeatable behavior across seeds or evaluation runs. Wide bands indicate unstable learning or strongly variable environment outcomes. More independent runs usually improve confidence in observed performance.

Using Targets and Trend Measures

The baseline line represents a target, heuristic, or previous policy score. Crossing that line shows when average performance meets expectations. Sustained crossings matter more than brief reward spikes.

The trend slope measures average directional change across the selected episodes. Positive slopes suggest improvement, while negative slopes suggest declining behavior. Near-zero slopes may indicate convergence, stagnation, or balanced noise.

Area under the reward curve summarizes total performance across training. Higher areas generally reflect stronger rewards over more episodes. Comparisons remain meaningful only when episode ranges remain identical.

Detecting Convergence

Convergence detection checks whether smoothed rewards remain near the final level. The tolerance controls how much deviation remains acceptable. A stricter tolerance delays convergence or may prevent detection.

Practical Workflow

Enter each training run on a separate line. Separate reward values using commas, spaces, or semicolons. Choose smoothing, confidence, clipping, and aggregation settings before calculating.

Review summary cards before interpreting the interactive graph. Toggle traces to compare raw, smoothed, cumulative, and baseline patterns. Exported CSV data supports reports, experiments, and later model reviews.

Responsible Interpretation

Reward curves should support decisions rather than replace careful evaluation. Inspect safety, fairness, sample efficiency, and task success separately. Average rewards can still hide rare but serious failures.