Adam Optimiser Calculator

Calculation Results

Ready

Enter values and select Calculate Adam Update.

New parameter value

—

Update amount

—

Selected method

Adam

First moment mₜ

—

Second moment vₜ

—

Corrected m̂ₜ

—

Corrected v̂ₜ

—

Effective learning rate

—

Gradient magnitude

—

Update direction

—

Parameter change

—

Calculation steps will appear here.

Iteration	Previous θ	Gradient	mₜ	vₜ	m̂ₜ	v̂ₜ	Update	New θ	Effective step
No simulation data yet.

Optimiser	Update	New parameter	Magnitude	State	Observation
No comparison data yet.

No detailed output yet.

Input Parameters

Calculation mode

Optimiser method

Decimal precision

Number format

Initial parameter value θ

Current gradient gₜ

Learning rate α

Iteration number t

Beta 1 β₁

Beta 2 β₂

Epsilon ε

Previous first moment mₜ₋₁

Previous second moment vₜ₋₁

Preset example

Result Tools

Formula Used

mₜ = β₁mₜ₋₁ + (1 − β₁)gₜ vₜ = β₂vₜ₋₁ + (1 − β₂)gₜ² m̂ₜ = mₜ ÷ (1 − β₁ᵗ) v̂ₜ = vₜ ÷ (1 − β₂ᵗ) θₜ = θₜ₋₁ − αm̂ₜ ÷ (√v̂ₜ + ε)

Adam combines momentum with adaptive variance scaling. Bias correction compensates for zero-initialised moments. Epsilon prevents unstable division near zero.

How to Use

Select a calculation mode and optimiser method.
Enter the parameter, gradient, learning rate, and moment values.
Use sequence, repeated, or multi-parameter inputs when required.
Select Calculate Adam Update to generate detailed results.
Review warnings, charts, comparisons, and export options.

Example Data

Scenario	Parameter	Gradient	Learning rate	β₁	β₂	Use
Default Adam	1.000	0.100	0.001	0.9	0.999	General training
Noisy gradient	1.000	Sequence	0.001	0.9	0.999	Momentum smoothing
AdamW	1.000	0.100	0.001	0.9	0.999	Weight decay
Large gradient	1.000	20.000	0.001	0.9	0.999	Scale adaptation

Adam Hyperparameter Guide

Learning rate α

Controls update size. Large values accelerate movement but can destabilise training.

Beta 1 β₁

Controls momentum memory. Larger values smooth noisy gradients more strongly.

Beta 2 β₂

Controls squared-gradient smoothing. Large values produce slower variance adaptation.

Epsilon ε

Improves numerical stability. Oversized epsilon weakens adaptive scaling.

Frequently Asked Questions

What does Adam mean?

Adam means Adaptive Moment Estimation. It tracks moving averages of gradients and squared gradients. These estimates control every parameter update.

Why does Adam need bias correction?

Moment estimates begin at zero. Early values are therefore biased downward. Bias correction compensates during initial iterations.

What is a good learning rate?

A common starting value is 0.001. The best value depends on data and architecture. Validation performance should guide final tuning.

What is AdamW?

AdamW decouples weight decay from gradient adaptation. This produces clearer regularisation behaviour. It is common in modern deep learning.

What is AMSGrad?

AMSGrad keeps the maximum historical variance estimate. This prevents some adaptive denominator decreases. It may improve stability in selected problems.

Why is epsilon required?

Epsilon prevents division by zero. It also affects very small denominator values. Extremely large epsilon reduces Adam’s adaptiveness.

Can Adam handle sparse gradients?

Adam adapts each parameter independently. This often helps sparse-gradient problems. Performance still depends on model and data.

Does Adam always converge better?

No optimiser dominates every task. Adam often trains quickly. SGD may generalise better in some settings.

How are multiple parameters calculated?

Each parameter maintains separate moment states. This calculator applies the same hyperparameters independently. Parameter and gradient counts must match.