Advanced Learning Rate Calculator for Machine Learning

Core Calculation Settings

Calculation mode

Learning-rate schedule

Optimizer

Initial learning rate

Minimum learning rate

Maximum learning rate

Batch Scaling and Distributed Training

Reference batch size

Per-device batch size

Devices per worker

Distributed workers

Gradient accumulation steps

Scaling rule

Custom scaling exponent

Training Progress and Schedule Controls

Total epochs

Current epoch

Steps per epoch

Current training step

Warmup epochs

Warmup steps override

Use zero to derive warmup steps from epochs.

Decay rate or gamma

Decay interval

Interpreted as a percentage interval for step decay.

Polynomial power

Inverse-time coefficient

Cycle length percentage

Restart multiplier

One-cycle rise fraction

Plateau reduction factor

Estimated plateau reductions

Optimizer and Stability Options

Momentum

Beta 1

Beta 2

Epsilon

Weight decay

Gradient clipping threshold

Advanced Parameter Groups

Backbone rate multiplier

Classifier rate multiplier

Layer-wise learning-rate decay

Parameter group count

Learning Rate Range Test

Range start

Range end

Iterations

Increase scale

Divergence multiplier

Learning-rate and loss pairs

Enter one learning-rate,loss pair per line.

Quick Presets

Load practical starting points, then adjust them for your data and model.

Results appear above this form after calculation.

Formula Used

Global batch size = Per-device batch × Devices × Workers × Gradient accumulation

Linear scaling:
New LR = Base LR × (Global batch ÷ Reference batch)

Square-root scaling:
New LR = Base LR × √(Global batch ÷ Reference batch)

Custom scaling:
New LR = Base LR × (Global batch ÷ Reference batch)^Exponent

Cosine annealing:
LR(t) = LRmin + 0.5 × (LRstart − LRmin) × [1 + cos(πt)]

Polynomial decay:
LR(t) = (LRstart − LRmin) × (1 − t)^power + LRmin

Inverse-time decay:
LR(t) = LRstart ÷ (1 + coefficient × t)

How to Use

Choose a learning-rate schedule and optimizer.
Enter the initial, minimum, and maximum rates.
Configure batch scaling and distributed-training values.
Set epochs, steps, warmup, decay, and cycle controls.
Optionally paste learning-rate and loss pairs.
Select comparison schedules and display precision.
Calculate, inspect charts, then export the results.

Example Data

Scenario	Optimizer	Initial LR	Batch	Schedule	Warmup
CNN classification	SGD momentum	0.1	256	Cosine	5 epochs
Transformer fine-tuning	AdamW	0.00003	32	Warmup and decay	10%
Transfer learning	AdamW	0.001	64	Cosine	3 epochs
Large-batch training	SGD momentum	0.1	1024	Linear warmup	10 epochs

Learning Rate Guidance

A high rate may overshoot useful minima. Training can oscillate or diverge. Lower it when losses rise sharply.

A low rate may train very slowly. Progress can stall before useful convergence. Increase it cautiously after stable tests.

Warmup protects unstable early training. Decay improves late-stage fine adjustments. Always confirm settings with validation performance.

SGD often uses larger rates Adam-family rates are usually smaller Large batches often need scaling Fine-tuning usually needs conservative rates

Frequently Asked Questions

What is a learning rate?

It controls update size during optimization. Larger values move parameters farther. Smaller values make gentler training steps.

Why use learning-rate warmup?

Warmup starts training with smaller updates. It can reduce early instability. Transformers frequently benefit from this approach.

What happens when the rate is too high?

Loss may oscillate or increase. Parameters can skip useful minima. Training may fail to converge.

What happens when the rate is too low?

Training becomes slow and inefficient. The model may appear stuck. More epochs may be required.

Should learning rate scale with batch size?

Often it should scale upward. Linear scaling is common for large batches. Square-root scaling is more conservative.

Which schedule is best?

No schedule wins universally. Cosine decay is widely useful. Validation results should guide selection.

What is an effective learning rate?

This calculator adjusts the scheduled rate for accumulation. It provides a practical comparison value. Framework definitions can still differ.

How does the range test work?

The rate increases across iterations. Loss trends reveal promising values. Stop before strong divergence begins.

Can parameter groups use different rates?

Yes, separate multipliers are supported. Backbones often receive smaller updates. New classifiers can learn faster.

Advanced Learning Rate Calculator