ReLU calculator
Formula used
ReLU(x) = max(0, x)
ReLU′(x) = 0 for x < 0, and 1 for x > 0.
z = Σ(wᵢxᵢ) + b, then a = ReLU(z)
Z = XW + b, then A = ReLU(Z)
f(x) = x when x ≥ 0; otherwise f(x) = αx
f(x) = min(max(0, x), 6)
How to use
- Select a calculation mode and activation function.
- Enter scalar, vector, matrix, weight, and bias values.
- Adjust alpha, threshold, derivative, precision, and rounding settings.
- Choose optional validation, transpose, and comparison controls.
- Submit the form to view outputs, statistics, steps, and graphs.
- Copy, print, or export the completed calculation.
Example data
| Scenario | Input | Expected ReLU output |
|---|---|---|
| Positive scalar | 4 | 4 |
| Negative scalar | -3 | 0 |
| Mixed vector | -4, -1.5, 0, 2, 7 | 0, 0, 0, 2, 7 |
| Matrix | -1, 2; 3, -4 | 0, 2; 3, 0 |
| Dying ReLU example | -5, -2, -1 | 0, 0, 0 |
Understanding ReLU
ReLU means Rectified Linear Unit. It returns zero for negative inputs. Positive inputs pass through unchanged.
ReLU introduces non-linearity into neural networks. It is computationally simple and often trains efficiently. Its positive gradient helps many deep models.
Negative values receive a zero gradient. Neurons can therefore become permanently inactive. This behaviour is called the dying ReLU problem.
Leaky ReLU keeps a small negative slope. ELU and GELU provide smoother alternatives. The best choice depends on model architecture and data.
Frequently asked questions
What does ReLU calculate?
ReLU replaces every negative input with zero. It preserves positive inputs. Zero normally remains zero.
Why is ReLU common in neural networks?
It is simple and fast. It also supports useful gradients for positive values. These properties often improve deep-network training.
What is the derivative at zero?
The mathematical derivative is not unique there. Software usually chooses zero. This calculator supports several conventions.
What is dying ReLU?
A neuron may receive only negative pre-activations. Its gradient then remains zero. Training may stop changing that neuron.
When should Leaky ReLU be used?
Leaky ReLU preserves a small negative gradient. It may reduce inactive neurons. Alpha controls that negative slope.
How does ReLU compare with sigmoid?
Sigmoid compresses outputs between zero and one. ReLU is unbounded above. ReLU usually avoids strong positive-side saturation.
How does ReLU compare with tanh?
Tanh produces values between minus one and one. ReLU outputs are nonnegative. Their gradient behaviours also differ.
Can this calculator process matrices?
Yes, it applies activations element by element. Dense mode also multiplies matrices. It validates incompatible dimensions.
Can results be exported?
Results can be copied or printed. CSV and PDF downloads are included. A reusable CSV template is available.