Results
Activation graph
Derivative graph
Statistical summary
Calculated values
Formula and characteristics
Suitability guide
Formula used
Each selected activation function transforms an input value into an output. Numerical central differences estimate first and second derivatives where closed forms are not used. Saturation is flagged when the derivative magnitude falls below the chosen limit.
First derivative: f′(x) ≈ [f(x + h) − f(x − h)] / (2h) Second derivative: f″(x) ≈ [f(x + h) − 2f(x) + f(x − h)] / h²
How to use
Select one or more activation functions. Enter a range, manual values, generated samples, or a CSV file. Adjust graph settings and press Generate Graph.
Review output curves, derivative behavior, statistical summaries, and suitability notes. Use presets for common comparisons. Export results when your analysis is complete.
Example data
| x | Sigmoid | Tanh | ReLU |
|---|---|---|---|
| -2 | 0.119203 | -0.964028 | 0 |
| -1 | 0.268941 | -0.761594 | 0 |
| 0 | 0.500000 | 0.000000 | 0 |
| 1 | 0.731059 | 0.761594 | 1 |
| 2 | 0.880797 | 0.964028 | 2 |
Activation function guidance
Activation functions introduce nonlinear behavior into neural networks. Hidden layers often use ReLU-family, GELU, Swish, or Mish functions. Output layers depend on the prediction task.
Sigmoid suits binary probabilities, while linear activation suits unrestricted regression. Tanh remains useful when centered bounded outputs are needed. Softmax is typically applied across vectors, not isolated scalar points.
Study derivatives to understand learning behavior. Flat derivatives may produce vanishing gradients. Persistent zero derivatives can create inactive neurons.
Frequently asked questions
1. What is an activation function?
An activation function transforms a neuron’s weighted input. It gives neural networks nonlinear modeling ability.
2. Why are nonlinear activations necessary?
Stacked linear layers remain equivalent to one linear transformation. Nonlinear activations let networks learn complex patterns.
3. Which activation works well in hidden layers?
ReLU is a common baseline. GELU, Swish, Mish, Leaky ReLU, and SELU are useful alternatives.
4. When should sigmoid be used?
Sigmoid is widely used for binary classification outputs. Its values stay between zero and one.
5. What causes the dead-ReLU problem?
ReLU returns zero for negative inputs. Neurons can stop updating when their inputs remain negative.
6. How does Leaky ReLU differ from ReLU?
Leaky ReLU keeps a small negative slope. This can reduce permanently inactive neurons.
7. Why is GELU used in transformers?
GELU smoothly weights inputs rather than applying a hard cutoff. Its behavior works well in many transformer architectures.
8. What is activation saturation?
Saturation occurs when output changes very little across input changes. Derivatives become small and learning may slow.
9. Which functions are zero-centered?
Tanh and several symmetric custom functions are zero-centered. Sigmoid is not zero-centered.
10. How do derivatives affect training?
Backpropagation multiplies gradients through derivatives. Very small or very large derivatives can destabilize learning.
11. Can I graph a custom activation?
Yes. Enable Custom, enter a math.js-compatible expression using x, validate it, and generate the graph.