Compare the shape and derivative of every major neural network activation function on one interactive chart.
If one activation function worked best for every situation, there would be no need for alternatives, but each one makes a different trade-off between computational cost, gradient behavior during training, and output range, and these trade-offs matter enormously at the scale modern neural networks operate at. Understanding the shape of each function, and critically, the shape of its derivative, explains why certain functions dominate certain types of architectures today.
Sigmoid squashes any input into a range between 0 and 1, making it historically popular for output layers in binary classification. Tanh does something similar but ranges from -1 to 1, and is zero-centered, generally preferred over sigmoid for hidden layers when this family of functions is used. Both share a serious weakness though, toggle on derivatives and look at the far left and right of the chart, both curves flatten to nearly zero. This is the vanishing gradient problem: in deep networks, when gradients this small get multiplied together across many layers during backpropagation, the resulting gradient reaching early layers becomes vanishingly small, effectively halting learning in those layers.
ReLU (Rectified Linear Unit) is almost embarrassingly simple, output the input directly if positive, otherwise output zero. Look at its derivative, it's a constant 1 for any positive input, no vanishing gradient in that region at all, which is exactly why ReLU enabled training of much deeper networks than sigmoid/tanh-based architectures could practically support. It's also far cheaper to compute than sigmoid or tanh, which involve exponentials, a real consideration when a network computes this function billions of times during training.
ReLU has its own weakness, visible in the chart, for any negative input, both the function and its derivative are exactly zero. A neuron that ends up always receiving negative input essentially "dies," it contributes nothing and never recovers, since its gradient is permanently zero. Leaky ReLU fixes this with a small non-zero slope for negative inputs (visible as the slight downward tilt below zero in the chart), keeping a small gradient alive even for negative inputs, preventing neurons from getting permanently stuck.
GELU (Gaussian Error Linear Unit) is smoother than ReLU, notice it curves gently near zero rather than having ReLU's sharp corner, and has become the standard activation function in transformer architectures, including the ones underlying most modern large language models. Its smoothness tends to produce slightly better empirical results in these specific architectures, though it costs somewhat more to compute than plain ReLU, a trade-off that large-scale model training has generally found worthwhile.