Watch gradient descent find a minimum step by step. Adjust the learning rate to see convergence, overshooting, and divergence live.
Gradient descent is the core optimization algorithm behind training nearly every machine learning model, from simple linear regression to billion-parameter neural networks. At each step, it computes the gradient (the slope) of the loss function at the current position, then moves in the opposite direction of that slope, downhill, scaled by the learning rate. Repeat this enough times and the position converges toward a minimum, a point where the loss is as low as the algorithm can find from where it started.
Try setting the learning rate very low in the visualization above, convergence happens, but painfully slowly, taking many steps to reach the bottom. Now try setting it very high, watch the ball overshoot the minimum entirely, sometimes oscillating back and forth or even diverging further from the minimum with each step rather than converging. This single trade-off, slow-but-stable versus fast-but-unstable, is exactly why learning rate tuning is one of the first things every ML practitioner learns to take seriously, and why modern training almost always uses learning rate schedules rather than one fixed value for an entire training run.
Switch the loss surface to "Local Minima Trap" and try starting from different positions. Depending on where you start, gradient descent can get stuck in a shallow dip that isn't the true lowest point of the overall surface, since the algorithm only ever looks at the local slope right where it currently is, it has no way to "see" a better minimum further away. This is a real challenge in training deep neural networks, whose loss surfaces have millions of dimensions and countless local minima, though in practice, high-dimensional loss surfaces behave somewhat differently than 1D examples like this one, and techniques like momentum and adaptive learning rates (used in optimizers like Adam) help navigate around shallow local minima more effectively than plain gradient descent.
This visualization shows gradient descent on a single variable for clarity, but the same fundamental idea scales to models with millions or billions of parameters, at each training step, the algorithm computes the gradient of the loss with respect to every single parameter simultaneously, then updates all of them in the direction that reduces loss, using exactly the same core update rule you're watching here: new position = old position minus (learning rate × gradient).