See exactly how your learning rate changes across training epochs for different scheduling strategies.
The learning rate controls how large a step the optimizer takes when updating model weights during training. A fixed learning rate throughout training is rarely optimal, too high and training becomes unstable or fails to converge precisely, too low and training takes far longer than necessary or gets stuck in a suboptimal region. Learning rate schedules solve this by systematically changing the rate over the course of training, typically starting higher to make fast initial progress, then decreasing to allow fine-grained convergence.
Step decay drops the learning rate by a fixed factor at set intervals (e.g., halving every 30 epochs), simple and interpretable but can create visible "jumps" in training loss curves. Cosine annealing smoothly decreases following a cosine curve, popular in modern deep learning because the smooth decrease avoids the abrupt transitions of step decay, and its shape has empirically shown strong results across many architectures. Linear decay decreases at a constant rate to zero, straightforward and predictable. Exponential decay decreases multiplicatively each epoch, aggressive early reduction that slows over time.
Warmup periods, gradually increasing the learning rate from near-zero up to the target rate over the first several epochs before beginning the main decay schedule, have become standard practice for training large models, especially transformers. Without warmup, a high initial learning rate applied to randomly initialized weights can cause unstable, erratic early training, sometimes even diverging entirely. Warmup lets the model's weights settle into a reasonable region before applying the full learning rate, dramatically improving training stability for large-scale models.
Warmup + cosine annealing has become something close to a default choice for training modern transformer-based models, including most large language models, due to its consistently strong empirical results. Step decay remains common and effective for many computer vision tasks with well-established training recipes. When in doubt, cosine annealing (with or without warmup, depending on model size and training stability) is a reasonable, well-tested starting point before investing time in more exotic scheduling strategies.