Enter predicted probabilities and the true class to calculate cross-entropy loss.
Cross-entropy loss quantifies how far a model's predicted probability distribution is from the true label, penalizing confident wrong predictions much more heavily than uncertain ones. If a model predicts 99% probability for the wrong class, the loss is severe, far worse than if it had predicted 51% for the wrong class. This asymmetric penalty is precisely why cross-entropy is the standard loss function for training classification models, it directly rewards well-calibrated confidence, not just correct top predictions.
Binary cross-entropy applies to two-class problems (spam/not-spam, positive/negative), using the formula -[y·log(p) + (1-y)·log(1-p)], where y is the true label (0 or 1) and p is the predicted probability of class 1. Categorical cross-entropy generalizes this to multi-class problems, using -log(p_true_class), the negative log probability the model assigned specifically to the correct class, ignoring the probabilities assigned to incorrect classes entirely (as long as they don't affect normalization).
Accuracy is not differentiable in a way that's useful for gradient-based optimization, it's a step function (either right or wrong), giving no useful signal about how to adjust weights. Cross-entropy loss is smooth and differentiable, providing a continuous gradient that tells the optimizer exactly which direction to adjust each weight to reduce error, this is precisely why neural networks are trained by minimizing loss functions like cross-entropy rather than directly optimizing for accuracy or F1 score, even though those are usually the metrics we actually report and care about.
When training a language model to predict the next token, each prediction is fundamentally a classification problem, choosing the correct token from a vocabulary of tens of thousands of possibilities. The model's loss during training is categorical cross-entropy, averaged across every token position, and as covered in our Perplexity Calculator, perplexity is simply the exponential of this same cross-entropy loss, they are directly two views of the identical underlying quantity.