Enter token probabilities from your language model to calculate perplexity, the standard measure of prediction quality.
Perplexity quantifies how "surprised" a language model is by a sequence of text, specifically, how well it predicts each actual next token given everything before it. Mathematically, perplexity is the exponential of the average cross-entropy loss across all tokens. A perplexity of 1 would mean the model assigns 100% probability to every correct next token, perfect prediction. A perplexity of 10 means the model is, on average, as uncertain as if it were choosing uniformly among 10 equally likely options at each step, lower is always better.
Unlike task-specific metrics (BLEU for translation, accuracy for classification), perplexity is computed directly from the model's own probability outputs on a held-out text sample, requiring no separate labeled evaluation dataset or human reference. This makes it the default way researchers compare raw language modeling capability across model checkpoints during training, and across different model architectures on standard benchmark corpora like WikiText or Penn Treebank.
Perplexity and cross-entropy loss are directly related, perplexity is simply e raised to the power of the cross-entropy loss (when loss is measured in nats, the natural-log unit typically used in training). This is why you'll see language model training curves reported in either loss or perplexity interchangeably, they carry the same information, but perplexity is generally considered more interpretable since it has a more intuitive scale (roughly "how many equally-likely choices is the model effectively choosing among").
A model can achieve excellent perplexity on a benchmark by simply memorizing patterns common in its training data distribution, without that necessarily translating into being genuinely useful, coherent, or accurate for real downstream tasks. Perplexity also isn't directly comparable across models using different tokenizers, since the same text produces a different number of tokens depending on tokenization, changing the denominator in the average. This is why perplexity is treated as one useful signal among several during model evaluation, not a complete substitute for task-specific benchmarks or human evaluation.