Compare a candidate translation against a reference to compute BLEU score, the standard machine translation evaluation metric.
BLEU (Bilingual Evaluation Understudy) is the most widely used automatic metric for evaluating machine translation quality, introduced by IBM researchers in a 2002 paper. It works by comparing a machine-generated candidate translation against one or more human reference translations, measuring how many word sequences (n-grams) overlap between them. A BLEU score of 1.0 (or 100) means a perfect match to the reference; real-world good translations typically score between 0.3 and 0.6, since natural language allows many equally valid ways to express the same meaning.
BLEU calculates precision at multiple n-gram levels, 1-gram (individual words), 2-gram (word pairs), 3-gram (three-word sequences), and 4-gram (four-word sequences), then combines them using a geometric mean. Higher-order n-grams (3-gram, 4-gram) capture fluency and word order, a candidate can have every individual word correct (high 1-gram precision) while still reading awkwardly if word order is wrong (low 3-gram and 4-gram precision).
Without correction, a translation system could game precision scores by outputting very short, safe text, if you only output one correct word, your precision for that word is 100%. The brevity penalty (BP) counteracts this by penalizing candidate translations that are shorter than the reference, ensuring the score rewards complete, appropriately-length translations rather than sparse ones that happen to be technically correct as far as they go.
BLEU is popular because it's fast, automatic, and doesn't require human judges for every evaluation, but it has known weaknesses. It doesn't understand meaning or synonyms, "large" and "big" are scored as a complete mismatch even though they mean the same thing. It also correlates less well with human judgment for creative or highly variable text than for more literal, technical translation. Modern NLP research increasingly supplements BLEU with learned metrics like BERTScore or COMET, which better capture semantic similarity, though BLEU remains the standard baseline for comparability across research papers.