README

ROUGE + BLEU = violet.

Pure-PHP text-similarity metrics: ROUGE (ROUGE-1, ROUGE-2, ROUGE-N, ROUGE-L, ROUGE-Lsum) and BLEU. Use them to measure how closely a generated text (e.g. from an LLM) matches a reference (e.g. human-written). No dependencies beyond mbstring, PHP 8.1+.

Install

composer require lukapeharda/violet

Usage

use LukaPeharda\Violet\Rouge;

$rouge = new Rouge();

// reference = human text, candidate = AI text
$scores = $rouge->score($humanText, $aiText);

echo $scores['rouge1']->f1;      // unigram overlap
echo $scores['rouge2']->recall;  // bigram overlap
echo $scores['rougeL']->precision; // longest common subsequence

echo json_encode($scores, JSON_PRETTY_PRINT);

Individual metrics: rouge1(), rouge2(), rougeN($ref, $cand, 3), rougeL(), rougeLsum(). Several human references: scoreMulti([$ref1, $ref2], $candidate) returns the best F1 per metric.

What the numbers mean

	Question it answers
Recall	How much of the human text does the AI text cover?
Precision	How much of the AI text also appears in the human text?
F1	Harmonic mean of the two

Metric	Measures
ROUGE-1	Shared vocabulary (word order ignored)
ROUGE-2	Shared two-word phrases — sensitive to phrasing
ROUGE-L	Longest in-order sequence of shared words — sentence structure
ROUGE-Lsum	ROUGE-L per sentence, unioned — for multi-sentence texts

BLEU

use LukaPeharda\Violet\Bleu;

$bleu = new Bleu();                               // BLEU-4, epsilon smoothing
$result = $bleu->score($humanText, $aiText);      // or score([$human1, $human2], $aiText)

$result->score;           // 0..1
$result->precisions;      // modified precision for 1-, 2-, 3-, 4-grams
$result->brevityPenalty;  // < 1 when the AI text is shorter than the reference

// Corpus BLEU: sums counts over all pairs, the usual way to report a dataset
$bleu->corpus([$ref1, $ref2], [$cand1, $cand2]);

// BLEU-2, no smoothing
new Bleu(maxN: 2, weights: [0.5, 0.5], smoothing: Bleu::SMOOTH_NONE);

Smoothing: SMOOTH_NONE, SMOOTH_EPSILON (default), SMOOTH_ADD_ONE, SMOOTH_NIST. These are NLTK's methods 0–3, and the output matches NLTK's sentence_bleu / corpus_bleu for the same tokens. Without smoothing, a single n-gram order with no matches makes the whole score 0, which is common for short texts.

ROUGE vs BLEU

ROUGE leans on recall: how much of the human text the AI covered. BLEU is precision-based: how much of the AI text is backed by the human text, with a penalty if it's too short. Report both. BLEU is harsher on paraphrasing, because it multiplies precisions up to 4-grams.

Tokenizer options

use LukaPeharda\Violet\Tokenizer;

$rouge = new Rouge(new Tokenizer(  // same tokenizer works for new Bleu($tokenizer)
    lowercase: true,
    stopwords: ['the', 'a', 'an', 'and'],
    stemmer: fn (string $w) => $w, // plug in a Porter stemmer etc.
));

Implement TokenizerInterface for anything custom (e.g. language-specific stemming).

License

MIT

Tests

composer install && composer test

lukapeharda / violet

Package info

Security

Statistics