README

Detects the natural language and writing system of a text. 70 languages, 25 scripts, no extensions beyond mbstring, no network calls, no model files to download.

This is a port of whatlang-rs by Sergey Potapov.

Installation

composer require avvertix/language-detect-php

Requires PHP 8.4 and ext-mbstring.

Usage

use Avvertix\WhatLang\WhatLang;

$info = WhatLang::detect('Ĉu vi ne volas eklerni Esperanton? Bonvolu! Estas unu de la plej bonaj aferoj!');

$info->lang;                // Lang::Epo
$info->lang->value;         // 'epo'  (ISO 639-3)
$info->lang->nativeName();  // 'Esperanto'
$info->lang->englishName(); // 'Esperanto'
$info->script;              // Script::Latin
$info->confidence;          // 1.0
$info->isReliable();        // true

detect() returns null when the text carries no signal — empty, whitespace, digits and punctuation only, or every candidate language filtered out.

WhatLang::detect('1234567890-,;!'); // null

Just the language, or just the script

WhatLang::detectLang('There is no reason not to learn Esperanto.'); // Lang::Eng
WhatLang::detectScript('Благодаря Эсперанто');                      // Script::Cyrillic

detectScript() is much cheaper than a full detection: it only counts characters per Unicode block and skips language scoring entirely.

Narrowing the candidates

If you know the text can only be one of a handful of languages, say so. Accuracy on short texts improves considerably.

use Avvertix\WhatLang\Detector;
use Avvertix\WhatLang\Lang;

$detector = Detector::withAllowlist([Lang::Eng, Lang::Rus]);
$detector->detectLang('That is not Russian'); // Lang::Eng

$detector = Detector::withDenylist([Lang::Eng, Lang::Ita]);
$detector->detectLang('Jen la trinkejo fermitis, ni iras tra mallumo kaj pluvo.'); // Lang::Epo

Confidence

confidence runs from 0.0 to 1.0 and reflects how far ahead the winning language is of the runner-up, scaled by how much text there was to go on — a short text needs a bigger lead to earn the same confidence. isReliable() is true above 0.9.

$info = WhatLang::detect($text);

if (! $info?->isReliable()) {
    // too little to go on — fall back to a default, or ask
}

Detection method

use Avvertix\WhatLang\Method;
use Avvertix\WhatLang\Options;

WhatLang::detect($text, new Options(method: Method::Trigram));

Method	What it does	When it helps
`Combined` (default)	Blends the two below, weighted by text length	Almost always
`Trigram`	Compares three-character frequency profiles	Longer texts
`Alphabet`	Scores languages by the characters they use	Very short texts; fastest

The combined method leans on the alphabet score for short input and shifts its weight to trigrams as the text grows past roughly a hundred characters.

Supported languages and scripts

use Avvertix\WhatLang\Lang;
use Avvertix\WhatLang\Script;

Lang::all();               // 70 languages
Script::all();             // 25 scripts
Script::Cyrillic->langs(); // [Lang::Rus, Lang::Ukr, Lang::Srp, Lang::Bel, Lang::Bul, Lang::Mkd]
Lang::Jpn->scripts();      // [Script::Hiragana, Script::Katakana]

How it works

Script detection. Every character is matched against the Unicode ranges of 25 scripts; the script with the most characters wins. Punctuation and digits are ignored.
Script to language. Most scripts imply a single language — Georgian script means Georgian — and detection stops there with full confidence. Five scripts (Latin, Cyrillic, Arabic, Devanagari, Hebrew) are shared, and go on to step 3.
Language scoring. The alphabet pass scores each candidate by the characters it uses. The trigram pass ranks the text's three-character sequences by frequency and compares that ranking against each language's stored profile of its 300 most common trigrams.

Mandarin is special-cased: its script is also used to write Japanese, so the two are told apart by how much Kana is mixed in.

Parity with whatlang-rs

tests/fixtures/reference.json holds 783 samples with the language, script and confidence whatlang-rs produces. Test ReferenceParityTest ensure parity between the PHP implementation and the Rust implementation.

To regenerate the fixture you need cargo and a checkout of whatlang-rs beside this one:

php tools/generate-fixture.php [path-to-whatlang-rs]

The trigram profiles and alphabet tables under resources/ are likewise generated from the Rust sources rather than transcribed by hand:

php tools/generate-data.php [path-to-whatlang-rs]

Testing

composer test

Credits

Derivation

Language Detection PHP is a derivative work of whatlang-rs (Rust, MIT) by Sergey Potapov. The detection algorithm, the trigram profiles and the alphabet tables all originate there.

License

The MIT License (MIT). Please see License File for more information.

avvertix / language-detect-php

Maintainers

Package info

Statistics

Security