lalbert / tokenizer-engine-for-wp

0.3.0 2026-11-08 14:46 UTC

Requires

• php: >=8.2
• ext-mbstring: *
• composer/installers: ^1.0 || ^2.0
• lalbert/tokenizer-engine: ^1.2

Requires (Dev)

• friendsofphp/php-cs-fixer: ^3.64
• php-stubs/wordpress-stubs: ^6.5 || ^7.0
• phpstan/phpstan: ^2.1
• phpunit/phpunit: ^9.6
• szepeviktor/phpstan-wordpress: ^2.0
• wp-phpunit/wp-phpunit: ^6.5 || ^7.0
• yoast/phpunit-polyfills: ^2.0

Suggests

• ext-intl: Unicode word segmentation and ICU folding in the analyzers

GPL-2.0-or-later

• plugin
• search
• wordpress
• text analysis
• related posts
• internal links

Published on Packagist.org on 2026-11-08 14:47 UTC

README

Search, related content and internal links for WordPress, with the text analysis of lalbert/tokenizer-engine (stemming, elisions, stop words per language, Elasticsearch-compatible analyzers). No search server needed.

Status: 0.3.

Français : README.fr.md (translation; this English documentation is the reference).

What it does

• Index the post types and taxonomies you choose, with the fields you choose (title, content, excerpt, taxonomies, custom fields) and a weight each. Shortcodes, blocks and synced patterns are cleaned so that the index holds what visitors read.
• Related content between any post types and taxonomies ("more like this"), stable over time, with pinned and excluded items in the editor: block "Related content", shortcode [tokenizer_engine_related], PHP and REST APIs.
• Search ranked with BM25 and highlighted, used for the search of the site (option) or through the APIs.
• Internal link suggestions in the block editor: passages mentioning the title of another post or term, linked in one click; suggested terms (categories, tags...) for a post.
• Automatic internal links (option): in the stored content (API, WP-CLI, with a dry run) or at display time, with guardrails (maximum per content, one link per target, no link in headings, exclusions) and the marker data-te-link="auto".
• Suggestions while typing in the search fields of the site (option).
• A public API for developers: PHP, REST, WP-CLI and hooks.

Installation

With Composer, in a project that manages WordPress with Composer (Bedrock or similar): the package has the type wordpress-plugin, so composer/installers puts it in the plugins directory.

composer require lalbert/tokenizer-engine-for-wp

Allow the plugin of composer/installers if Composer asks (config.allow-plugins).

Without Composer for WordPress, clone the plugin and install its dependencies:

cd wp-content/plugins && git clone https://github.com/lalbert/tokenizer-engine-for-wp.git && cd tokenizer-engine-for-wp && composer install --no-dev

Then activate it, go to Settings > Tokenizer Engine, enable the post types and taxonomies to index, and save: the index is built in the background (WP-Cron), or now with wp tokenizer-engine-for-wp index rebuild.

WP-Cron runs on page views: on a site with little traffic, run it from a real cron job (wp cron event run --due-now) so that the index and related content stay up to date.

Sizes

Related content is recomputed for every indexed object after changes (grouped) and every night: this suits up to some thousands of objects. See the measures of the SQL index in the README of lalbert/tokenizer-engine ("When to use it") for the search.

Requirements: WordPress 6.5+, PHP 8.2+, MySQL or MariaDB. License: GPL-2.0-or-later.