ghostwriter/plex

Provides the fastest lexer for PHP, tokenizing text with named regex patterns for efficient processing

0.1.3 2024-09-05 17:49 UTC

Requires

• php: >=8.3

Requires (Dev)

• ghostwriter/coding-standard: dev-main

BSD-3-Clause

• Nathanael Esayeas

• plex
• ghostwriter

README

Provides the fastest lexer for PHP, tokenizing text with named regex patterns for efficient processing.

Installation

You can install the package via composer:

composer require ghostwriter/plex

Star ⭐️ this repo if you find it useful

You can also star (🌟) this repo to find it easier later.

Usage

<?php

declare(strict_types=1);

use Ghostwriter\Plex\Grammar;
use Ghostwriter\Plex\Token;
use Ghostwriter\Plex\Lexer;


$grammar = Grammar::new([
    'T_NUMBER' => '\d+',
    'T_DOT' => '\.'
]);

$lexer = Lexer::new($grammar);

$expected = [
    Token::new('T_NUMBER', '1', 1, 1, []),
    Token::new('T_DOT', '.', 1, 2, []),
    Token::new('T_NUMBER', '2', 1, 3, []),
    Token::new('T_DOT', '.', 1, 4, []),
    Token::new('T_NUMBER', '3', 1, 5, []),
];

$content = '1.2.3';

assert($expected == iterator_to_array($lexer->lex($content)));

Named Patterns

When defining a grammar, you can reference other grammar rules by name using the (?&NAME) syntax, where NAME is the name of the grammar rule.

This allows you to create complex patterns that are easier to read and maintain.

<?php

declare(strict_types=1);

use Ghostwriter\Plex\Grammar;
use Ghostwriter\Plex\Token;
use Ghostwriter\Plex\Lexer;


$grammar = Grammar::new([
    // References both 'T_NUM' and 'ref-str' (which references T_STR)
    'T_ID' => '(?:(?&T_NUM)|(?&ref-str))*',
    
    // Matches numbers
    'T_NUM' => '\d+',
    
    // Matches word characters
    'T_STR' => '\w+',
    
    // References T_STR
    'ref-str' => '(?&T_STR)',
]);

$lexer = Lexer::new($grammar);

$expected = [
    Token::new('T_ID', '456def', 1, 6, [])
];

$content = '456def';

assert($expected == iterator_to_array($lexer->lex($content)));

Credits

• Nathanael Esayeas
• All Contributors

Changelog

Please see CHANGELOG.md for more information on what has changed recently.

License

Please see LICENSE for more information on the license that applies to this project.

Security

Please see SECURITY.md for more information on security disclosure process.