xp-framework/tokenize

Text tokenizing

v9.1.0 2024-03-24 13:28 UTC

This package is auto-updated.

Last update: 2024-11-05 11:22:15 UTC


README

Build status on GitHub XP Framework Module BSD Licence Requires PHP 7.0+ Supports PHP 8.0+ Latest Stable Version

Tokenizing text

use text\{StringTokenizer, StreamTokenizer};
use io\File;

// Supports strings and streams
$tokens= new StringTokenizer('He asked: Can you parse this?', ' .?!,;:', true);
$tokens= new StreamTokenizer((new File('parse-me.txt'))->in(), ' .?!,;:', true);

// Can iterate using foreach...
foreach ($tokens as $token) {
  Console::writeLine($token);
}

// ...or with an iterator API
while ($tokens->hasMoreTokens()) {
  Console::writeLine($tokens->nextToken());
}

// Returns: ["He", " ", "asked", ":", " ", "Can", " ", "you", " ", "parse", " ", "this", "?"]