xp-framework / tokenize
Text tokenizing
Installs: 91 891
Dependents: 7
Suggesters: 0
Security: 0
Stars: 0
Watchers: 3
Forks: 0
Open Issues: 0
Requires
- php: >=7.0.0
- xp-framework/core: ^12.0 | ^11.0 | ^10.0 | ^9.0 | ^8.0 | ^7.0
Requires (Dev)
- xp-framework/test: ^2.0 | ^1.0
README
Tokenizing text
use text\{StringTokenizer, StreamTokenizer}; use io\File; // Supports strings and streams $tokens= new StringTokenizer('He asked: Can you parse this?', ' .?!,;:', true); $tokens= new StreamTokenizer((new File('parse-me.txt'))->in(), ' .?!,;:', true); // Can iterate using foreach... foreach ($tokens as $token) { Console::writeLine($token); } // ...or with an iterator API while ($tokens->hasMoreTokens()) { Console::writeLine($tokens->nextToken()); } // Returns: ["He", " ", "asked", ":", " ", "Can", " ", "you", " ", "parse", " ", "this", "?"]