danny50610/bpe-tokeniser dependents (2) Order by: name | downloads Show: all | require | require-dev
-
-
PHP
token27/nexus-ai-tokenizer
Provider-agnostic LLM token counter for PHP 8.3+. Supports Tiktoken BPE (OpenAI, Claude approx.), HuggingFace JSON vocabularies (DeepSeek, LLaMA, Mistral), SentencePiece (Gemini), and extensible custom strategies.