cxrxexo/tiktoken

PHP version of tiktoken. PHP 7.4+ required.

1.0.1 2023-06-24 09:38 UTC

Requires

• php: ^7.4
• symfony/service-contracts: ^2.5

Requires (Dev)

• doctrine/coding-standard: ^11.1
• phpunit/phpunit: ^9.0
• psalm/plugin-phpunit: ^0.18.4
• vimeo/psalm: ^5.12

MIT

• tokenizer
• encode
• decode
• bpe
• openai
• tiktoken

README

This is a port of the tiktoken-php.

Installation

$ composer require cxrxexo/tiktoken

Usage

use CxRxExO\Tiktoken\EncoderProvider;

$provider = new EncoderProvider();

$encoder = $provider->getForModel('gpt-3.5-turbo-0301');
$tokens = $encoder->encode('Hello world!');
print_r($tokens);
// OUT: [9906, 1917, 0]

$encoder = $provider->get('p50k_base');
$tokens = $encoder->encode('Hello world!');
print_r($tokens);
// OUT: [15496, 995, 0]

Limitations

• Encoding for GPT-2 is not supported.
• Special tokens (like <|endofprompt|>) are not supported.

License

MIT