README

Most compatible PHP implementation of OpenAI's original Tiktoken.

Get Started

Requires PHP 8.1+

Install Tiktoken via Composer package manager.

composer require rahul900day/tiktoken-php

Supported Models

Usage

Basic Usage

use Rahul900day\Tiktoken\Tiktoken;

$encoder = Tiktoken::getEncodingForModel('gpt-4');
$encoder->encode("hello world aaaaaaaaaaaa");
$encoder->decode([9906, 4435]);

Special Tokens

use Rahul900day\Tiktoken\Tiktoken;

$encoder = Tiktoken::getEncodingForModel('gpt-4');
$encoder->encode('<|endoftext|>', allowedSpecial: 'all');

Caching

Tiktoken always cache the server's responses when downloading them.

By default it uses the system's default temporary directory to cache a response but you can still overwrite the cache location by setting TIKTOKEN_CACHE_DIR environment variable.

Registering Custom Encoding

use Rahul900day\Tiktoken\Encodings\OpenAiPublic\Cl100KBaseEncoding;

class Cl100KIm extends Cl100KBaseEncoding 
{
    protected function getName(): string
    {
        return 'cl100k_im';
    }
    
    protected function getSpecialTokens(): array
    {
        return [
            ...parent::getSpecialTokens(),
            "<|im_start|>" => 100264,
            "<|im_end|>" => 100265,
        ];
    }
}

use Rahul900day\Tiktoken\Registry;
use Rahul900day\Tiktoken\Tiktoken;

Registry::registerCustomEncoding('cl100k_im', new Cl100KIm);
$encoding = Tiktoken::getEncoding('cl100k_im');

// Expect: 100264
$encoding->encode("<|im_start|>", allowedSpecial: 'all');

Changelog

Please see CHANGELOG for more information on what has changed recently.

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities.

Credits

License

This package is released under the MIT License.

rahul900day / tiktoken-php

Maintainers

Details