rahul900day/gpt-3-encoder

GPT-3-Encoder is a PHP implementation of OpenAI's original python encoder/decoder.

1.1.0 2023-03-22 20:12 UTC

Requires

• php: ^7.4 || ^8.0
• ext-mbstring: *

Requires (Dev)

• pestphp/pest: ^1.22
• phpstan/phpstan: ^1.10
• rector/rector: ^0.15.21
• symfony/var-dumper: ^5.4 || ^6.0

MIT

• Rahul Dey

• php
• encoder
• tokenizer
• decoder
• GPT-3
• gpt-2
• GPT-3-Tokenizer
• GPT-2-Tokenizer
• GPT-3-Encoder
• GPT-2-Encoder

README

PHP BPE Encoder Decoder for GPT-2/GPT-3

About

GPT-2 and GPT-3 use byte pair encoding to turn text into a series of integers to feed into the model. This is a PHP implementation of OpenAI's original python encoder/decoder which can be found here.

Get Started

Requires PHP 7.4+

Install GPT-3-Encoder via the Composer package manager

composer require rahul900day/gpt-3-encoder

Usage

Encoding a text to tokens

use Rahul900day\Gpt3Encoder\Encoder;

Encoder::encode("Your prompt.");

Decoding to text from tokens

use Rahul900day\Gpt3Encoder\Encoder;

Encoder::decode([8582, 242, 98]);

Credits

• Rahul Dey
• All Contributors

This packages has some codes and test inspiration from node's gpt-3-encoder

License

This package is released under the MIT License.