awesome/crc_fast

Fast SIMD hardware-accelerated CRC-32 and CRC-64 calculation capable of >100GiB/s

1.0.0 2025-04-11 16:29 UTC

Requires

• php: ^8.1

Apache-2.0, MIT

• Don MacAskill

• checksum
• crc32
• crc
• crc64
• simd

Extension name: ext-crc_fast

Supports Zend Thread Safety (ZTS): ✅

Supports non-Thread Safe mode (NTS): ✅

Supported operating systems: no restrictions

Configure flags:

• --with-crc-fast=<value> Specify the path to the crc-fast library

README

Fast, hardware-accelerated CRC calculation in PHP for all known CRC-32 and CRC-64 variants using SIMD intrinsics, which can exceed 100GiB/s for CRC-32, and 50GiB/s for CRC-64, on modern systems.

It is much, much faster (up to >200X) than the native crc32, crc32b, and crc32c implementations in PHP, plus adds many more variants (particularly CRC-64/NVME).

The performance gains are especially pronounced on aarch64 (Arm) systems, since PHP doesn't currently use hardware acceleration there.

CRC-64/NVME

CRC-64/NVME is in use in a variety of large-scale and mission-critical systems, software, and hardware, such as:

• The AWS S3 recommended checksum
• The Linux kernel
• The NVMe specification

CRC-32/PHP, one of PHP's special flowers 🌼

hash('crc32') in PHP is not the same as crc32() in PHP, and doesn't match the crc32 definition or output in many other programming languages and implementations (which is typically CRC-32/ISO-HDLC)

Instead, it's actually CRC-32/BZIP2 which is then byte-reversed. This extension provides a CrcFast\CRC_32_PHP algorithm const that performs the same calculation, but accelerated.

$checksum = CrcFast\checksum(
    algorithm: CrcFast\CRC_32_PHP,
    string: '123456789'
); // 181989fc

Related SIMD-accelerated PHP extensions

• simdutf PHP extension for Unicode validation and transcoding at billions of characters per second using the simdutf project.
• simdjson_plus PHP extension for parsing gigabytes of JSON per second using the simdjson project.

Requirements

Uses the crc_fast Rust package and its C-compatible shared library under the hood, so you'll need to have built and installed it. (See Usage, below).

Changes

See the change log.

Installing

Use Composer to install this library using PIE (note the Requirements above):

composer install awesome/crc-fast

If you're using a non-standard installation location for the crc_fast library, you may need to specify where the crc_fast header (in include) and shared library (in lib) can be found:

composer install awesome/crc-fast --with-crc-fast=/path/to/crc-fast

Building

Like most PHP extensions, you can also build yourself:

phpize
./configure
make

or with a custom crc_fast location where the crc_fast header and shared library can be found:

phpize
./configure --with-crc-fast=/path/to/crc-fast
make

Usage

Examples are for CRC-64/NVME, but you can use any supported algorithm variant.

Calculate CRC-64/NVME checksums:

// calculate the checksum of a string
$checksum = CrcFast\hash(
    algorithm: CrcFast\CRC_64_NVME,
    string: '123456789'
); // ae8b14860a799888

// calculate the checksum of a file, which will chunk through the file optimally,
// limiting RAM usage and maximizing throughput
$checksum = CrcFast\hash_file(
    algorithm: CrcFast\CRC_64_NVME,
    filename: 'path/to/123456789.txt',
); // ae8b14860a799888

Calculate CRC-64/NVME checksums with a Digest for intermittent / streaming / etc workloads:

$crc64Digest = CrcFast\Digest::new(
    algorithm: CrcFast\CRC_64_NVME,
);

// write some data to the digest
$crc64Digest->write('1234');

// write some more data to the digest
$crc64Digest->write('56789');

// calculate the entire digest
$checksum = $crc64Digest->finalize(); // ae8b14860a799888

Get a list of supported algorithm variants

$algorithms = get_supported_algorithms();

var_dump($algorithms);

Equivalents to PHP functions

crc32()

Calculates CRC-32/ISO-HDLC as an integer.

$checksum = CrcFast\crc32(
    data: '123456789'
); // 3421780262

hash('crc32') 🌼

Calculates CRC-32/PHP as binary or hex.

$checksum = CrcFast\hash(
    algorithm: CrcFast\CRC_32_PHP,
    string: '123456789'
    binary: false,
); // 181989fc

hash('crc32b')

Calculates CRC-32/ISO-HDLC as binary or hex.

$checksum = CrcFast\hash(
    algorithm: CrcFast\CRC_32_ISO_HDLC,
    string: '123456789'
    binary: false,
); // cbf43926

hash('crc32c')

Calculates CRC-32/ISCSI as binary or hex.

$checksum = CrcFast\hash(
    algorithm: CrcFast\CRC_32_ISCSI,
    string: '123456789'
    binary: false,
); // b798b438

IDE Stubs

This extension comes with IDE stubs for use with your favorite development environment.

Tests

See the tests directory for test coverage, which also double as useful examples.

make test

Platform support

This extension has been extensively tested on macOS and Linux, on both aarch64 and x86_64.

At Awesome we use it in production at very large scale on Linux on both Flickr and SmugMug.

This extension is not currently supported on Windows. :(

The underlying crc_fast library (same authors) builds and works on Windows, so this is likely just a build issue with creating a working config.w32 implementation. (I took a quick stab, failed, and moved on since we don't use Windows in production.)

If you want to help, please open a working PR. I'd love to merge it.

Performance

PHP already uses SIMD intrinsics for CRC-32 calculations on x86_64 but not on aarch64. Even on x86_64, this library provides considerable improvements, in addition to supporting many more variants.

Tested using the maximum settings for crc_fast for each platform, using 1MiB random payloads.

CRC-32/ISCSI and CRC-32/ISO-HDLC

Arch	Brand	CPU	System	PHP	crc_fast	Speedup
x86_64	Intel	Sapphire Rapids	EC2 c7i.metal-48xl	~27.0 GiB/s	~108.1 GiB/s	~4X
x86_64	AMD	Genoa	EC2 c7a.metal-48xl	~13.6 GiB/s	~53.7 GiB/s	~4X
aarch64	AWS	Graviton4	EC2 c8g.metal-48xl	~0.4 GiB/s	~52.3 GiB/s	~141X
aarch64	Apple	M3 Ultra	Mac Studio (32 core)	~0.4 GiB/s	~99.6 GiB/s	~233X

CRC-32/PHP 🌼

Arch	Brand	CPU	System	PHP	crc_fast	Speedup
x86_64	Intel	Sapphire Rapids	EC2 c7i.metal-48xl	~26.6 GiB/s	~27.4 GiB/s	n/a
x86_64	AMD	Genoa	EC2 c7a.metal-48xl	~13.6 GiB/s	~25.4 GiB/s	~2X
aarch64	AWS	Graviton4	EC2 c8g.metal-48xl	~0.4 GiB/s	~31.5 GiB/s	~73X
aarch64	Apple	M3 Ultra	Mac Studio (32 core)	~0.4 GiB/s	~57.8 GiB/s	~134X

CRC-64/NVME

Note that PHP has no native equivalent.

Arch	Brand	CPU	System	crc_fast
x86_64	Intel	Sapphire Rapids	EC2 c7i.metal-48xl	~54.6 GiB/s
x86_64	AMD	Genoa	EC2 c7a.metal-48xl	~27.0 GiB/s
aarch64	AWS	Graviton4	EC2 c8g.metal-48xl	~37.0 GiB/s
aarch64	Apple	M3 Ultra	Mac Studio (32 core)	~70.0 GiB/s

License

cfc-fast is dual-licensed under

• Apache 2.0 license (LICENSE-Apache or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)