README

Near-duplicate document detection for RAG pipelines — SimHash, MinHash, LSH, SHA-256, and shingling.

LombokSimHash is library #02 of LombokRAGFrameworks — a Tier 4 Hub in the Lombok Ecosystem. It identifies redundant documents before they enter your vector store, preventing wasteful indexing of duplicate content.

Zero runtime dependencies — pure implementation in every language (dev-only test tooling allowed)
no_std compatible — Rust core runs in embedded/WASM environments (core + alloc only)
Deterministic across languages — bit-identical hash values in Rust, TypeScript, Python, Go, and PHP, verified against one shared vector file (vectors/lomboksimhash-vectors-v2.json)
Streaming API — process large documents chunk-by-chunk (Sha256State, Shingler)
Fail-closed errors — uniform [KODE] pesan error contract across all ports

Installation

Language	Command
Rust	`cargo add lomboksimhash`
TypeScript	`npm install lomboksimhash` (primary npm publication of the ecosystem)
Python	`pip install lomboksimhash`
Go	`go get github.com/codinglombok/LombokSimHash/ports/go`
PHP	`composer require codinglombok/lomboksimhash`

Mirrors (GitHub Packages)

Every tagged release is also published automatically to GitHub Packages:

npm mirror — @codinglombok/lomboksimhash on npm.pkg.github.com. Installing from the mirror requires a GitHub token in .npmrc (a GitHub Packages quirk — even public scoped packages need read auth), which is why the npmjs.com package above stays primary.
Container image — ghcr.io/codinglombok/lomboksimhash (static musl binary, FROM scratch, zero base layer). Pulls anonymously, no token:
```
docker run ghcr.io/codinglombok/lomboksimhash:v0.1.0 simhash "the fox"
# 54b080184441c10c
```

What's inside

Feature	Purpose
SimHash (64/128-bit)	Fast fingerprinting; Hamming distance for near-duplicate detection
MinHash	Jaccard similarity estimation via k-permutation hashing
LSH	Band+row locality-sensitive hashing for approximate nearest neighbors
SHA-256	Exact content deduplication (FIPS 180-4)
Shingling	Character / word n-gram generation with streaming support

Quick Start

Rust

use lomboksimhash::{simhash64_text, hamming64};

let a = simhash64_text("the quick brown fox jumps over the lazy dog");
let b = simhash64_text("the quick brown cat jumps over the lazy dog");

let distance = hamming64(a, b);   // 8
let similar = distance <= 3;      // false — different enough

TypeScript

import { simhash64Text, hamming64, MinHasher, jaccardEstimate, LshIndex } from 'lomboksimhash';

const a = simhash64Text('the quick brown fox jumps over the lazy dog');
const b = simhash64Text('the quick brown cat jumps over the lazy dog');
console.log(hamming64(a, b));  // 8

// MinHash + Jaccard
const mh1 = new MinHasher(128);
mh1.addTokens(['the', 'quick', 'brown', 'fox']);
const mh2 = new MinHasher(128);
mh2.addTokens(['the', 'quick', 'brown', 'fox']);
console.log(jaccardEstimate(mh1.signature, mh2.signature));  // 1.0

Python

from lomboksimhash import simhash64_text, hamming64, MinHasher, jaccard_estimate, LshIndex

a = simhash64_text("the quick brown fox jumps over the lazy dog")
b = simhash64_text("the quick brown cat jumps over the lazy dog")
print(hamming64(a, b))  # 8

# LSH index for approximate nearest neighbors
lsh = LshIndex(bands=20, rows=5)          # threshold ≈ 0.549
mh = MinHasher(num_perm=100)              # 20 * 5 = 100
mh.add_tokens(["the", "quick", "brown", "fox"])
lsh.insert("doc1", mh.signature)
print(lsh.query(mh.signature))            # ['doc1']

Go

import "github.com/codinglombok/LombokSimHash/ports/go"

a := lomboksimhash.SimHash64Text("the quick brown fox jumps over the lazy dog")
b := lomboksimhash.SimHash64Text("the quick brown cat jumps over the lazy dog")
dist := lomboksimhash.Hamming64(a, b)  // 8

PHP

use CodingLombok\LombokSimHash\LombokSimHash;

$a = LombokSimHash::simhash64Text('the quick brown fox jumps over the lazy dog');
$b = LombokSimHash::simhash64Text('the quick brown cat jumps over the lazy dog');
echo LombokSimHash::hamming64($a, $b);  // 8

CLI (Rust)

$ lomboksimhash-cli simhash "hello world"
$ lomboksimhash-cli sha256 "hello world"
$ lomboksimhash-cli compare "text one" "text two"
Text 1 SimHash: ...
Text 2 SimHash: ...
Hamming distance: 12
Similarity: 81.3%

Deduplication in a RAG pipeline

Document ──┬─→ SHA-256 ──────────→ exact duplicate?   → skip
           ├─→ SimHash64 ─────────→ Hamming ≤ 3?       → near-duplicate, skip
           └─→ Shingle → MinHash → LSH candidate + Jaccard ≥ 0.8? → skip
                                              ↓
                                        otherwise → chunk + embed

Configuration

Parameter	Default	Notes
SimHash bits	64	128-bit variant available for lower collision
Shingle size	configurable	character or word n-grams
MinHash permutations	128	more permutations = better Jaccard estimate
LSH bands × rows	20 × 5	`bands * rows` must equal signature length; threshold `(1/bands)^(1/rows)`

Cross-language determinism

All ports produce bit-identical hash values, verified against the shared, deterministically generated vector file vectors/lomboksimhash-vectors-v2.json (75 cases: SHA-256, FNV-1a, shingles, SimHash 64/128, exact MinHash signatures, LSH band keys and index scenarios). The vectors are produced by vectors/generate_vectors.py — never hand-edited; CI re-verifies determinism.

Guaranteed by fixed constants: FNV-1a (0xcbf29ce484222325 / 0x100000001b3), LCG seed 0x517cc1b727220a95, and Mersenne prime 2^61 - 1.

Input	SHA-256	SimHash64
`"the quick brown fox jumps over the lazy dog"`	`05c6e08f…7450bec`	`cab7991c5475edee`
`"the quick brown cat jumps over the lazy dog"`	—	`c2e7991c1465e56f`

Error contract

All ports fail closed with identical byte-for-byte messages:

Kode	Pesan
`[ERR_SHINGLE_N]`	shingle size must be > 0
`[ERR_MINHASH_PERM]`	num_perm must be > 0
`[ERR_LSH_CONFIG]`	bands and rows must be > 0
`[ERR_SIG_LEN]`	signature length must be bands * rows = {n}
`[ERR_JACCARD_LEN]`	signatures must have same length
`[ERR_JACCARD_EMPTY]`	signatures must not be empty

Documentation

Changelog — version history
Contributing — vectors-first workflow + per-port gates
Security policy — supported versions, reporting, scope
Code of Conduct
TypeScript port · Python port · Go port · PHP port
Internal design docs (SPEC, Architecture, Masterplan, …) live in docs/ per the Lombok Ecosystem documentation pack.

codinglombok / lomboksimhash

Package info

Security

Statistics