Token Counter

v0.1.0Added Sep 26, 2026Sample data

Counts BPE tokens for OpenAI-compatible encodings (cl100k_base, o200k_base) without building the token list.

Matches tiktoken
5,000 / 5,000cases
Up to
1.7×faster
Break-even
≈ 1,285inputs

Sample data. These figures are placeholders, not measurements.

Reference: tiktoken ↗

Installation

cargo install kura-rs
kura add token-counter

Copies 3 source files into your project. You own the code from then on.

If another tool already installs a kura command, run cargo install kura-rs --bin kura-rs and use kura-rs add instead.

Usage

use crate::parts::token_counter::{Encoding, count};

let n = count(Encoding::O200kBase, &prompt);
assert!(n <= 128_000, "prompt too long: {n} tokens");

Verification

Sample data

Checked against tiktoken 0.9.0 (Python) using differential testing: the same inputs go to both implementations and the outputs are compared.

Reference
tiktoken
Python 0.9.0
Test cases
5,000
Differential testing
Passed
5,000 / 5,000
All cases match
Last run
Sep 26, 2026
UTC

These numbers are placeholders. They have not been measured yet and must not be read as real results. This part is marked "sample": true in its registry file.

Benchmarks

Sample data

kura-rs (Rust) against tiktoken across input sizes. Faster from about 1,285 inputs, up to 1.7× faster.

These numbers are placeholders. They have not been measured yet and must not be read as real results. This part is marked "sample": true in its registry file.

kura-rs (Rust)tiktoken
Wall time per run (lower is better). The shaded region is where the reference implementation is faster.
Show data table
Input sizekura-rs (Rust)tiktokenSpeedup
1000.05 ms0.03 ms1.7× slower
1,0000.31 ms0.3 ms1.0× slower
10,0002.6 ms3.4 ms1.3×
100,00024 ms36 ms1.5×
1,000,000235 ms390 ms1.7×

Source & dependencies

Files

  • parts/token_counter/mod.rs
  • parts/token_counter/bpe.rs
  • parts/token_counter/ranks.rs

Crate dependencies