bge-m3.c

Multilingual embeddings at 1800 req/s on a $400 GPU — BGE-M3, every kernel written by hand in C and CUDA.

No PyTorch. No cuBLAS. No cuDNN. No FlashAttention. No tokenizers library. No Python at runtime. The whole thing — SentencePiece tokenizer, tensor-core GEMM, flash attention, int8 quantization, the batching scheduler, the servers — is hand-written C, CUDA and a single C++ file for gRPC. It matches HuggingFace's own embeddings to cosine 0.999999 and beats PyTorch's fastest path at every sequence length.

How fast?

RTX 3060 Ti, batch 1, fp16, same weights, same tokens:

End to end (tokenize → embed):

tokens	HF pipeline	torch sdpa¹	bge-m3.c	vs sdpa
16	11.3 ms	10.8 ms	2.11 ms	5.10×
128	9.8 ms	9.0 ms	4.17 ms	2.17×
512	17.4 ms	13.6 ms	12.78 ms	1.06×
4096	144.1 ms	138.5 ms	130.4 ms	1.06×
8192	—	360.1 ms	357.6 ms	1.01×

¹ scaled_dot_product_attention = FlashAttention-2 + cuBLAS, PyTorch's fastest path.

Serving — gRPC with deadline-aware dynamic batching, short-text workload:

CPU only (Ryzen 9 5900X, AVX2 int8, no GPU): 15 ms at 16 tokens, 286 ms at 512 — 2.6–8.7× faster than torch fp32 on the same cores.

And it is not a lossy shortcut. On MIRACL reranking (ko+en, 300 queries, real first-stage candidate pools) the engine scores nDCG@10 0.7995 against the fp32 oracle's 0.7995, picking the same top-1 document on 100% of queries. A paired bootstrap cannot distinguish them. Full numbers →

Pick your path

Three ways in, each about a minute. All of them need the model artifacts once (2.3 GB, downloaded from Hugging Face and converted).

1 · Direct — CLI and C library

make artifacts                      # weights.bin + tokenizer.bin (no torch needed)
make cuda                           # or: make int8   (CPU)  |  make  (fp32 reference)
./build/bgem3-cuda embed build "안녕하세요 세계" --json | jq '.dense[:4]'

Link it into your own program — the whole API is five calls:

#include "bgem3.h"
bgem3_tokenizer *tok = tok_load("build/tokenizer.bin");
bgem3_model     *m   = model_load("build/weights.bin");
int32_t ids[512];
int n = tok_encode(tok, "hello world", 512, ids);
float dense[1024];
model_forward(m, ids, n, dense, NULL, NULL);   // L2-normalized, ready to dot

make lib && cc my.c -Ibuild/include -Lbuild -lbgem3 -lm -fopenmp

Build options, backends, tuning → · API reference →

2 · gRPC server — batching, deadlines, streaming

./build/bgemserve-grpc build --port 8792        # boots in ~2 s (captures CUDA graphs)
pip install ./clients/python
python -c "from bgem3_client import Bgem3Client; print(len(Bgem3Client('localhost:8792').embed('안녕')))"

The scheduler is deadline-aware, and your RPC's own gRPC deadline is the contract: it reorders that item's queue, caps how long another batch may park the GPU, and gets refused up front if the calibrated latency model says it is unreachable. A 20 ms interactive query and a 500 ms bulk indexing job share one GPU and each get what they asked for — the relaxed one simply rides bigger batches.

c.embed(query, deadline_ms=20)                     # latency-critical
for v in c.embed_stream(corpus, window=8): ...     # throughput-critical

Serving guide, scheduler internals, tuning flags →

3 · Docker — nothing installed locally

docker compose --profile artifacts up            # one-shot: fetch + convert weights
docker compose --profile grpc up -d              # GPU server on :8792

CPU-only host? --profile grpc-cpu. Prefer HTTP/GraphQL over gRPC? --profile http (port 8791). Docker guide →

What's actually in here

src/tokenizer.c	SentencePiece Unigram + precompiled-charsmap normalization, from scratch. Byte-exact with HuggingFace on 2050 adversarial cases (Korean, emoji ZWJ, NFKC, RTL, Hangul jamo).
src/kernels_cuda.cu	Tensor-core GEMM (cp.async multi-stage pipeline, fp16 accumulate with chunked fp32 promotion), flash-attention-v2 written in raw mma.sync/ldmatrix PTX, fused epilogues.
src/kernels_int8.c	AVX2 W8A8 microkernel via the pmaddubsw abs/sign trick (no VNNI needed), with runtime dispatch to AVX-VNNI / AVX512-VNNI when the CPU has them.
src/bgem3.cu	Sequence-length buckets × batch sizes, one captured CUDA graph each; one launch per forward.
server/sched.c	Length-bucketed queues, deadline admission, boot-calibrated latency model, tokenize/GPU pipelining.
server/grpc_service.cc	The only C++ in the build.

Every optimization is gated: token-level tokenizer parity, per-layer numeric parity against a PyTorch fp32 reference, bitwise graph-replay determinism, compute-sanitizer and ASan/UBSan clean, and a retrieval-quality benchmark that has to stay level with the oracle. How the engine works →

Honest limits

• Hardware: CUDA path is tuned for sm_86 (Ampere). Other architectures need a retune, not a rewrite. No FP8 (needs Ada/Hopper), no 2:4 sparsity (would need pruning).
• Batch-1 focus: the engine beats PyTorch decisively on single sequences and short text. At equal batch size on long sequences it is ~1.16×, not 5× — both are near the same memory wall.
• Short-bucket bit-stability: sequences padded to 16 or 32 tokens are not bitwise identical between batch sizes (the GEMM dispatch keys on total rows). Cosine stays ≥ 0.999; pin batch size 1 if you need bit-exact reproducibility.
• int8 fast mode (BGEM3_INT8_FAST=1) costs ~0.7% nDCG for 2.6× speed. The default int8 mode is quality-neutral.

Roadmap

bge-reranker-v2-m3 is the next model in, and it is a small addition rather than a second engine: it is XLMRobertaForSequenceClassification with the identical backbone — 24 layers, d=1024, 16 heads, FFN 4096, vocab 250002, 8194 positions. Same tokenizer, same kernels, same buckets, same CUDA graphs, same scheduler. What differs is the head (a classifier on CLS instead of the three pooling heads), the input format (a query/passage pair rather than one text) and the output (one relevance score). Cross- encoder reranking is also the friendliest possible batching workload: one query against N candidates, all of similar length, arriving together.

Repo layout

src/        engine: tokenizer, weights loader, kernels (cpu/cuda/int8), backends
server/     scheduler + HTTP/GraphQL server (C) + gRPC service (C++)
clients/    Python gRPC client package
proto/      bgem3.proto — the service contract
export/     artifact builders (fetch.py needs no torch; dump*.py build test refs)
test/       gate harnesses: tokenizer, numeric parity, kernels, batching, sweeps
docs/       build, usage, serving, docker, benchmarks, internals
docs/spec/  SPEC1-7: the design contracts each phase was built against
examples/   minimal C and Python programs

make help lists every target.

License

MIT. Model weights are BAAI's, also MIT, and are downloaded at build time — none are vendored here.