Post-training INT8 quantization for Linear layers. Weights are stored as
int8 plus a per-channel float32 scale, and the matmul runs in integers.
This is real quantization — not fake/simulated — so the weights actually shrink
on disk and in memory (~4x), and the forward pass does integer int8 @ int8
arithmetic.
model = Sequential(Linear(784, 256), ReLU(), Linear(256, 10))
# ... train the model ...
model.quant() # convert every Linear to int8 in place
pred = model(x) # forward now runs the integer path
model.save("model_int8.tt") # checkpoint stores int8 weights + scalesquant() is recursive: it walks the whole module tree (including inside
Sequential) and quantizes every Linear. Layers without a quant() method of
their own (activations, Conv2d, Embedding, norms) are left untouched.
Quantization maps float32 numbers onto the integer range [-127, 127]
(symmetric int8). For an array W:
scale = max(|W|) / 127
W_q = clip(round(W / scale), -127, 127) # int8
Dequantization is just the inverse: W ≈ W_q * scale. The scale is a single
float you store alongside the integers. The round() is where precision is
lost — usually around a 1% change in the output.
Weights are quantized per-channel (one scale per output neuron). Different neurons have different weight magnitudes, so a shared scale would over-compress the small ones; a per-neuron scale is much more accurate at almost no cost (one extra float per neuron).
Activations are quantized dynamically at each forward — their range depends on the input and isn't known ahead of time, so the scale is computed on the fly (per-tensor).
The integer matmul works because the scales factor out of the sum:
x @ W ≈ (x_q · scale_x) @ (W_q · scale_w)
= (x_q @ W_q) · scale_x · scale_w
So the forward pass is: quantize x, compute x_q @ W_q in int32, then
multiply the result by scale_x * scale_w and add the (float) bias.
Only Linear. In an MLP or transformer that's the bulk of the parameters, so
this is the standard choice. bias stays float32 (it's tiny and quantizing it
would only add error). Because bias and scales stay float, the compression ratio
is a little under 4x on small layers and approaches 4x as layers grow.
model.quant()
# fc1.weight -> fc1.w_q (int8) + fc1.w_scale (float32, one per neuron)
# fc1.bias -> fc1.b_data (float32, unchanged)
# the original float32 weight/bias are dropped to actually free memorysave()/load() understand quantized checkpoints. The state dict stores the
int8 buffers (w_q, w_scale, b_data) instead of float weights, so the file
is ~4x smaller. On load(), a layer that finds int8 buffers for itself switches
into quantized mode automatically — no extra call needed.
model.quant()
model.save("model_int8.tt")
fresh = Sequential(Linear(784, 256), ReLU(), Linear(256, 10))
fresh.load("model_int8.tt") # comes back already quantized
pred = fresh(x)- CPU/NumPy only. Quantize on CPU: call
model.cpu()beforemodel.quant(). int8 on the GPU wouldn't be faster here without custom kernels. - Post-training only — there is no quantization-aware training, and no
dequant()to go back to float. Keep a float checkpoint from beforequant()if you might want to fine-tune later. int32accumulation can only overflow at absurdin_features(>~130k), which you won't hit in practice.
See docs/nn.md for
Linearinternals and docs/model_saving.md for the checkpoint format.