HF Adapters for Spyre

Minimal runtime patches that make stock HuggingFace Transformers models run on Spyre accelerators.

No forks, no custom model classes — each adapter monkey-patches the standard HF model at load time, replacing only the operations Spyre cannot execute natively (RoPE, RMSNorm, KV cache management, generation loop). Everything else — weights, tokenizer, config — comes straight from transformers.

Supported Models

34 adapters · 55 verified checkpoints · 10K+ compatible models

Coverage spans generative (causal-LM), embedding (sentence-transformers), sequence classification (sentiment / text categorisation), token classification (NER/POS), vision-language (image→text), and speculative-decoding drafter models — from Llama / Qwen / Granite / Mistral / Phi / Gemma / OLMo / GPT decoders to BERT / XLM-RoBERTa / MPNet / ModernBERT encoders, the Granite Vision 4.1 (SigLIP tower + Granite text), Mistral3 Vision (Pixtral tower + Mistral text), and Gemma 4 (encoder-free) multimodal VLMs, plus the DSpark block-propose drafters for Qwen 3 / Granite / Gemma 4.

Each adapter covers all size variants and fine-tuned checkpoints sharing the same HuggingFace model_type. The canonical, per-adapter model lists — verified checkpoints, also-compatible models, head_dim / stick-alignment details, and Spyre numerical accuracy — live in ARCHITECTURE.md.

Installation

# Install core deps
uv sync

# Install core + dev deps
uv sync --group dev

# Install core + torch-spyre deps
uv sync --group spyre

# Install core + test deps
uv sync --group test

# Install everything
uv sync --group dev --group spyre --group test

Quick Start

from hf_adapters import AutoSpyreModelForCausalLM
from transformers import AutoTokenizer

model = AutoSpyreModelForCausalLM.from_pretrained("ibm-granite/granite-3.3-8b-instruct")
tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-3.3-8b-instruct")

inputs = tokenizer(["What is 2+2?"], return_tensors="pt", padding=True)
sequences = model.generate(**inputs, max_new_tokens=5)
outputs = tokenizer.batch_decode(
    sequences[:, inputs["input_ids"].shape[1] :],
    skip_special_tokens=True,
)
print(outputs[0])

The only change from a stock Hugging Face script is the model class — AutoSpyreModelForCausalLM instead of AutoModelForCausalLM. Tokenization, generation arguments, and decoding all work the same way.

model.generate() follows the stock Hugging Face input and basic tensor-output conventions, but supports a smaller set of generation features (see docs/generate_vs_stock_hf.md).

For instruct checkpoints, the convenience helper encode_prompts() applies the model's chat template automatically (or plain tokenizer post-processing for base models). It is recommended when you want canonical tokenization without manual template handling:

from hf_adapters import AutoSpyreModelForCausalLM, encode_prompts

inputs = encode_prompts(tokenizer, ["What is 2+2?"])
sequences = model.generate(**inputs, max_new_tokens=5)

Embedding Models

For embedding models, use the sentence-transformers library with the backend="spyre" parameter:

import hf_adapters.st_backend  # Register Spyre backend
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B", backend="spyre")
embeddings = model.encode(["hello world", "how are you"])

The st_backend module automatically patches sentence-transformers to apply the relevant Spyre adapter when loading the model. All standard SentenceTransformer methods (encode(), similarity(), etc.) work unchanged.

Masked Language Models

Use AutoSpyreModelForMaskedLM for bidirectional masked-token prediction with encoder models. The encoder runs on Spyre and the model-specific MLM head runs on CPU:

import torch
from transformers import AutoTokenizer
from hf_adapters import AutoSpyreModelForMaskedLM

model_path = "google-bert/bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoSpyreModelForMaskedLM.from_pretrained(model_path)
batch = tokenizer(
    [f"The capital of France is {tokenizer.mask_token}."],
    return_tensors="pt",
)

with torch.no_grad():
    outputs = model(**batch)
logits = outputs.logits
mask = batch["input_ids"].eq(tokenizer.mask_token_id)
print(tokenizer.decode(logits[mask].argmax(dim=-1)))

outputs.logits is a CPU tensor shaped [batch, sequence, vocab]. This is inference-only masked-language modeling, not left-to-right causal generation.

Extractive Question Answering

Use AutoSpyreModelForQuestionAnswering with fine-tuned BERT-family QA models. The encoder runs on Spyre, the small token-classification head runs on CPU, and the normal Hugging Face output contract is preserved:

from transformers import AutoTokenizer
from hf_adapters import AutoSpyreModelForQuestionAnswering

model_path = "deepset/roberta-base-squad2"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoSpyreModelForQuestionAnswering.from_pretrained(model_path)
batch = tokenizer(
    "Where do I live?",
    "My name is Wolfgang and I live in Berlin.",
    return_tensors="pt",
)
outputs = model(**batch)
start = outputs.start_logits.argmax(dim=-1)
end = outputs.end_logits.argmax(dim=-1)
answer = tokenizer.decode(batch["input_ids"][0, start.item() : end.item() + 1])

Encoder task inputs must be right-padded. Masked-LM and question-answering support inference from input_ids; training/loss, inputs_embeds, attentions, and hidden-state collection are not currently supported.

Sequence Classification

Use AutoSpyreModelForSequenceClassification for models that return a single label per input (sentiment analysis, topic classification, natural language inference). The encoder runs on Spyre; the classification head runs on CPU. Returns a standard HuggingFace SequenceClassifierOutput with logits [B, num_labels] on CPU:

from transformers import AutoTokenizer
from hf_adapters import AutoSpyreModelForSequenceClassification

model_path = "distilbert/distilbert-base-uncased-finetuned-sst-2-english"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoSpyreModelForSequenceClassification.from_pretrained(model_path)
batch = tokenizer(
    ["I really enjoyed this film!", "The plot was confusing and dull."],
    return_tensors="pt",
    padding=True,
)
outputs = model(**batch)
label_ids = outputs.logits.argmax(dim=-1)
labels = [model.config.id2label[i.item()] for i in label_ids]
print(labels)  # → ['POSITIVE', 'NEGATIVE']

Token Classification (NER / POS)

Use AutoSpyreModelForTokenClassification for token-level label prediction (named-entity recognition, part-of-speech tagging, chunking). The encoder runs on Spyre; the linear classifier head runs on CPU. Returns a standard HuggingFace TokenClassifierOutput with logits [B, L, num_labels]:

from transformers import AutoTokenizer
from hf_adapters import AutoSpyreModelForTokenClassification

model_path = "dslim/bert-base-NER"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoSpyreModelForTokenClassification.from_pretrained(model_path)
batch = tokenizer(
    ["John lives in New York and works at IBM."],
    return_tensors="pt",
    padding=True,
    return_attention_mask=True,
)
outputs = model(**batch)
label_ids = outputs.logits.argmax(dim=-1)
labels = [model.config.id2label[i] for i in label_ids[0].tolist()]
print(list(zip(tokenizer.convert_ids_to_tokens(batch["input_ids"][0]), labels)))

Multimodal Models (image → text)

For vision-language models, use AutoSpyreModelForImageTextToText. It loads the full VLM via AutoModelForImageTextToText, prepares both towers (vision + text decoder) for Spyre, and exposes a multimodal generate:

from hf_adapters import AutoSpyreModelForImageTextToText
from transformers import AutoProcessor
from PIL import Image

# --- Granite Vision 4.1 ---
model = AutoSpyreModelForImageTextToText.from_pretrained("ibm-granite/granite-vision-4.1-4b")
processor = AutoProcessor.from_pretrained("ibm-granite/granite-vision-4.1-4b")
# Build the batch the official way — the chat template tokenizes and expands the
# image tokens in one call (the two-step text/images path mis-tiles anyres images).
image = Image.open("cat.jpg").convert("RGB")
conv = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": "Briefly describe this image."},
]}]
batch = processor.apply_chat_template(
    conv, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt"
)

sequences = model.generate(**batch, max_new_tokens=64)
prompt_len = batch["input_ids"].shape[1]
texts = processor.batch_decode(sequences[:, prompt_len:], skip_special_tokens=True)
print(texts[0])

A multimodal checkpoint's config is registered under both auto classes: AutoSpyreModelForCausalLM selects the text-only adapter (vision tower discarded), while AutoSpyreModelForImageTextToText selects the combined multimodal adapter. This works for Granite Vision (Granite4VisionConfig), Mistral3 Vision (Mistral3Config), and Gemma 4 (Gemma4UnifiedConfig, an encoder-free VLM — no vision tower; see ARCHITECTURE.md).

Repo Structure

README.md
ARCHITECTURE.md                        Detailed status, architecture docs

hf_adapters/
├── auto_spyre_model.py        Unified auto-loading interface (AutoSpyreModel, AutoSpyreModelForCausalLM)
├── hf_common.py               Shared utilities: RoPE precomputation,
│                               RMSNorm patching, LM head padding,
│                               head-dim padding, mask builders,
│                               KV cache helpers, generate loop
├── hf_*.py                    One adapter per model family (see the Model Family
│                               Coverage table in ARCHITECTURE.md for the full list)
├── st_backend.py               sentence-transformers `backend="spyre"` integration for embedding models
└── __init__.py

tests/                                 CPU tests (no Spyre required)
├── test_adapter_cpu_accuracy.py       CPU: adapter vs stock HF (causal-LM)
├── test_embed_cpu_accuracy.py         CPU: embedding hidden-states vs stock HF
├── test_vlm_e2e_cpu.py                CPU: multimodal adapter vs stock generate
├── test_load_cpu.py                   CPU: models load without errors
└── spyre/                             Spyre tests (require hardware + torch_spyre)
    ├── test_e2e_smoke_spyre.py        E2E: load + generate on Spyre
    ├── test_e2e_token_compare_spyre.py E2E: HF CPU vs adapter Spyre tokens
    ├── test_e2e_embed_compare_spyre.py E2E: HF CPU vs adapter Spyre embeddings
    ├── test_e2e_seq_classification_compare_spyre.py E2E: HF CPU vs adapter Spyre seq-classification logits
    ├── test_vlm_e2e_spyre.py          E2E: multimodal adapter on Spyre (teacher-forced)
    └── test_load_spyre.py             Spyre: models load without errors

Requirements

• Python 3.10+
• PyTorch 2.x
• transformers
• sentencepiece
• accelerate
• sentence_transformers
• torch_spyre (for Spyre hardware only — not needed for CPU tests)

Running Tests

Two classes: CPU-only (adapter vs stock HF on CPU) and Spyre (requires Spyre hardware + torch_spyre).

CPU Tests (no Spyre required)

Compares adapter's patched forward pass against stock HF on CPU. Greedy tokens must match at every step. Downloads weights on first run.

Important: CPU tests must be run from the repository root with pytest to ensure proper module patching:

# Adapter accuracy tests (causal-LM logits)
uv run pytest tests/test_adapter_cpu_accuracy.py                  # all causal-LM models
uv run pytest tests/test_adapter_cpu_accuracy.py -k qwen3         # one model (manual + auto-loader)
uv run pytest tests/test_adapter_cpu_accuracy.py -k "qwen3 and manual"    # manual adapter only

# Embedding accuracy tests (hidden-states)
uv run pytest tests/test_embed_cpu_accuracy.py                    # all embedding models
uv run pytest tests/test_embed_cpu_accuracy.py -k bge_base        # one model

# Load test (verify models load without errors)
uv run pytest tests/test_load_cpu.py                              # CPU load test

Note: Do not run CPU tests with python tests/test_*.py — this bypasses pytest's conftest.py setup and will cause import errors. Always use pytest (or uv run pytest).

Spyre Tests (requires Spyre hardware)

The Spyre lane lives under tests/spyre/ and is also pytest-driven (not python tests/...). Each test is parametrized off the model registry, so a single model is selected with -k <key> (e.g. granite2b, qwen3, bge_base). Run the whole file to cover every registered model. Run from the repository root.

# E2E smoke test (real weights, verify non-trivial output)
uv run pytest -s -vvv tests/spyre/test_e2e_smoke_spyre.py                  # one representative model per adapter
uv run pytest -s -vvv tests/spyre/test_e2e_smoke_spyre.py -k granite2b     # one model

# E2E token comparison (HF CPU vs adapter Spyre, per-step greedy tokens)
uv run pytest -s -vvv tests/spyre/test_e2e_token_compare_spyre.py -k granite2b

# E2E embedding comparison (HF CPU vs adapter Spyre, hidden-states cosine)
uv run pytest -s -vvv tests/spyre/test_e2e_embed_compare_spyre.py -k bge_base

# E2E multimodal VLM (image→text; teacher-forced per-step logit comparison)
uv run pytest -s -vvv tests/spyre/test_vlm_e2e_spyre.py -k granite_vision_mm

# Load test (verify a model loads on Spyre without errors)
uv run pytest -s -vvv tests/spyre/test_load_spyre.py

-s -vvv matches each test's documented usage and shows the per-step comparison tables the token / embedding / VLM tests print.

Numerical gating depends on the workload. The blocking causal token-comparison lane requires exact greedy top-1 agreement with CPU over prefill and four decode steps. The VLM lane instead asserts a per-step logit cosine floor because its open-ended caption prompts can produce near-tied top-1 candidates (see ARCHITECTURE.md).

Development

Pre-commit Hooks

This project uses pre-commit to enforce code quality checks before each commit. The following hooks are configured:

• Trailing whitespace / end-of-file fixer / mixed line endings
• File checks: YAML, TOML, JSON validation; large file guard (>1 MB); merge conflict markers; debug statements
• Black — code formatting
• Ruff — linting with auto-fix
• mypy — static type checking (runs on hf_adapters/ only)

Setup

uv sync --group dev

pre-commit install          # activate hooks in your local clone

Usage

Hooks run automatically on git commit. To run manually against all files:

pre-commit run --all-files

License

Apache 2.0