LongCat-Image

LongCat-Image — Multi-Granularity Captioning

Re-implementation of the Multi-Granularity Captioning (MGC) framework from the LongCat-Image Technical Report (arXiv:2512.07584, §2.1.3), by the Meituan LongCat team.

MGC is a pre-training data pipeline: its captions are consumed while building the ~1.2 B-image training corpus, so the per-image budget is the binding constraint. Everything about the design follows from that.

Figure 10 from the report (CC BY 4.0) — four captions per image, in both English and Chinese.

The idea

The report opens with three concrete complaints about the standard "throw the image at a VLM" approach:

1. Insufficient world knowledge — named entities get described generically, so the caption says "a hilltop sign" instead of "the Hollywood Sign".
2. Restricted format diversity — one VLM produces captions of near-identical structure and length, so the diffuser only ever learns one prompt shape and is brittle to real users.
3. Low information density — verbose captions burn the token budget on connective prose.

MGC's answer is not one better caption but four captions per image at increasing levels of abstraction:

Level	What it captures	Shape
Entity	the principal visual subject, identified by name	short noun phrase
Phrase	salient attributes, colors, shapes, related objects	comma-separated keywords
Composition	the overall semantic structure of the scene	one objective sentence
Photographic	finest-grained depiction incl. style, OCR, lighting, camera, world knowledge	dense paragraph

During training these are sampled with probabilities [0.05, 0.1, 0.2, 0.65] from Entity to Photographic — heavily weighted toward the densest level to maximize information, while the lighter levels keep the model usable when a real user types three words.

How the report builds them

image ──┬─► Qwen2.5-VL ────────────────────► Entity + Phrase   (generated concurrently)
        │
        ├─► InternVL2.5 ───────────────────► Composition
        │     inputs: image + Entity-level captions + extracted metadata
        │
        └─► "Photographic Captioner" ──────► Photographic
              Qwen2.5-VL backbone, LoRA-tuned on curated synthetic image–text pairs

MGC does not run in isolation. It sits third in a four-stage data curation pipeline, and the metadata extracted in stage 2 is fed into the captioner — this is what fixes the world-knowledge complaint:

1. Filtering — MD5 + SigLIP dedup, min short edge 384 px, aspect ratio 0.25–4.0, watermark detection, LAION-Aesthetics ≥ 4.5, and an in-house AIGC detector (they found even a small fraction of AI-generated training images produces a "plastic" texture).
2. Meta information extraction — category (16 classes: portrait, sport, activity, plant, animal, food, object, landscape, cityscape, indoor, UI, cartoon, chart, rich-text, poster, synthetic text), style phrases, named entities (celebrities, fictional characters, species, brands, IP), OCR text, and a two-axis aesthetic score (Quality + Artistry).
3. Multi-granularity captioning — the four levels above.
4. Stratification — pyramid split by style / quality / content for progressive training.

Named entities and OCR text are explicitly described as the inputs that raise caption accuracy; the LoRA fine-tune of the Photographic captioner exists to raise information density while retaining that world knowledge.

Contents

longcat_prompt.py — the captioning system prompt in English (CAPTION_PROMPT_EN) and Chinese (CAPTION_PROMPT_CN).

caption.py — the runner. One VLM call per image against any OpenAI-compatible endpoint.

The prompt emits all four layers in a single JSON object:

{
  "Entity":       "Hollywood Sign in Los Angeles",
  "Phrase":       ["iconic landmark", "cloudy sky", "modern house", "Los Angeles"],
  "Composition":  "Cinematic shot of the Hollywood sign under a gloomy sky, ...",
  "Photographic": "Low-saturation, dark tones enhance a solemn mood. The white ..."
}

Encoded rules worth noting:

• Absolute objectivity — no "beautiful" or "sad"; aesthetic qualities must be expressed through concrete color, light, shadow and composition.
• OCR language preservation — recognized non-English text is reproduced verbatim, never translated. This matters because LongCat-Image targets bilingual text rendering; translating the OCR in the caption would break the supervision signal for Chinese glyphs.
• Structured traversal — overall scene → background → mid-ground → foreground, with explicit directional terms.
• Style inference — inferred from content, defaulting to photorealistic, then applied consistently across the Composition and Photographic layers.
• Four worked examples covering mosaic art, a bilingual poster, a street portrait and a landmark.

Run

The report splits MGC across three models. That split is a 2024-era constraint: a current VLM emits all four levels correctly in one pass, so the whole pipeline collapses to a single call.

pip install -r requirements.txt
vllm serve Qwen/Qwen3-VL-8B-Instruct --port 8000

python caption.py images/*.jpg -o captions.jsonl

One JSON line per image, with the four levels validated:

{"image": "images/0001.jpg", "Entity": "...", "Phrase": [...], "Composition": "...", "Photographic": "..."}

Flag	Effect
--lang {en,zh}	which system prompt to use (default en)
--url	OpenAI-compatible base URL (default http://localhost:8000/v1)
--model	deployment / model id; auto asks the server
--workers N	images in flight (default 32)

Cost profile

~0.1 s/image sustained. That is throughput, not latency: a single call takes a couple of seconds, but the calls are independent and vLLM's continuous batching keeps the GPU saturated, so wall time per image collapses as concurrency rises. Raise --workers until the server's queue depth stops improving.

This is roughly 100× cheaper per image than ../ByteDance-ImageGen/ (~10 s/image), which is exactly the difference you would expect between a pre-training pipeline that must run on every image and a post-training pipeline that runs on a curated set. MGC has no serial dependency between stages, no specialist models, and no second reconciliation pass.

Reproduced vs. not

Reproduced. The four-level caption schema, the output contract and the style / objectivity / OCR rules, in both languages.

Not reproduced.

• The multi-model topology. The report generates Entity+Phrase with Qwen2.5-VL and Composition with InternVL2.5 in a separate sequential stage. This is deliberately collapsed into a single Qwen3-VL call — a stronger backbone makes the split unnecessary, and one call is both cheaper and internally consistent across the four levels. If you want the report's exact topology, run caption.py twice against two endpoints and merge.
• The Photographic Captioner. A LoRA fine-tune of Qwen2.5-VL on curated synthetic image–text pairs. Weights are not released and the fine-tuning data is not described in reproducible detail. Its stated purpose — higher information density — is approached here only through prompting.
• Metadata injection. No named-entity extractor, OCR model or aesthetic scorer is wired up, so the captioner works from the image alone — exactly the world-knowledge gap MGC was designed to close. A modern VLM knows a lot of the named entities outright, but not the long tail.
• Filtering and stratification. Stages 1 and 4 of the data pipeline are out of scope.
• The [0.05, 0.1, 0.2, 0.65] sampling. That is a training-time detail, not a captioning one.

Status

Runs end-to-end. Compare with ../ByteDance-ImageGen/, which implements a complete pipeline for a very different captioning philosophy: one deep structured record per image for post-training, instead of four independent captions at different granularities for pre-training.