work.log

# Pulpie Work Log

## Goal
Build a fast, CPU-only pipeline for extracting main content from raw HTML and outputting clean markdown. Target: close the gap to Dripper-0.6B (0.878 ROUGE-5 on WebMainBench).

## Architecture
6-stage Rust pipeline (~2K LOC):
1. **Sanitize** — Remove script/style tags, decode entities
2. **Prune** — Strip boilerplate elements (ads, trackers, consent banners)
3. **Segment** — Walk DOM, extract paragraph-level blocks with ~40 structural features
4. **Classify** — LightGBM binary classifier (keep/discard per block)
5. **Markdown** — Convert kept blocks to markdown
6. **Cleanup** — Postprocess markdown output

Exports: `extract(html) -> markdown` and `extract_html(html) -> html` (for piping through html2text).

## Training Data

### WebMainBench (primary)
- 7,809 annotated pages with `cc-select="true"` on main content elements
- DOM-based label projection: walk DOM same way pulpie does, check for `cc-select` descendants, match to Rust-extracted blocks by normalized text
- **446,509 training blocks**: 264,512 KEEP (59.4%) / 181,997 DISCARD (40.6%)
- Script: `data/generate_training_data_dom.py`

### Common Crawl (labeled + double-checking)
- CC-MAIN-2026-12, 3 WARC files, 16,670 English pages (one per unique domain)
- **DeepSeek V3.2 labeling** (complete): 16,376/16,670 OK, 294 LLM failures, 0 parse failures
  - Cost: $129.23, 5.1 hours on Bedrock
  - Script: `data/label_cc.py`, output: `data/cc_labeled.jsonl`
- **Filtering** (complete): 15,880 pages kept after removing tiny (<5 blocks), all-main (>90%), capping empty at 500
  - 1,227,257 total blocks: 337,856 main (27.5%) / 889,401 discard (72.5%)
  - Script: `data/filter_and_audit_cc.py`, output: `data/cc_labeled_filtered.jsonl`
- **Quality audit**: 20 random pages audited by LLM sub-agents → 11 GOOD, 6 ACCEPTABLE, 1 POOR
  - Main systematic error: article titles/bylines/dates labeled OTHER (consistent, low-impact)
- **Truncation fix**: 43 pages had output_tokens >= 3000 (MAX_TOKENS=4096 in original run)
  - Re-labeled with MAX_TOKENS=16384 → same output, confirming model attention degrades on long prompts (not hard truncation)
  - Dropped these 43 pages from dataset
  - Script: `data/relabel_truncated.py`
- **Dripper double-check** (in progress): Running Dripper 0.6B on all 15,880 pages to cross-validate DeepSeek labels
  - 20-page pilot: 90.3% block-level agreement, median per-page 94.7%
  - Disagreement is asymmetric: Dripper labels more blocks as main (8.2% dr→main vs 1.5% ds→main)
  - Script: `data/doublecheck_with_dripper.py`, vLLM on port 8235 (64K context)
  - ETA: ~4 hours for full run
  - **Plan**: Keep only blocks where both models agree → high-confidence training labels
  - Discard disagreeing blocks (~10%) — safe for block-level classifier training since each block is independent

## Feature Engineering (40 features)

Top features by LightGBM importance:

| Feature | Importance | Description |
|---------|-----------|-------------|
| link_ratio | 573K | Fraction of block text that is link text |
| page_total_blocks | 335K | Total blocks on page |
| parent_tag_type | 282K | Parent element type |
| page_total_link_ratio | 281K | Page-level link density |
| tag_type | 244K | Block element type (p, h1-h6, li, pre, td, etc.) |
| section_block_count | 235K | Blocks in current section |
| page_total_text_len | 214K | Total text on page |
| dom_depth | 141K | Depth in DOM tree |
| section_link_density | 131K | Link density in current section |
| position | 114K | Normalized position on page (0.0-1.0) |

Full list in `data/selected_features.json`.

## GBM Model

### Hyperparameter tuning (7 configs tested):

| Config | Leaves | LR | CV Loss |
|--------|--------|-----|---------|
| baseline | 31 | 0.05 | — |
| leaves63 | 63 | 0.05 | — |
| **leaves127** | **127** | **0.05** | **0.06925** |
| leaves127_reg | 127 | 0.05 | — (regularized) |
| leaves127_lr02 | 127 | 0.02 | — |
| leaves255 | 255 | 0.05 | — |
| depth10_leaves127 | 127 | 0.05 | — (depth=10) |

### Best config:
```
num_leaves: 127
learning_rate: 0.05
min_data_in_leaf: 20
max_depth: -1 (unlimited)
feature_fraction: 0.8
bagging_fraction: 0.8
bagging_freq: 5
scale_pos_weight: 0.68
num_boost_round: 3,083 (early stopping @100)
```

### Model files:

| File | Size | Description |
|------|------|-------------|
| model_dom.txt | 102 MB | **Deployed** — Combined WMB+CC, 7696 rounds |
| model_dom_wmb_only.txt | 41 MB | Previous — WMB-only, 3083 rounds |
| model_combined.txt | 102 MB | Copy of current model_dom.txt |
| model.txt | 51 MB | Word-overlap labels (older, inferior) |
| model_emb.txt | 29 MB | With 256-dim model2vec embeddings (experimental) |
| model_split_test.txt | 1.3 MB | Test split model |

## Benchmark Results

### ROUGE-5 F1 (WebMainBench, English only, 6,647 pages):

| Method | ROUGE-5 F1 | Notes |
|--------|-----------|-------|
| Pulpie (WMB+CC) | **0.808** | Combined training data, 7696 rounds |
| Pulpie (WMB only) | 0.806 | Previous model, 3083 rounds |
| Pulpie (native md) | 0.745 | Custom markdown converter |
| Dripper 0.6B (paper) | 0.878 | 7,809 pages including non-English |
| magic-html (paper) | 0.714 | 7,809 pages including non-English |

### Qrater Clean Rate (WebMainBench, 500 English pages):

| Method | Clean% | Simple | Mid | Hard |
|--------|--------|--------|-----|------|
| Dripper 0.6B | **80.0%** | 83% | 78% | 80% |
| Pulpie (GBM) | 66.0% | 76% | 65% | 58% |
| Raw html2text | 5.6% | 12% | 3% | 3% |

**Key finding**: 14pp gap between pulpie and Dripper on qrater clean rate. On hard pages, gap is 22pp (58% vs 80%). Dripper is remarkably consistent across difficulty levels while pulpie degrades sharply.

This confirms the GBM structural approach is **saturated at ~0.81 ROUGE / ~66% qrater clean**. More training data (CC) gave negligible gains (+0.2pp ROUGE). The remaining gap is architectural — structural features cannot capture the page-level context that Dripper's full-page classification provides.

By difficulty (html2text path):

| Level | Pages | F1 | P | R | Empty |
|-------|-------|-----|---|---|-------|
| Simple | 144 | 0.893 | 0.894 | 0.908 | 1 |
| Mid | 193 | 0.825 | 0.839 | 0.849 | 4 |
| Hard | 163 | 0.742 | 0.748 | 0.761 | 6 |

F1 distribution (500-page sample):
```
  0.9-1.0:  300  (60.0%)  — working well
  0.8-0.9:   55  (11.0%)
  0.6-0.8:   63  (12.6%)
  0.4-0.6:   31  ( 6.2%)
  0.2-0.4:   20  ( 4.0%)
  0.0-0.2:   17  ( 3.4%)
      0.0:   14  ( 2.8%)  — empty extraction
```

71% of pages score >= 0.8. The 29% tail drags the average down.

**Gap to Dripper: ~7pp**, split as:
- Formatting: ~3pp (solved by html2text canonicalization)
- Content extraction: ~4pp (fundamental)

## Error Analysis (500 English pages, html2text)

| Category | Pages | % | Avg F1 | Est. F1 Cost |
|----------|-------|---|--------|-------------|
| Good (F1>=0.8) | 355 | 71.0% | 0.961 | — |
| Boilerplate leaking | 36 | 7.2% | 0.522 | ~2pp |
| Formatting mismatch | 26 | 5.2% | 0.601 | ~3pp (native md only) |
| Mixed issues | 30 | 6.0% | 0.624 | — |
| Low recall | 21 | 4.2% | 0.557 | ~1pp |
| Complete failures | 21 | 4.2% | 0.137 | ~0.5pp |
| Empty | 11 | 2.2% | 0.000 | ~0.5pp |

### Boilerplate leaking (biggest content issue)
- Classifier keeps navigation tables, sidebars, product specs
- Worst: esaral.com (P=0.044, pred=30K vs ref=1.3K), automobilemag.com (P=0.035)
- **Root cause**: GBM lacks signal to distinguish content tables from nav tables

### Low recall
- Forums/multi-section pages: only first post extracted
- Example: squarespace forum (P=1.0, R=0.081) — headline only
- **Root cause**: classifier keeps first content cluster, drops subsequent identical-structure blocks

### Empty extractions
- Forums with non-block content, JS-rendered, comment sections
- **Root cause**: no fallback for pages without block-level elements

## Experiments Tried

### 1. Word-overlap labels → DOM-based labels
- **Before**: `generate_training_data.py` — project labels via word overlap between block text and reference
- **After**: `generate_training_data_dom.py` — walk annotated DOM, check `cc-select="true"` descendants
- **Result**: DOM labels are more accurate, model improved (model_dom.txt deployed over model.txt)

### 2. Text features (TF-IDF)
- Script: `data/prototype_text_classifier.py`
- Tried: GBM structural (0.0447 loss) vs TF-IDF+LogReg (0.0589) vs GBM+SVD-TF-IDF (0.0441)
- **Result**: Text features give <0.1pp improvement. Not worth the runtime cost.
- **Decision**: Structural features only.

### 3. Embedding features (model2vec)
- Script: `data/add_embeddings.py` — Potion-base-8M, 256-dim per block
- Trained: `model_emb.txt` (29MB, 296 features = 256 embedding + 40 structural)
- **Result**: No improvement over structural-only. Smaller model but not better.
- **Decision**: Don't use embeddings in production.

### 4. CRF sequence smoothing
- Scripts: `data/prototype_crf.py`, `data/prototype_crf_v2.py`
- Approach: CRF on block sequences using GBM probability + position + link ratio + tag type
- Learned: KEEP→KEEP slightly favored, DISCARD→DISCARD moderately favored
- **Result**: CRF flipped ~5% of blocks but net-zero improvement. Some blocks corrected, others broken.
- **Decision**: Don't use CRF. Simple GBM thresholding is good enough.

### 5. Label noise estimation + cleaning
- Script: `data/estimate_label_noise.py` — confident learning on 5-fold OOF predictions
- Found: ~2,390 noisy labels (0.54% of dataset)
- LLM audit (`data/audit_labels_llm.py`): sampled 50 high-confidence KEEP labels where OOF P(keep)<0.05 → **63% error rate**
- Script: `data/relabel_noisy.py` — flipped 1,213 extreme KEEP labels (P<0.05)
- **Result**: Cleaner labels, retrained model uses cleaned data. Marginal improvement.

### 6. LLM block classification (Qwen3.5-27B via vLLM)
- Scripts: `data/test_llm_classifier.py`, `data/test_llm_classifier_v2.py`
- Prompted with ancestor chain, page position, surrounding block context
- **Result**: ~90% block accuracy, but slow and infrastructure-dependent. Runs incomplete.
- **Used for**: Label auditing, not production.

### 7. DeepSeek V3.2 + MinerU-HTML pipeline (Dripper reproduction)
- Script: `eval/test_deepseek_pipeline.py`
- Full pipeline: MinerU-HTML simplify_html → Dripper v0 prompt → DeepSeek V3.2 on Bedrock → parse → reconstruct → html2text → jieba ROUGE-5
- 18 pages: F1=0.8161 (jieba tokenization). Paper reports 0.9098 on full set.
- **Conclusion**: Pipeline works. Used as-is for CC labeling.

### 8. Potion static embedding classifier
- Script: `data/train_potion_classifier.py`
- Trained StaticModelForClassification (MLP head on potion-base-8M embeddings) on WebMainBench blocks
- **Result**: 0.618 ROUGE-5 F1 on 200 val pages, 75.6% block accuracy
- 19pp behind GBM (0.806) — structural features are 3x more predictive than embeddings for block classification
- **Conclusion**: Confirms structural approach is correct. Embeddings alone can't solve this task.

### 9. Dripper double-check of DeepSeek labels
- Script: `data/doublecheck_with_dripper.py`
- Uses Dripper 0.6B (opendatalab/MinerU-HTML-v1.1-hunyuan0.5B-compact) via local vLLM
- Key implementation details (had to match Dripper's actual pipeline):
  - Prompt version: `short_compact` (not `v0`) — Dripper's actual default
  - `guided_regex` for constrained decoding: `<answer>\s*1(main|other)2(main|other)...\s*</answer>`
  - vLLM context: 64K tokens (0.5B model fits easily on single A100)
- Pilot (20 pages): 90.3% agreement, Dripper tends to be more permissive (labels more as main)
- **Strategy**: For block-level GBM training, only keep blocks where both models agree. Discard ~10% disagreeing blocks — each block is an independent training sample, so this is safe.

### 10. Combined WMB+CC GBM training
- Script: `data/generate_cc_training_data.py` — CC feature extraction pipeline
  - Filters to pages with >=70% Dripper agreement (14,959 of 15,880 pages)
  - Simplifies HTML with MinerU-HTML to get block texts per _item_id
  - Runs pulpie `export_features` on raw HTML for Rust features
  - Matches by normalized text → 346,959 matched blocks (57% match rate)
  - KEEP: 164,591 (47.4%) / DISCARD: 182,368 (52.6%)
- Script: `data/train_combined.py` — trains on WMB + CC
  - Combined: 793,468 blocks (446K WMB + 347K CC)
  - CV best rounds: 7,696 (up from 3,083 on WMB-only)
  - Held-out block accuracy: 96%
  - Model: `model_combined.txt` (102 MB)
- **ROUGE-5 result: 0.808** — only +0.2pp over WMB-only (0.806)
- **Conclusion**: More data from the same feature space doesn't fix structural limitations. GBM is saturated.

### 11. Qrater clean rate comparison (pulpie vs Dripper)
- Script: `eval/eval_qrater_dripper.py`
- Ran full Dripper pipeline (simplify → Dripper 0.6B via vLLM → reconstruct → html2text) on 500 WMB pages
- Scored all outputs with qrater EuroBERT-210m clean/dirty classifier
- **Results**:
  - Raw html2text: 5.6% clean
  - Pulpie GBM: 66.0% clean (+60pp over raw)
  - Dripper 0.6B: 80.0% clean (+14pp over pulpie)
  - Hard pages: pulpie 58% vs Dripper 80% (22pp gap)
- **Conclusion**: GBM approach is saturated at ~66% qrater clean rate. The 14pp gap to Dripper is architectural — need a model that sees full page context to close it.

### 12. Empty page analysis
- Script: `eval/test_empty_pages.py`
- DeepSeek produces 0 empty pages on WebMainBench (always finds some main content)
- On CC, expect ~10-20% empty pages (login walls, image galleries, etc.)
- **Decision**: Include empties in training — they provide all-negative examples the GBM currently lacks.

## Overfitting Problem

The reason for the CC labeling effort:

| Split | F1 |
|-------|-----|
| Full WebMainBench (train+test) | 0.806 |
| Held-out test set | 0.723 |
| **Gap** | **8.3pp** |

GBM is massively overfitting to WebMainBench's page distribution. Need diverse CC data with zero eval-set overlap. Dripper used 986K DeepSeek-labeled CC pages for their 0.6B model.

## Key Decisions

1. **Structural features only** — text/embeddings don't help enough to justify runtime cost
2. **DOM-based label projection** — more accurate than word overlap
3. **html2text canonicalization for eval** — recovers 3pp from formatting, matches paper methodology
4. **No CRF/sequence models** — didn't improve over simple thresholding
5. **CC labeling with DeepSeek V3.2** — address overfitting with diverse training data
6. **MinerU-HTML simplification + Dripper v0 prompt** — proven pipeline, skip custom simplification

## Next Steps

### GBM approach: saturated (done)
- Combined WMB+CC training gave +0.2pp ROUGE (0.806 → 0.808) — diminishing returns
- 14pp qrater clean rate gap to Dripper (66% vs 80%) is architectural, not data-limited
- More data from the same feature space won't close it

### 13. [BLOCK] marker encoder classifier (Qwen3-Embedding-0.6B)
- **Architecture**: PL-Marker approach — insert `[BLOCK]` token before each `_item_id` in simplified HTML, classify binary (main/other) at marker positions only, IGNORE_LABEL (-100) everywhere else
- **Model**: Qwen3-Embedding-0.6B with bidirectional SDPA (is_causal=False + dict mask bypass)
- **Training**: 14,959 CC pages (≥70% Dripper agreement), 3 epochs on 4×A100, 38 min
  - Block F1=0.840 on held-out CC val set
  - Class weights: other=0.700, main=1.748 (28.6% main rate)
- **BIO approach tried first**: Extreme class imbalance (B-MAIN=0.7% of tokens), F1=0.014 even with class weights → abandoned
- **Bidirectional attention challenge**: Qwen3 applies causal masking despite `is_decoder: False`
  - 4D zero mask works but prevents SDPA efficient kernels (O(n²) memory)
  - Dict mask `{"full_attention": None}` bypasses `create_causal_mask` in Qwen3Model.forward
  - `is_causal=False` on all attention modules → SDPA runs bidirectional with efficient kernels
- Scripts: `data/train_bio_classifier.py`, `eval/eval_block_classifier.py`
- Model saved: `data/block_classifier/final/` (1.1GB)

**WebMainBench ROUGE-5 F1 (6,647 English pages):**

| Method | All | Simple | Mid | Hard |
|--------|-----|--------|-----|------|
| DeepSeek-V3.2 (LLM) | 0.9098 | 0.9415 | 0.9104 | 0.8771 |
| GPT-4 (LLM) | 0.9024 | 0.9382 | 0.9042 | 0.8638 |
| Dripper 0.6B | 0.8779 | 0.9205 | 0.8804 | 0.8313 |
| Pulpie GBM (h2t) | 0.808 | 0.885 | 0.805 | 0.740 |
| magic-html | 0.714 | 0.786 | 0.712 | 0.643 |
| **[BLOCK] classifier** | **0.687** | **0.772** | **0.698** | **0.594** |
| Readability | 0.654 | 0.742 | 0.655 | 0.565 |
| Trafilatura | 0.640 | 0.731 | 0.642 | 0.547 |

- **12pp below GBM** (0.687 vs 0.808) — worse, not better
- 222 empty extractions (3.3%), 428 zero-score pages (6.4%)
- Main block rate: 19.6% — model too conservative (under-recalling)
- Throughput: 6.8 pg/s (single A100, BS=1), 148 ms/page avg

**Likely issues:**
1. Trained only on CC data, never saw WMB pages — distribution mismatch
2. Bidirectional attention may not be working correctly (needs verification)
3. 3 epochs on 15K pages may be undertrained for 0.6B model
4. No WMB-based eval during training — block F1 on CC doesn't reflect end-to-end ROUGE-5

### Error-mode-specific improvements (lower priority)
- Boilerplate leaking (~2pp) — encoder should handle this natively
- Forum/multi-section recall (~1pp) — encoder sees full page, should generalize better

## Timeline

| Date | Event |
|------|-------|
| Apr 7 | Initial commit; core Rust pipeline, baseline GBM (word-overlap labels) |
| Apr 7 | Feature engineering, hyperparameter tuning (7 configs), embedding experiments |
| Apr 8 | DOM-based label projection, GBM retrained → model_dom.txt (deployed) |
| Apr 8 | Text classifier experiments (TF-IDF, CRF) — found: not helpful |
| Apr 8 | Label noise analysis + cleaning (confident learning, LLM audit, 1,213 labels flipped) |
| Apr 9 | Error analysis on 500-page eval set, categorized failure modes |
| Apr 9 | LLM classification experiments (Qwen3.5-27B via vLLM) — incomplete |
| Apr 10 | Studied Dripper paper, MinerU-HTML pipeline, scoring methodology |
| Apr 10 | DeepSeek V3.2 pipeline validation (F1=0.8161 on 18 pages) |
| Apr 10 | CC-MAIN-2026-12 download (3 WARCs), sampled 16,670 English pages |
| Apr 10 | Launched CC labeling job (tmux "label", ~15h, ~$130) |
| Apr 11 | CC labeling complete: 16,376 OK, $129.23, 5.1h |
| Apr 15 | Filtered CC data → 15,880 pages, 1.23M blocks |
| Apr 15 | Quality audit (20 pages) → 85% GOOD/ACCEPTABLE |
| Apr 15 | Truncation investigation → 43 pages dropped (model attention, not hard truncation) |
| Apr 15 | Potion embedding classifier → 0.618 F1 (confirms structural features dominate) |
| Apr 15 | Dripper double-check complete: 93.3% agreement, 15,848 pages OK |
| Apr 16 | CC feature extraction → 347K blocks from 14,959 high-agreement pages |
| Apr 16 | Combined GBM training: 793K blocks, 7696 rounds → 0.808 ROUGE-5 (+0.2pp, saturated) |
| Apr 16 | Qrater comparison: pulpie 66% vs Dripper 80% clean rate — 14pp architectural gap confirmed |
| Apr 16 | [BLOCK] classifier trained: Qwen3-0.6B, PL-Marker, 4×A100, 38 min, block F1=0.840 |
| Apr 16 | [BLOCK] classifier eval: 0.687 ROUGE-5 on WMB — 12pp below GBM (0.808), needs investigation |
| Apr 17 | EuroBERT-2.1B block classifier trained: <|sep|> chunking, 8K context, plateaued at 0.864 ROUGE-5 |
| Apr 17 | Uploaded pulpie-orange-large-v1 to HuggingFace (chonkie-ai, private) |
| Apr 18 | DeepSeek V3.2 label quality verification: v0=0.840, short_compact=0.865, compact=0.862, v1=0.847, v2=0.835 |
| Apr 19 | Dripper native vLLM eval: 0.854 (valid only) on 200 English pages — below paper's 0.878 claim |
| Apr 20 | Error analysis: paper's 0.9001 uses jieba tokenization (+1-2pp), all languages, and eval conditions we can't reproduce |
| Apr 22 | EuroBERT 2.1B→610M distillation complete: 0.849 ROUGE-5, uploaded as orange-base-v1 |
| Apr 22 | EuroBERT 2.1B→210M distillation complete: 0.864 ROUGE-5 (matches teacher!), uploaded as orange-small-v1 |
| Apr 23 | 64-token truncation eval: -2.1pp quality, only 1.25x GPU speedup (HTML structure dominates tokens) |
| Apr 23 | Real-page throughput benchmark: 15.9 pps sequential, 43 pps GPU-only, CPU simplify_html is bottleneck |
| Apr 23 | Pipelined multi-GPU architecture designed: 4×A100 → ~170 pps, 1B in ~55-68 days |
| Apr 29 | Dripper vs Pulpie head-to-head: 5.4 vs 43 pps (8x), $77K vs $9.7K for 1B pages on A100 |
| Apr 29 | Cost analysis: Pulpie on L4 ~$6.1K vs Dripper on A100 ~$77K → ~13x cheaper (est.) |
| Apr 29 | L4 empirical benchmarks: Pulpie 15.1 pps, Dripper 0.92 pps — 16.4x gap confirmed |
| Apr 29 | 1B pages on 28× L4: ~27 days, ~$6.5K (RunPod). Dripper on L4 unusable: $105K for same job |

### 14. Pulpie Orange Large (EuroBERT-2.1B, <|sep|> chunking)
- **Architecture**: EuroBERT-2.1B with `<|sep|>` token separating blocks, packed into 8K chunks
- **Training**: 14,959 CC pages, lr=2e-5, BS=4, grad_accum=2, class-weighted CE, gradient checkpointing
- **Result**: ROUGE-5 = 0.864 on 200 English WMB pages (plateaued by epoch 0.75)
- **Key insight**: Trained on DeepSeek v0 labels (0.840 quality) but EXCEEDS its teacher by +2.4pp
- Scripts: `data/train_eurobert_classifier.py`, model: `data/block_classifier_eurobert_2.1B/checkpoint-5250`
- Uploaded: `chonkie-ai/pulpie-orange-large-v1` (private)

### 15. Dripper / MinerU-HTML reproduction & label ceiling investigation
- **Goal**: Reproduce Dripper's paper score (0.878) and understand the label quality ceiling
- **Method**: Native vLLM with StructuredOutputsParams, short_compact prompt, 32K context
- **Result**: 0.854 on 200 English pages (valid only), 0.828 on all 200 (6 too long)
- Also ran with their full MinerUHTML class (256K context, trafilatura fallback): similar scores

**DeepSeek V3.2 prompt comparison (200 English pages):**

| Prompt | ROUGE-5 |
|--------|---------|
| short_compact | **0.865** |
| compact | 0.862 |
| v1 (Chinese) | 0.847 |
| v0 (current labels) | 0.840 |
| v2 (Chinese) | 0.835 |

**Controlled comparison (same 200 English pages):**

| Method | All | Simple | Mid | Hard |
|--------|-----|--------|-----|------|
| DeepSeek V3.2 (short_compact) | 0.865 | 0.932 | 0.875 | 0.786 |
| Pulpie Orange Large (2.1B) | 0.862 | 0.928 | 0.856 | 0.807 |
| Dripper native (valid only) | 0.854 | 0.922 | 0.868 | 0.768 |
| Pulpie Orange Base (0.6B) | 0.847 | 0.907 | 0.848 | 0.787 |
| DeepSeek V3.2 (v0, current labels) | 0.840 | 0.930 | 0.823 | 0.774 |

**Key findings on paper's 0.9001 claim:**
1. Jieba tokenization (designed for Chinese) gives +1-2pp on English text vs whitespace split
2. Paper evaluates on ALL 7809 pages (1162 non-English where jieba is native)
3. Even with their exact pipeline class (256K context, trafilatura fallback), we get 0.77-0.85 on English
4. The 0.9001 is measured under evaluation conditions we cannot fully reproduce on our English subset
5. No better labeling model exists — DeepSeek short_compact (0.865) is our practical ceiling

### 16. Removing heuristic pruning — letting GBM learn from all blocks
- Removed nav/footer/aside/header pruning from clean.rs
- Added 4 boolean features: in_nav, in_footer, in_aside, in_header (30 features total)
- Added nav/footer/aside/header to CONTAINER_TAGS in segment.rs
- Trained v3 models on WMB, CC, and combined data
- **Result**: v3 (no pruning, 30 features) WORSE than v2 (pruning + 26 features)
  - v3 WMB: 0.780 vs v2 WMB: 0.805
  - v3 combined: 0.780 vs v2: 0.806
- **Conclusion**: GBM can't learn nav/footer filtering as well as hard-coded pruning rules

### 17. GBM generalization failure
- CC-only GBM on WMB (honest out-of-sample): **0.68 ROUGE-5**
- magic-html (zero training, pure heuristics): **0.71 ROUGE-5**
- The GBM loses to hand-written rules on unseen data
- 30 numeric features encode the same signals magic-html hard-codes in XPaths — the model can't read text
- **The 0.80 ROUGE reported earlier was trained on the test set — not a real score**

### 18. GPU inference benchmarks (EuroBERT-210M, torch.compile + SDPA)
- Script: `data/bench_gpu_compile.py`
- Tested on A100 with torch.compile(mode="reduce-overhead") + attn_implementation="sdpa"
- Model uses only 433MB VRAM — fits on any GPU

**A100 throughput (best batch size per seq length):**

| SeqLen | Pages/sec | ms/page | Peak VRAM |
|--------|-----------|---------|-----------|
| 512 | 508 | 1.97ms | 870MB |
| 1024 | 220 | 4.54ms | 1.4GB |
| 2048 | 84 | 11.88ms | 2.8GB |
| 4096 | 28 | 36ms | 6.8GB |
| 8192 | 8.2 | 122ms | 20GB |

**Cost per 1B pages (RTX 3090 @ $0.22/hr RunPod):**

| SeqLen | $/1B pages |
|--------|------------|
| 512 | $267 |
| 1024 | $616 |
| 2048 | $1,613 |
| 4096 | $4,894 |
| 8192 | $16,533 |

Quadratic attention scaling: seq=8192 is 27x more expensive than seq=1024.

### 19. Cascade / pre-filter analysis

**GBM as pre-filter (CC model on WMB, out-of-sample):**
- At 99.5% content recall, GBM only strips 8% of text — useless as pre-filter
- Distribution shift kills confidence: CC model isn't sure enough about WMB pages

**Heuristic pre-filter (nav/footer/aside/header + link_ratio>0.5):**
- Strips 38% of blocks but only 14% of text (boilerplate blocks are short)
- Content loss: 0.31% (semantic tags only) to 2.25% (with link ratio)
- **Fundamental limit**: boilerplate is only 32% of total text chars. Even perfect pre-filtering = ~32% seq reduction.

### 20. Simplified HTML token budget analysis

With 128 tokens/block truncation (Dripper uses 500 chars ≈ 125 tokens):

| Percentile | Tokens/page (raw) | With heuristic pre-filter |
|------------|-------------------|--------------------------|
| p25 | 942 | 511 |
| p50 | 1,900 | 1,150 |
| p75 | 3,476 | 2,273 |
| p90 | 6,539 | 4,799 |
| p95 | 10,241 | 8,282 |

Pages fitting in context window (with pre-filter):
- 2048 tokens: 72% of pages
- 4096 tokens: 88% of pages
- 8192 tokens: 95% of pages

### 21. Production inference plan: 1B pages in 30 days

**Target**: 1B pages, 30 days, <$5K budget

**Winning config: EuroBERT-210M + 64 tok/block + heuristic pre-filter on 8× L4 GPUs**
- 66 pages/sec per GPU × 8 = 531 pages/sec sustained
- 1.38B pages in 30 days (38% headroom)
- Most pages fit in seq=1024-2048 range with 64 tok/block truncation

| Config (8× L4) | Pages/30d | Feasible? |
|-----------------|-----------|-----------|
| EuroBERT-210M, 128 tok/block | 0.34B | NO |
| **EuroBERT-210M, 64 tok/block** | **1.38B** | **YES** |
| DistilBERT-66M, 128 tok/block | 1.01B | barely |
| DistilBERT-66M, 64 tok/block | 4.13B | overkill |
| EuroBERT-610M, 128 tok/block | 0.12B | NO |
| EuroBERT-610M, 64 tok/block | 0.48B | NO |

**Key decisions:**
1. **Token-based truncation, not character-based**: Dripper uses 500 chars/block — imprecise. Use tokenizer.encode()[:N] for exact control. Tokenizers are fast (~millions tok/sec), negligible overhead.
2. **64 tokens/block** (~250-300 chars): Halves seq length vs 128. Classification signal is usually in first few words + HTML structure. Needs empirical validation.
3. **Heuristic pre-filter before transformer**: Strip nav/footer/aside/header + high-link-ratio blocks. Drops block count 38%, saves ~22% compute (quadratic scaling).
4. **Test truncation levels**: Benchmark 32, 64, 128 tokens/block on ROUGE-5 to find quality/speed sweet spot.
5. **Dripper's list simplification**: ul/ol with >2 items → keep first + last + "...". Saves many blocks on nav menus and long lists. We should adopt this.

### 22. EuroBERT 2.1B → 610M distillation (complete)
- Script: `data/train_distill_eurobert.py`
- Teacher: EuroBERT-2.1B checkpoint-5250 (0.864 ROUGE-5)
- Student: EuroBERT-610M (randomly initialized classifier head)
- 4× A100, alpha=0.7 (KL weight), temperature=2.0
- 3,499 steps, ~2.15 sec/step, ~2 hours
- **Result**: ROUGE-5 = 0.849 (All), 0.889 (Simple), 0.842 (Mid), 0.833 (Hard)
- F1=0.925, Precision=0.917, Recall=0.933
- Model saved: `data/block_classifier_eurobert_610m_distill/final/`
- Uploaded: `chonkie-ai/pulpie-orange-base-v1` (replaced Qwen3 model)

### 23. EuroBERT 2.1B → 210M distillation (complete)
- Script: `data/train_distill_eurobert_210m.py`
- Same setup as 610M: alpha=0.7, temperature=2.0, 4× A100
- 3,499 steps, ~2.15 sec/step, ~2h5m
- **Result**: ROUGE-5 = 0.864 (All), 0.885 (Simple), 0.841 (Mid), 0.866 (Hard)
- F1=0.925, Precision=0.915, Recall=0.936
- **Matches the 2.1B teacher** (0.864 vs 0.864) at 1/10th the size
- Actually beats 610M distilled (0.849) — smaller model learned better
- Model saved: `data/block_classifier_eurobert_210m_distill/final/`
- Uploaded: `chonkie-ai/pulpie-orange-small-v1` (private)

### 24. 64-token truncation quality evaluation
- Tested 2.1B teacher and 210M distilled on WMB with full (500-char) vs 64-token block truncation
- Script: inline eval on 201 WMB pages

**ROUGE-5 comparison:**

| Model | Truncation | All | Simple | Mid | Hard |
|-------|-----------|------|--------|-----|------|
| 2.1B teacher | Full (500-char) | 0.864 | 0.891 | 0.872 | 0.829 |
| 2.1B teacher | 64-token | 0.839 | 0.873 | 0.844 | 0.799 |
| 210M distilled | Full (500-char) | 0.864 | 0.885 | 0.841 | 0.866 |
| 210M distilled | 64-token | 0.843 | 0.869 | 0.823 | 0.838 |

- **64-token truncation costs ~2.1-2.5 pp** across both models
- Consistent degradation — not model-dependent, information loss from truncation
- Even at 64-tok, 210M (0.843) slightly beats the 2.1B (0.839)
- **Opportunity**: Re-distill on 64-token truncated data to recover ~1-1.5pp (train/eval distribution match)

### 25. Real-page throughput benchmark (EuroBERT-210M)
- Script: `data/bench_throughput_real.py`
- Tested on 500 real CC pages (avg 120KB HTML) with length-bucketed batching

**End-to-end pipeline breakdown (single A100, SDPA, torch.compile):**

| Stage | Time | % of total |
|-------|------|------------|
| simplify_html (CPU) | 13.2s | 42% |
| extract+tokenize+chunk (CPU) | 6.5s | 21% |
| GPU inference | 11.7s | 37% |

- **End-to-end: 15.9 pages/sec** (sequential CPU→GPU)
- GPU alone: 190K tokens/sec, 51 chunks/sec (~43 pages/sec)
- **CPU is the bottleneck** — simplify_html at 38 pages/sec single-threaded

**Key optimizations discovered:**
1. **Length-bucketed batching**: Sorting chunks by length before batching → 5x speedup (padding waste 50% → 4.3%)
2. **Dynamic batch sizing**: Cap batch×seq_len ≤ 128K to avoid OOM on long sequences

**Real page statistics (500 CC pages):**
- Blocks/page: median=57, mean=91, p95=246
- Chunks/page: median=1, mean=1.2, max=8
- Tokens/chunk: median=2,992, mean=3,728, p95=8,177

### 26. 64-token truncation does NOT give 3x GPU speedup
- Full (500-char): 595 chunks, 2.22M tokens, 14.0s → 35.7 pages/sec GPU-only
- 64-token: 565 chunks, 1.82M tokens, 11.2s → 44.6 pages/sec GPU-only
- **Only 1.25x speedup**, not the 3x estimated earlier
- Reason: HTML tags and structure dominate token count, not text content
  - 500→128 char cutoff only drops 7% of tokens
  - 64-token truncation drops 18% of tokens
  - The simplified HTML structure (tags, attributes, nesting) itself is token-heavy regardless of text truncation

**CPU time for simplify_html is identical** regardless of cutoff (500/256/128 chars all ~14s) — the cost is DOM parsing, not truncation.

### 27. Production architecture: pipelined CPU→multi-GPU

**Sequential (current)**: 15.9 pages/sec on single GPU — CPU idle during inference, GPU idle during preprocessing

**Pipelined architecture (not yet implemented):**
```
CPU pool (8 workers, ~280 pps) → Queue per GPU → 4 GPUs (~43 pps each)
```
- 4×A100: ~170 pages/sec → 1B pages in 68 days
- With 64-tok truncation: ~210 pps → ~55 days
- CPU (8 cores × 35 pps = 280 pps) has headroom until ~7 GPUs

**1B pages in 30 days requires:**
- 4×A100 + CPU pipeline: ~55-68 days (need ~6-8 GPUs)
- 8×L4 equivalent: similar compute, cheaper per GPU-hour

### 28. Dripper vs Pulpie head-to-head throughput (same GPU, same data)

**Setup**: 500 real CC pages, single A100 80GB, SDPA attention

| Model | Architecture | Throughput (GPU) | GPU-hours/1B pages | Notes |
|-------|-------------|-----------------|-------------------|-------|
| Dripper 0.6B | Autoregressive (vLLM) | 5.38 pps | 51,632 | Memory-bandwidth bound |
| **Pulpie 210M** | **Encoder (SDPA)** | **43 pps** | **6,460** | **Compute-bound** |
| **Speedup** | | **8x** | **8x** | |

**Why the 8x gap**: Dripper generates tokens autoregressively (sequential decoding, KV-cache bound by HBM bandwidth). Pulpie classifies in a single forward pass with batched encoder inference. Encoders are compute-bound → scale with FLOPS, not memory bandwidth.

**Cost comparison for 1B pages:**

| Setup | Throughput | GPU-hours | $/GPU-hr | Total cost |
|-------|-----------|-----------|----------|------------|
| Dripper on A100 | 5.4 pps | 51,632 | $1.50 | **~$77K** |
| Pulpie on A100 | 43 pps | 6,460 | $1.50 | **~$9.7K** |
| Pulpie on L4 (est.) | ~16 pps | ~17,360 | $0.35 | **~$6.1K** |

- A100 vs A100: **8x cheaper** (same hardware, raw throughput advantage)
- Pulpie on L4 vs Dripper on A100: **~13x cheaper** (cheaper hardware + still good throughput)
- Dripper **cannot** run efficiently on L4s — autoregressive decoding needs high memory bandwidth (A100: 2 TB/s, L4: 300 GB/s). Throughput would collapse to <1 pps.
- Pulpie 210M is compute-bound and fits in 433MB VRAM — runs on any GPU class (L4, T4, RTX 3090)

**L4 throughput estimate** (not yet benchmarked, extrapolated from FLOPS ratio):
- A100: 312 TFLOPS FP16 → 43 pps
- L4: ~120 TFLOPS FP16 → ~16-17 pps (proportional scaling expected for compute-bound workloads)
- Needs empirical validation — actual L4 tensor core utilization with SDPA may differ

## Key Insight: The Economics of Web-Scale Content Extraction

Feature-based (GBM) and heuristic approaches are effectively free at web scale but cap at 0.68-0.71 ROUGE on unseen data. Transformer-based classification closes the gap to 0.85+ but introduces GPU cost.

The 210M distilled model is the sweet spot — matches the 2.1B teacher at 0.864 ROUGE with 10x fewer parameters and ~40 pages/sec per GPU. The 610M model is actually worse (0.849) despite being 3x larger.

64-token truncation gives a modest 1.25x throughput gain for 2.1pp quality loss. The bigger lever is pipelining CPU and GPU work (currently sequential) and multi-GPU parallelism. The simplify_html CPU stage (28ms/page) is the true bottleneck at scale, not the GPU.

Head-to-head against Dripper 0.6B on the same A100: **8x faster** (43 vs 5.4 pps). On L4 GPUs (empirically validated): **16.4x faster** (15.1 vs 0.92 pps) — the gap widens because autoregressive decoding collapses on L4's low memory bandwidth (300 GB/s vs A100's 2 TB/s). Cost for 1B pages: **$6.5K (Pulpie on L4) vs $105K (Dripper on L4) vs $77K (Dripper on A100)**. Dripper's autoregressive architecture is locked to high-bandwidth GPUs.

### 29. L4 GPU benchmarks — real CC pages (empirical, 500 pages)

**Setup**: 8× NVIDIA L4 (23GB each), 500 real Common Crawl pages (median 40KB HTML), EuroBERT-210M distilled (Orange Small), bf16 + SDPA, no torch.compile.

**Pulpie Orange Small (210M) — end-to-end pipeline:**

| Stage | Time (500pg) | % | Rate |
|-------|-------------|---|------|
| simplify_html (CPU) | 13.1s | 24% | 38 pps |
| extract+chunk (CPU) | 5.8s | 11% | 86 pps |
| GPU inference | 33.2s | 65% | 15.1 pps |
| **Total (sequential)** | **53.4s** | | **9.3 pps** |

- Best config: bf16 + SDPA + max_batch_tokens=16384 (length-sorted batching)
- bf16 is 13% faster than fp16 on L4 tensor cores
- Smaller batch token budgets better: 16K > 32K > 64K (less padding waste)
- Real-page token distribution: median=2,692, mean=3,593, p95=8,180 tokens/chunk
- 37% of chunks land in the 8K bucket — quadratic attention makes these 9x more expensive than median chunks
- SDPA 4D mask fix: model creates [B,1,S,S] expanded mask that prevents efficient SDPA kernel. Patched to pass None (no padding) or [B,1,1,S] (with padding). Saves ~3GB VRAM but no throughput change — efficient kernel was already dispatched.
- torch.compile with CUDA graphs: same 14.2 pps as eager (dynamic shapes cause excessive graph captures). Fixed-bucket approach with multiple compiled models was worse (7.8-9.2 pps) due to padding waste.

**Pulpie vs A100 comparison:**

| Metric | A100 (work log) | L4 (measured) | L4/A100 |
|--------|----------------|---------------|---------|
| GPU-only pps | 43 | 15.1 | 35% |
| Sequential pps | 15.9 | 9.3 | 58% |
| simplify_html pps | 38 | 38 | same |

GPU-only is 35% of A100, worse than the FLOPS-ratio estimate of 40%. Real pages have heavy-tail long sequences where quadratic attention scaling hurts L4 more.

**Dripper 0.6B on L4 — vLLM benchmark (100 pages):**

| Metric | Value |
|--------|-------|
| Pages/sec | 0.92 |
| Output tok/s | 145 |
| Avg prompt tokens | 4,372 |
| Avg output tokens | 157 |
| GPU time % | 95% |

Dripper on L4 is 17% of its A100 speed (0.92 vs 5.38 pps), matching the memory bandwidth ratio (300/2039 GB/s = 15%). Guided decoding regex adds overhead on top.

**Head-to-head cost comparison for 1B pages:**

| Setup | pps | GPU-hours | RunPod ($0.35/hr) | GCP ($0.80/hr) |
|-------|-----|-----------|-------------------|----------------|
| **Pulpie 210M on L4** | **15.1** | **18,400** | **$6,500** | **$14,700** |
| Pulpie 210M on A100 | 43 | 6,460 | — | $9,700 |
| Dripper 0.6B on A100 | 5.38 | 51,600 | — | $77,000 |
| Dripper 0.6B on L4 | 0.92 | 301,000 | $105,000 | $241,000 |

- Pulpie on L4 vs Dripper on L4: **16.4x faster, 16.4x cheaper**
- Pulpie on L4 vs Dripper on A100: **12x cheaper** ($6.5K vs $77K)
- Dripper on L4 is worse than Dripper on A100 — cheaper GPU-hours but 6x more of them

**1B pages in 30 days (pipelined, 8 CPU workers):**

| GPUs | Effective pps | Days | RunPod cost |
|------|--------------|------|-------------|
| 8× L4 (Pulpie) | 121 | 96 | $6,500 |
| 28× L4 (Pulpie) | 423 | **27 days** | **$6,500** |
| 420× L4 (Dripper) | 386 | 30 days | $105,000 |

## TODO: Pulpie simplify.py Rewrite

**Problem**: Pulpie's `simplify.py` (369 lines) produces fundamentally different block segmentation
than MinerU-HTML's `simplify_html.py` (1172 lines) that the model was trained on. Using Pulpie's
pipeline scores 0.731 ROUGE-5 vs 0.862 using MinerU's pipeline — a 13pp quality gap.

**Root causes**:
1. **Block segmentation logic**: MinerU does deep recursive splitting with table/list type analysis
   (data table vs layout table), splits lists into individual `<li>` items. Pulpie treats lists as
   atomic blocks, producing 71 blocks where MinerU produces 92 on the same page.
2. **Tag removal**: MinerU removes `<nav>` tags. Pulpie doesn't. (But adding nav removal alone
   didn't help — went from 0.731 to 0.727.)
3. **Attribute handling**: MinerU uses allow-list (class, id, _item_id, charset only). Pulpie uses
   deny-list, leaking data-anno-uid, style, cc-select etc. causing token bloat. (But switching to
   allow-list alone also hurt — dropped to 0.708.)
4. **Combined effect**: The issues compound. The model expects MinerU's specific segmentation
   boundaries, and fixing attrs/nav in isolation without fixing segmentation makes things worse.

**Fix options**:
- Port MinerU's segmentation logic faithfully into Pulpie (significant rewrite, ~800 lines)
- Vendor MinerU-HTML as an optional dependency and use it directly
- Hybrid: keep Pulpie's clean API but swap simplify internals to match MinerU's output

**Priority**: High — this is the main blocker for Pulpie being usable as a standalone library.
The current workaround (using MinerU's simplify externally) works for eval but not for users.

## Model Zoo

| Name | HuggingFace | Size | ROUGE-5 | Notes |
|------|-------------|------|---------|-------|
| Orange Large | chonkie-ai/pulpie-orange-large-v1 | 2.1B | 0.864 | Teacher, trained on CC |
| **Orange Base** | **chonkie-ai/pulpie-orange-base-v1** | **610M** | **0.849** | Distilled from 2.1B |
| **Orange Small** | **chonkie-ai/pulpie-orange-small-v1** | **210M** | **0.864** | Distilled from 2.1B, best value |
| Espresso | chonkie-ai/pulpie-espresso-v1 | GBM | 0.68* | *Out-of-sample; 0.81 on test set |