|
1 | | -# Lev: one checkpoint, multiple depth operating points |
| 1 | +<p align="center"> |
| 2 | + <a href="https://openlair.github.io/lev/"> |
| 3 | + <img src="https://openlair.github.io/lev/assets/social.png" alt="Lev: Thinking fast and deep. One checkpoint. Multiple depths." width="100%"> |
| 4 | + </a> |
| 5 | +</p> |
2 | 6 |
|
3 | | -[Project website](https://openlair.github.io/lev/) · [Paper](https://openlair.github.io/lev/#paper) · [GitHub](https://github.com/OpenLAIR/lev) · [Hugging Face collection — private](https://huggingface.co/collections/songdj/lev-thinking-fast-and-deep-with-looped-decision-models-6acefb135880c9d373cf42ba) |
| 7 | +<h1 align="center">Lev · System 1.5 Decision Models</h1> |
4 | 8 |
|
5 | | -Lev turns pretrained recurrent language models into typed decision models. A shared pointer head returns one probability per option after each recurrent step, without generating an answer string. The same checkpoint provides several accuracy–compute operating points. |
| 9 | +<p align="center"> |
| 10 | + <strong>Thinking Fast and Deep with Looped Decision Models</strong><br> |
| 11 | + One checkpoint. Multiple depth operating points. Probabilities over your options. |
| 12 | +</p> |
6 | 13 |
|
7 | | -**Research release candidate.** Source code is maintained under [OpenLAIR/lev](https://github.com/OpenLAIR/lev). The linked Hugging Face collection and seed-0 model repositories are private; dataset releases and the arXiv identifier remain pending. Weights and question text are not bundled. See [resource availability](docs/RESOURCES.md) and [release status](docs/RELEASE_STATUS.md). |
| 14 | +<p align="center"> |
| 15 | + <a href="docs/RELEASE_STATUS.md"><img src="https://img.shields.io/badge/research-release%20candidate-2448ff?style=flat-square" alt="Research release candidate"></a> |
| 16 | + <a href="https://openlair.github.io/lev/#paper"><img src="https://img.shields.io/badge/paper-forthcoming-5b6467?style=flat-square" alt="Paper forthcoming"></a> |
| 17 | + <a href="docs/VALIDATION.md"><img src="https://img.shields.io/badge/CPU%20tests-187%20passed-2448ff?style=flat-square" alt="CPU validation: 187 passed"></a> |
| 18 | + <a href="LICENSE"><img src="https://img.shields.io/badge/code-Apache%202.0-5b6467?style=flat-square" alt="Code license: Apache 2.0"></a> |
| 19 | +</p> |
8 | 20 |
|
9 | | -| Model | Backbone | Supported native depths | Hugging Face repository (private) | |
10 | | -|---|---|---|---| |
11 | | -| Lev-1.4B | Ouro-1.4B | 1–4 | [songdj/Lev-1.4B](https://huggingface.co/songdj/Lev-1.4B) | |
12 | | -| Lev-2.6B | Ouro-2.6B | 1–4 | [songdj/Lev-2.6B](https://huggingface.co/songdj/Lev-2.6B) | |
13 | | -| Lev-3B | Nanbeige4.2-3B-Base | 1–2 | [songdj/Lev-3B](https://huggingface.co/songdj/Lev-3B) | |
| 21 | +<p align="center"> |
| 22 | + <a href="https://openlair.github.io/lev/"><strong>Website</strong></a> · |
| 23 | + <a href="https://openlair.github.io/lev/#paper"><strong>Paper</strong></a> · |
| 24 | + <a href="https://huggingface.co/collections/songdj/lev-thinking-fast-and-deep-with-looped-decision-models-6acefb135880c9d373cf42ba"><strong>Models — private</strong></a> · |
| 25 | + <a href="#quick-start"><strong>Quick start</strong></a> · |
| 26 | + <a href="#reproduce-the-study"><strong>Reproduce</strong></a> · |
| 27 | + <a href="#citation"><strong>Citation</strong></a> |
| 28 | +</p> |
14 | 29 |
|
15 | | -On JevBench's 231 public tasks, Lev-2.6B improves from 63.2% at depth 1 to 76.2% at depth 4. An agreement stopper supports **offline** analysis of which questions could exit early. Its benefit over a matched fixed-depth frontier varies by workload: Lev-2.6B did not pass the locked LevBench G3 criterion. Latency comparisons depend on the measured implementation and hardware. |
| 30 | +> **Release status:** GitHub source and Hugging Face model repositories are private research candidates. The paper is forthcoming; no arXiv identifier has been assigned. The datasets have **not been uploaded**. [Resource availability →](docs/RESOURCES.md) |
16 | 31 |
|
17 | | - |
| 32 | +## One checkpoint, more room to think |
18 | 33 |
|
19 | | -*Figure 1 from the study; adaptive points summarize offline routing analysis.* |
| 34 | +Lev adapts pretrained recurrent language models into typed decision models. A shared pointer head returns a probability for each caller-provided option after each recurrent step. You choose the depth—and therefore the amount of computation—without switching checkpoints or generating an answer string. |
20 | 35 |
|
21 | | -## Install |
| 36 | +| Flexible depth | Measured accuracy | Structured decisions | |
| 37 | +|---|---|---| |
| 38 | +| **Up to 4 native depths** from the same checkpoint | **63.2% → 76.2%** for Lev-2.6B, depth 1 → 4 | **One probability per option** with a shared pointer head | |
22 | 39 |
|
23 | | -Use Linux and Python 3.12 or 3.13. Create a new environment; the distribution is named `lev`, while Python imports remain `kev`. |
| 40 | +The accuracy result uses JevBench's **231 public tasks** and fp32 readouts. Fixed-depth scaling is the primary result. The agreement stopper provides an additional **offline** routing analysis; its gains depend on the workload. **Lev-2.6B did not pass G3 on locked LevBench in any of its three study seeds.** See [scope and limitations](#scope-and-limitations). |
| 41 | + |
| 42 | +## Model family |
| 43 | + |
| 44 | +All linked model repositories currently require authorized access. The source package contains model cards and calibration metadata; weights are staged separately. |
| 45 | + |
| 46 | +| Model | Backbone | Native depths | JevBench accuracy, first → deepest | Artifacts | |
| 47 | +|---|---|---|---|---| |
| 48 | +| **Lev-1.4B** | Ouro-1.4B | 1–4 | 58.0% → 68.0% | [HF · private](https://huggingface.co/songdj/Lev-1.4B) · [Model card](model-cards/Lev-1.4B.md) | |
| 49 | +| **Lev-2.6B** | Ouro-2.6B | 1–4 | 63.2% → 76.2% | [HF · private](https://huggingface.co/songdj/Lev-2.6B) · [Model card](model-cards/Lev-2.6B.md) | |
| 50 | +| **Lev-3B** | Nanbeige4.2-3B-Base | 1–2 | 69.3% → 72.7% | [HF · private](https://huggingface.co/songdj/Lev-3B) · [Model card](model-cards/Lev-3B.md) | |
| 51 | + |
| 52 | +These are existing study results, not new evaluations of the release candidate. The cards document backbone revisions, seed coverage, calibration and limitations. |
| 53 | + |
| 54 | +## Quick start |
| 55 | + |
| 56 | +### 1. Install and run CPU tests |
| 57 | + |
| 58 | +Use Linux and Python 3.12 or 3.13. From the source checkout, create a separate environment and working copy. The distribution is named `lev`; Python imports remain `kev`. |
24 | 59 |
|
25 | 60 | ```bash |
26 | 61 | python3.13 -m venv ../lev-env |
27 | 62 | source ../lev-env/bin/activate |
28 | | -python -m pip install --upgrade pip |
| 63 | + |
| 64 | +# Use the CPU wheel for validation; no CUDA dependency download is needed. |
| 65 | +python -m pip install --no-cache-dir 'torch==2.8.0+cpu' \ |
| 66 | + --index-url https://download.pytorch.org/whl/cpu |
29 | 67 | cp -R . ../lev-work |
30 | | -python -m pip install '../lev-work[dev]' |
31 | | -(cd ../lev-work && PYTHONDONTWRITEBYTECODE=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python -m pytest -q -p no:cacheprovider) |
| 68 | +python -m pip install --no-cache-dir '../lev-work[dev]' |
| 69 | +python -m pip check |
| 70 | + |
| 71 | +(cd ../lev-work && PYTHONDONTWRITEBYTECODE=1 OMP_NUM_THREADS=1 \ |
| 72 | + MKL_NUM_THREADS=1 python -m pytest -q -p no:cacheprovider) |
32 | 73 | ``` |
33 | 74 |
|
34 | | -The CPU tests use tiny randomly initialized models and require no downloads or pretrained weights. [INSTALL.md](docs/INSTALL.md) records tested dependency versions, optional CUDA kernels for hybrid dense controls, and the pinned JevBench installation. The original environment's lockfile is not reused for this differently scoped package. |
| 75 | +The validated fresh CPU installation passed **187 tests**, with **14 opt-in network tests skipped**. Tests use tiny randomly initialized models and require no pretrained weights or dataset downloads. Exact dependency versions and evidence are in [validation](docs/VALIDATION.md). |
35 | 76 |
|
36 | | -## Fixed-depth quick start |
| 77 | +### 2. Score a request at a chosen depth |
37 | 78 |
|
38 | | -After obtaining a checkpoint directory containing `adapter_config.json`, adapter weights, and `head.pt`: |
| 79 | +For model inference, obtain the appropriate private checkpoint and prepare a **separate CUDA-capable PyTorch environment** using the [installation notes](docs/INSTALL.md). The CPU-only test environment above is for validation. A checkpoint needs its adapter configuration, adapter weights and `head.pt`; the loader fetches the pinned backbone, so cache it first for offline use. |
39 | 80 |
|
40 | 81 | ```bash |
41 | | -python examples/fixed_depth.py --checkpoint ../lev-weights/Lev-2.6B --depth 2 --device cuda |
| 82 | +python examples/fixed_depth.py \ |
| 83 | + --checkpoint ../lev-weights/Lev-2.6B \ |
| 84 | + --depth 2 --device cuda |
42 | 85 | ``` |
43 | 86 |
|
44 | | -The example uses an original two-option toy question. `Checkpoint.load` loads the pinned backbone from the Hub; cache it first for offline use. The CLI supports `--all-depths` to read all outputs in one full-depth pass. **All-depth output does not save computation.** A chosen fixed depth uses only that number of recurrent steps. |
| 87 | +The example scores an original two-option toy request. A chosen fixed depth executes only that many recurrent steps. `--all-depths` returns every depth's output from a full-depth pass; **it does not save computation**. Full-size GPU parity remains a release requirement. |
45 | 88 |
|
46 | | -```python |
47 | | -from kev.checkpoint import Checkpoint, LoadOptions |
| 89 | +For calibrated probabilities, verify checkpoint identity and add `--temperatures calibration/Lev-2.6B-s0.json`. Per-depth temperatures are separate from the scalar temperature in `head.pt`. Without this option, the example explicitly labels its scores as uncalibrated. |
48 | 90 |
|
49 | | -tokenizer, model = Checkpoint('../lev-weights/Lev-2.6B').load('cuda', LoadOptions(temperature=1.0)) |
50 | | -model.set_depth(2) |
| 91 | +## Reproduce the study |
| 92 | + |
| 93 | +| What you want to do | Start here | |
| 94 | +|---|---| |
| 95 | +| Understand the training recipe, readouts and offline stopper | [Reproduction guide](docs/REPRODUCING.md) | |
| 96 | +| Redraw available paper figures from aggregate results | [Figure inventory and commands](figures/README.md) | |
| 97 | +| Inspect data sources, hashes and reconstruction prerequisites | [Data guide](docs/DATA.md) · [Dataset cards](data-cards/) | |
| 98 | +| Review checkpoint details and per-depth calibration | [Model cards](model-cards/) · [Calibration files](calibration/) | |
| 99 | +| Inspect installation checks and release conditions | [Validation](docs/VALIDATION.md) · [Release gates](docs/RELEASE_STATUS.md) | |
| 100 | + |
| 101 | +**Seven of thirteen paper figures** can currently be redrawn from the packaged aggregates: |
| 102 | + |
| 103 | +```bash |
| 104 | +PYTHONDONTWRITEBYTECODE=1 python figures/reproduce.py \ |
| 105 | + --output-dir ../lev-figures |
51 | 106 | ``` |
52 | 107 |
|
53 | | -Per-depth calibration temperatures are a separate JSON artifact, not implied by `head.pt`'s scalar temperature. The existing seed-0 calibration JSON files are in `calibration/`; for Lev-2.6B, supply `--temperatures calibration/Lev-2.6B-s0.json` after verifying checkpoint identity. Without it, the example prints uncalibrated softmax scores. |
| 108 | +The redraws preserve the study's table-value assertions. They do not recompute confidence intervals from the complete per-question collection. Dataset text and that full collection are not bundled or uploaded; source terms and reconstruction prerequisites remain under review. |
| 109 | + |
| 110 | +## Scope and limitations |
54 | 111 |
|
55 | | -## Reproduction |
| 112 | +- **Adaptive routing is offline.** The saved all-depth logits support analysis of early exits. Online hidden-state continuation and adaptive serving are not implemented. The paper's stopper is cross-fitted per workload, rather than supplied as a universal checkpoint threshold. |
| 113 | +- **Gains are workload-dependent.** Lev-2.6B failed the locked LevBench G3 criterion in all three study seeds. The Lev-1.4B and Lev-3B headline seeds passed there; that does not establish a general advantage over fixed-depth frontiers. Greater depth can also overturn an earlier correct decision. |
| 114 | +- **Accuracy and timing have different measurement paths.** Accuracy uses fp32 readouts; reported H200 timing uses bf16, batch one and implementation-specific kernels. Latency comparisons are not portable speed guarantees. G3 and the paper's token-weighted cost intervals are different analyses. |
| 115 | +- **Release readiness is tracked separately.** CPU installation and tests have passed. Full-size GPU parity, final public weight identities, data rights, complete reconstruction, all-figure reproduction, public endpoints and author review remain gated. Never use the fixed locked test to choose a new rule or checkpoint. |
56 | 116 |
|
57 | | -- [Data and exact-input requirements](docs/DATA.md): source revisions, manifest hashes, training subsets, and distribution restrictions. |
58 | | -- [Training, readouts and offline stopping](docs/REPRODUCING.md): portable commands and the distinction between cross-fitting and deployment calibration. |
59 | | -- [Paper figures](figures/README.md): runnable aggregate-input redraws and an honest inventory of remaining inputs. |
60 | | -- [Model cards](model-cards/) and [dataset cards](data-cards/): release drafts, provenance and limitations. |
61 | | -- [Source and weight licenses](docs/LICENSING.md): source code is Apache-2.0; dataset and backbone terms are separate. |
| 117 | +[Full reproduction details](docs/REPRODUCING.md) · [Current release conditions](docs/RELEASE_STATUS.md) |
| 118 | + |
| 119 | +### Verify a source artifact |
| 120 | + |
| 121 | +The checker requires **Gitleaks 8.30.1** and PyYAML. Run it against a pristine source snapshot; see the [snapshot instructions](docs/RELEASE_STATUS.md) for Git checkouts. Environments, caches, downloads and generated outputs must remain outside that snapshot—nothing is silently ignored. |
62 | 122 |
|
63 | 123 | ```bash |
64 | | -PYTHONDONTWRITEBYTECODE=1 python figures/reproduce.py --output-dir ../lev-figures |
65 | 124 | PYTHONDONTWRITEBYTECODE=1 python scripts/verify_release.py --mode hygiene |
66 | | -# Default release mode intentionally fails while readiness conditions remain blocked. |
| 125 | + |
| 126 | +# Formal release verification remains blocked while required conditions are open. |
67 | 127 | PYTHONDONTWRITEBYTECODE=1 python scripts/verify_release.py |
68 | 128 | ``` |
69 | 129 |
|
70 | | -Keep installation build files, environments, downloads and generated outputs outside a pristine artifact snapshot used for validation. The checker requires Gitleaks 8.30.1 and PyYAML; it ignores no files or directories. See [gate semantics](docs/RELEASE_STATUS.md). |
| 130 | +## Citation |
| 131 | + |
| 132 | +**Lev: Thinking Fast and Deep with Looped Decision Models** |
| 133 | +Dingjie Song, Zhiling Yan, Yuxuan Zhang, and Lichao Sun. |
71 | 134 |
|
72 | | -This candidate includes adapted analysis code and existing study aggregate results. It does not bundle the complete per-question result collection or provide online adaptive state continuation. Never use locked test outcomes to choose a new rule or checkpoint. |
| 135 | +The paper is forthcoming. Its arXiv identifier and final citation will be added when assigned; the [project paper section](https://openlair.github.io/lev/#paper) is the current reference point. |
73 | 136 |
|
74 | | -## Citation and attribution |
| 137 | +## License and acknowledgments |
75 | 138 |
|
76 | | -Paper title: **Lev: Thinking Fast and Deep with Looped Decision Models**. Authors: Dingjie Song, Zhiling Yan, Yuxuan Zhang, and Lichao Sun. The arXiv identifier and final citation will be added when assigned. See the [paper section](https://openlair.github.io/lev/#paper) for paper and citation updates. |
| 139 | +The source code is licensed under [Apache-2.0](LICENSE) and derived from [Jared Palmer's Kev](https://github.com/jaredpalmer/kev). See [NOTICE](NOTICE) for attribution. |
77 | 140 |
|
78 | | -Derived from [Jared Palmer's Kev](https://github.com/jaredpalmer/kev). See [NOTICE](NOTICE) and [LICENSE](LICENSE). |
| 141 | +Backbone weights and dataset text have separate terms. Review [licensing](docs/LICENSING.md) and the relevant model or dataset card before redistributing artifacts. |
0 commit comments