Skip to content

Commit f4eb667

Browse files
committed
Polish project README with shared visuals and clearer research resources
1 parent 429aafe commit f4eb667

4 files changed

Lines changed: 112 additions & 45 deletions

File tree

‎MANIFEST.sha256‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@ c1fd9193ecaeea63c05f363a700a2e878dc595533439fa203126a5a7593063dd .gitleaks.toml
44
6b08bb37982c233aa12bcbdf19106da12f3f4fcf800773ea42e28ebeddd34fb8 LICENSE
55
10750add958d3530e8d639e92318080fafe45027fb6516adcb339e0f47937d25 NOTICE
66
52c840d49ecb88da6f48f0cc28480811f562c7895f3ce6b0c2693ebbd190e616 PROVENANCE.json
7-
4191d13b1e769319daba30cd80564a773e41002e850e0bd3e46aef12374300c9 README.md
7+
607830d9d1f88e038518e136f3cdae5092514e23ec397cbfb4a4884a1ca785c4 README.md
88
ae523491b69c806bbdae659119d5ce400f52a79137ed18bfd6f1b1efdc0dc8fb RELEASE_FILES.json
9-
b07d12cf0010a1fb972d695425e6eb6d04ea0a2a9bec0536bb6856bf92a2ba94 RELEASE_READINESS.json
9+
92b13a1618c05582007a772a6fc2b27a06f31ddf7802af924291409bfe1f0143 RELEASE_READINESS.json
1010
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 analysis/__init__.py
1111
e28d2302dc08a60c91f0856aec3b11d3b82f17219369c1fbe553cd8a87bfb76b analysis/adaptive_depth.py
1212
20e77d413cefd8b9a4e0233f75c15dfb8cb10d539445255feab47e9a6cde0d41 analysis/arm_compare.py
@@ -90,7 +90,7 @@ d21bf1efab87191ea5ef985c3ca2e98846e48bbac6485be04d1954a7e08d255d model-cards/Le
9090
d46be0ef17c561a79a63903024fb056108e8af4cf63d5945f67e742ed8366466 pyproject.toml
9191
2b5375dc8f07de832d5bb43f0e563090d5cf6870d4df995361aee5e9395c89a4 release-evidence/clean-install-packages.json
9292
3071d107c6417021a3809cc438840cdea00212bef4fab1a1ced42745b7e30121 release-evidence/clean-install.json
93-
2b0ee368b906a77dde06c839e27397afbbd740dabbbd8205b62e1ae17c2e4dad release-evidence/cpu-observations.md
93+
2faee730af3b1f0972487b86ec59cb5f340416c7a4e682447f3cfc25f6369f47 release-evidence/cpu-observations.md
9494
3bd8d825f88c7170845db8629a9f61a395bcece4c27ec8513d6de139f52cd587 release-evidence/cpu-tested-files.json
9595
304c6ec9aa6d5afef27a07f55e8e30622eaed8d2ccd49c1e6e19b369d3eede2e release-evidence/cpu-tests.json
9696
facc9b1c8b8290f4a60e78c80375694d577053f5e3659672a9c371485f189c84 requirements-tested.txt

‎README.md‎

Lines changed: 103 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -1,78 +1,141 @@
1-
# Lev: one checkpoint, multiple depth operating points
1+
<p align="center">
2+
<a href="https://openlair.github.io/lev/">
3+
<img src="https://openlair.github.io/lev/assets/social.png" alt="Lev: Thinking fast and deep. One checkpoint. Multiple depths." width="100%">
4+
</a>
5+
</p>
26

3-
[Project website](https://openlair.github.io/lev/) · [Paper](https://openlair.github.io/lev/#paper) · [GitHub](https://github.com/OpenLAIR/lev) · [Hugging Face collection — private](https://huggingface.co/collections/songdj/lev-thinking-fast-and-deep-with-looped-decision-models-6acefb135880c9d373cf42ba)
7+
<h1 align="center">Lev · System 1.5 Decision Models</h1>
48

5-
Lev turns pretrained recurrent language models into typed decision models. A shared pointer head returns one probability per option after each recurrent step, without generating an answer string. The same checkpoint provides several accuracy–compute operating points.
9+
<p align="center">
10+
<strong>Thinking Fast and Deep with Looped Decision Models</strong><br>
11+
One checkpoint. Multiple depth operating points. Probabilities over your options.
12+
</p>
613

7-
**Research release candidate.** Source code is maintained under [OpenLAIR/lev](https://github.com/OpenLAIR/lev). The linked Hugging Face collection and seed-0 model repositories are private; dataset releases and the arXiv identifier remain pending. Weights and question text are not bundled. See [resource availability](docs/RESOURCES.md) and [release status](docs/RELEASE_STATUS.md).
14+
<p align="center">
15+
<a href="docs/RELEASE_STATUS.md"><img src="https://img.shields.io/badge/research-release%20candidate-2448ff?style=flat-square" alt="Research release candidate"></a>
16+
<a href="https://openlair.github.io/lev/#paper"><img src="https://img.shields.io/badge/paper-forthcoming-5b6467?style=flat-square" alt="Paper forthcoming"></a>
17+
<a href="docs/VALIDATION.md"><img src="https://img.shields.io/badge/CPU%20tests-187%20passed-2448ff?style=flat-square" alt="CPU validation: 187 passed"></a>
18+
<a href="LICENSE"><img src="https://img.shields.io/badge/code-Apache%202.0-5b6467?style=flat-square" alt="Code license: Apache 2.0"></a>
19+
</p>
820

9-
| Model | Backbone | Supported native depths | Hugging Face repository (private) |
10-
|---|---|---|---|
11-
| Lev-1.4B | Ouro-1.4B | 1–4 | [songdj/Lev-1.4B](https://huggingface.co/songdj/Lev-1.4B) |
12-
| Lev-2.6B | Ouro-2.6B | 1–4 | [songdj/Lev-2.6B](https://huggingface.co/songdj/Lev-2.6B) |
13-
| Lev-3B | Nanbeige4.2-3B-Base | 1–2 | [songdj/Lev-3B](https://huggingface.co/songdj/Lev-3B) |
21+
<p align="center">
22+
<a href="https://openlair.github.io/lev/"><strong>Website</strong></a> ·
23+
<a href="https://openlair.github.io/lev/#paper"><strong>Paper</strong></a> ·
24+
<a href="https://huggingface.co/collections/songdj/lev-thinking-fast-and-deep-with-looped-decision-models-6acefb135880c9d373cf42ba"><strong>Models — private</strong></a> ·
25+
<a href="#quick-start"><strong>Quick start</strong></a> ·
26+
<a href="#reproduce-the-study"><strong>Reproduce</strong></a> ·
27+
<a href="#citation"><strong>Citation</strong></a>
28+
</p>
1429

15-
On JevBench's 231 public tasks, Lev-2.6B improves from 63.2% at depth 1 to 76.2% at depth 4. An agreement stopper supports **offline** analysis of which questions could exit early. Its benefit over a matched fixed-depth frontier varies by workload: Lev-2.6B did not pass the locked LevBench G3 criterion. Latency comparisons depend on the measured implementation and hardware.
30+
> **Release status:** GitHub source and Hugging Face model repositories are private research candidates. The paper is forthcoming; no arXiv identifier has been assigned. The datasets have **not been uploaded**. [Resource availability →](docs/RESOURCES.md)
1631
17-
![Lev overview from the paper](figures/figure-1-preview.svg)
32+
## One checkpoint, more room to think
1833

19-
*Figure 1 from the study; adaptive points summarize offline routing analysis.*
34+
Lev adapts pretrained recurrent language models into typed decision models. A shared pointer head returns a probability for each caller-provided option after each recurrent step. You choose the depth—and therefore the amount of computation—without switching checkpoints or generating an answer string.
2035

21-
## Install
36+
| Flexible depth | Measured accuracy | Structured decisions |
37+
|---|---|---|
38+
| **Up to 4 native depths** from the same checkpoint | **63.2% → 76.2%** for Lev-2.6B, depth 1 → 4 | **One probability per option** with a shared pointer head |
2239

23-
Use Linux and Python 3.12 or 3.13. Create a new environment; the distribution is named `lev`, while Python imports remain `kev`.
40+
The accuracy result uses JevBench's **231 public tasks** and fp32 readouts. Fixed-depth scaling is the primary result. The agreement stopper provides an additional **offline** routing analysis; its gains depend on the workload. **Lev-2.6B did not pass G3 on locked LevBench in any of its three study seeds.** See [scope and limitations](#scope-and-limitations).
41+
42+
## Model family
43+
44+
All linked model repositories currently require authorized access. The source package contains model cards and calibration metadata; weights are staged separately.
45+
46+
| Model | Backbone | Native depths | JevBench accuracy, first → deepest | Artifacts |
47+
|---|---|---|---|---|
48+
| **Lev-1.4B** | Ouro-1.4B | 1–4 | 58.0% → 68.0% | [HF · private](https://huggingface.co/songdj/Lev-1.4B) · [Model card](model-cards/Lev-1.4B.md) |
49+
| **Lev-2.6B** | Ouro-2.6B | 1–4 | 63.2% → 76.2% | [HF · private](https://huggingface.co/songdj/Lev-2.6B) · [Model card](model-cards/Lev-2.6B.md) |
50+
| **Lev-3B** | Nanbeige4.2-3B-Base | 1–2 | 69.3% → 72.7% | [HF · private](https://huggingface.co/songdj/Lev-3B) · [Model card](model-cards/Lev-3B.md) |
51+
52+
These are existing study results, not new evaluations of the release candidate. The cards document backbone revisions, seed coverage, calibration and limitations.
53+
54+
## Quick start
55+
56+
### 1. Install and run CPU tests
57+
58+
Use Linux and Python 3.12 or 3.13. From the source checkout, create a separate environment and working copy. The distribution is named `lev`; Python imports remain `kev`.
2459

2560
```bash
2661
python3.13 -m venv ../lev-env
2762
source ../lev-env/bin/activate
28-
python -m pip install --upgrade pip
63+
64+
# Use the CPU wheel for validation; no CUDA dependency download is needed.
65+
python -m pip install --no-cache-dir 'torch==2.8.0+cpu' \
66+
--index-url https://download.pytorch.org/whl/cpu
2967
cp -R . ../lev-work
30-
python -m pip install '../lev-work[dev]'
31-
(cd ../lev-work && PYTHONDONTWRITEBYTECODE=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python -m pytest -q -p no:cacheprovider)
68+
python -m pip install --no-cache-dir '../lev-work[dev]'
69+
python -m pip check
70+
71+
(cd ../lev-work && PYTHONDONTWRITEBYTECODE=1 OMP_NUM_THREADS=1 \
72+
MKL_NUM_THREADS=1 python -m pytest -q -p no:cacheprovider)
3273
```
3374

34-
The CPU tests use tiny randomly initialized models and require no downloads or pretrained weights. [INSTALL.md](docs/INSTALL.md) records tested dependency versions, optional CUDA kernels for hybrid dense controls, and the pinned JevBench installation. The original environment's lockfile is not reused for this differently scoped package.
75+
The validated fresh CPU installation passed **187 tests**, with **14 opt-in network tests skipped**. Tests use tiny randomly initialized models and require no pretrained weights or dataset downloads. Exact dependency versions and evidence are in [validation](docs/VALIDATION.md).
3576

36-
## Fixed-depth quick start
77+
### 2. Score a request at a chosen depth
3778

38-
After obtaining a checkpoint directory containing `adapter_config.json`, adapter weights, and `head.pt`:
79+
For model inference, obtain the appropriate private checkpoint and prepare a **separate CUDA-capable PyTorch environment** using the [installation notes](docs/INSTALL.md). The CPU-only test environment above is for validation. A checkpoint needs its adapter configuration, adapter weights and `head.pt`; the loader fetches the pinned backbone, so cache it first for offline use.
3980

4081
```bash
41-
python examples/fixed_depth.py --checkpoint ../lev-weights/Lev-2.6B --depth 2 --device cuda
82+
python examples/fixed_depth.py \
83+
--checkpoint ../lev-weights/Lev-2.6B \
84+
--depth 2 --device cuda
4285
```
4386

44-
The example uses an original two-option toy question. `Checkpoint.load` loads the pinned backbone from the Hub; cache it first for offline use. The CLI supports `--all-depths` to read all outputs in one full-depth pass. **All-depth output does not save computation.** A chosen fixed depth uses only that number of recurrent steps.
87+
The example scores an original two-option toy request. A chosen fixed depth executes only that many recurrent steps. `--all-depths` returns every depth's output from a full-depth pass; **it does not save computation**. Full-size GPU parity remains a release requirement.
4588

46-
```python
47-
from kev.checkpoint import Checkpoint, LoadOptions
89+
For calibrated probabilities, verify checkpoint identity and add `--temperatures calibration/Lev-2.6B-s0.json`. Per-depth temperatures are separate from the scalar temperature in `head.pt`. Without this option, the example explicitly labels its scores as uncalibrated.
4890

49-
tokenizer, model = Checkpoint('../lev-weights/Lev-2.6B').load('cuda', LoadOptions(temperature=1.0))
50-
model.set_depth(2)
91+
## Reproduce the study
92+
93+
| What you want to do | Start here |
94+
|---|---|
95+
| Understand the training recipe, readouts and offline stopper | [Reproduction guide](docs/REPRODUCING.md) |
96+
| Redraw available paper figures from aggregate results | [Figure inventory and commands](figures/README.md) |
97+
| Inspect data sources, hashes and reconstruction prerequisites | [Data guide](docs/DATA.md) · [Dataset cards](data-cards/) |
98+
| Review checkpoint details and per-depth calibration | [Model cards](model-cards/) · [Calibration files](calibration/) |
99+
| Inspect installation checks and release conditions | [Validation](docs/VALIDATION.md) · [Release gates](docs/RELEASE_STATUS.md) |
100+
101+
**Seven of thirteen paper figures** can currently be redrawn from the packaged aggregates:
102+
103+
```bash
104+
PYTHONDONTWRITEBYTECODE=1 python figures/reproduce.py \
105+
--output-dir ../lev-figures
51106
```
52107

53-
Per-depth calibration temperatures are a separate JSON artifact, not implied by `head.pt`'s scalar temperature. The existing seed-0 calibration JSON files are in `calibration/`; for Lev-2.6B, supply `--temperatures calibration/Lev-2.6B-s0.json` after verifying checkpoint identity. Without it, the example prints uncalibrated softmax scores.
108+
The redraws preserve the study's table-value assertions. They do not recompute confidence intervals from the complete per-question collection. Dataset text and that full collection are not bundled or uploaded; source terms and reconstruction prerequisites remain under review.
109+
110+
## Scope and limitations
54111

55-
## Reproduction
112+
- **Adaptive routing is offline.** The saved all-depth logits support analysis of early exits. Online hidden-state continuation and adaptive serving are not implemented. The paper's stopper is cross-fitted per workload, rather than supplied as a universal checkpoint threshold.
113+
- **Gains are workload-dependent.** Lev-2.6B failed the locked LevBench G3 criterion in all three study seeds. The Lev-1.4B and Lev-3B headline seeds passed there; that does not establish a general advantage over fixed-depth frontiers. Greater depth can also overturn an earlier correct decision.
114+
- **Accuracy and timing have different measurement paths.** Accuracy uses fp32 readouts; reported H200 timing uses bf16, batch one and implementation-specific kernels. Latency comparisons are not portable speed guarantees. G3 and the paper's token-weighted cost intervals are different analyses.
115+
- **Release readiness is tracked separately.** CPU installation and tests have passed. Full-size GPU parity, final public weight identities, data rights, complete reconstruction, all-figure reproduction, public endpoints and author review remain gated. Never use the fixed locked test to choose a new rule or checkpoint.
56116

57-
- [Data and exact-input requirements](docs/DATA.md): source revisions, manifest hashes, training subsets, and distribution restrictions.
58-
- [Training, readouts and offline stopping](docs/REPRODUCING.md): portable commands and the distinction between cross-fitting and deployment calibration.
59-
- [Paper figures](figures/README.md): runnable aggregate-input redraws and an honest inventory of remaining inputs.
60-
- [Model cards](model-cards/) and [dataset cards](data-cards/): release drafts, provenance and limitations.
61-
- [Source and weight licenses](docs/LICENSING.md): source code is Apache-2.0; dataset and backbone terms are separate.
117+
[Full reproduction details](docs/REPRODUCING.md) · [Current release conditions](docs/RELEASE_STATUS.md)
118+
119+
### Verify a source artifact
120+
121+
The checker requires **Gitleaks 8.30.1** and PyYAML. Run it against a pristine source snapshot; see the [snapshot instructions](docs/RELEASE_STATUS.md) for Git checkouts. Environments, caches, downloads and generated outputs must remain outside that snapshot—nothing is silently ignored.
62122

63123
```bash
64-
PYTHONDONTWRITEBYTECODE=1 python figures/reproduce.py --output-dir ../lev-figures
65124
PYTHONDONTWRITEBYTECODE=1 python scripts/verify_release.py --mode hygiene
66-
# Default release mode intentionally fails while readiness conditions remain blocked.
125+
126+
# Formal release verification remains blocked while required conditions are open.
67127
PYTHONDONTWRITEBYTECODE=1 python scripts/verify_release.py
68128
```
69129

70-
Keep installation build files, environments, downloads and generated outputs outside a pristine artifact snapshot used for validation. The checker requires Gitleaks 8.30.1 and PyYAML; it ignores no files or directories. See [gate semantics](docs/RELEASE_STATUS.md).
130+
## Citation
131+
132+
**Lev: Thinking Fast and Deep with Looped Decision Models**
133+
Dingjie Song, Zhiling Yan, Yuxuan Zhang, and Lichao Sun.
71134

72-
This candidate includes adapted analysis code and existing study aggregate results. It does not bundle the complete per-question result collection or provide online adaptive state continuation. Never use locked test outcomes to choose a new rule or checkpoint.
135+
The paper is forthcoming. Its arXiv identifier and final citation will be added when assigned; the [project paper section](https://openlair.github.io/lev/#paper) is the current reference point.
73136

74-
## Citation and attribution
137+
## License and acknowledgments
75138

76-
Paper title: **Lev: Thinking Fast and Deep with Looped Decision Models**. Authors: Dingjie Song, Zhiling Yan, Yuxuan Zhang, and Lichao Sun. The arXiv identifier and final citation will be added when assigned. See the [paper section](https://openlair.github.io/lev/#paper) for paper and citation updates.
139+
The source code is licensed under [Apache-2.0](LICENSE) and derived from [Jared Palmer's Kev](https://github.com/jaredpalmer/kev). See [NOTICE](NOTICE) for attribution.
77140

78-
Derived from [Jared Palmer's Kev](https://github.com/jaredpalmer/kev). See [NOTICE](NOTICE) and [LICENSE](LICENSE).
141+
Backbone weights and dataset text have separate terms. Review [licensing](docs/LICENSING.md) and the relevant model or dataset card before redistributing artifacts.

‎RELEASE_READINESS.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
"clean_install": {
77
"status": "passed",
88
"reason": "Fresh Python 3.13.14 CPU environment: complete dependency install, pip check and CPU suite passed.",
9-
"subject_sha256": "563435b9286805f5b7bb53f840d00afbf845a1b74e066b2c1702de7e7ed45ffa",
9+
"subject_sha256": "f917e00e974dd2e8b20aa442a17ca230c183fbc12101ae216f06a5fbc6b7f70b",
1010
"evidence": {
1111
"path": "release-evidence/clean-install.json",
1212
"sha256": "3071d107c6417021a3809cc438840cdea00212bef4fab1a1ced42745b7e30121"
@@ -15,7 +15,7 @@
1515
"cpu_tests": {
1616
"status": "passed",
1717
"reason": "Fresh CPU environment: 187 passed; 14 opt-in network tests skipped.",
18-
"subject_sha256": "563435b9286805f5b7bb53f840d00afbf845a1b74e066b2c1702de7e7ed45ffa",
18+
"subject_sha256": "f917e00e974dd2e8b20aa442a17ca230c183fbc12101ae216f06a5fbc6b7f70b",
1919
"evidence": {
2020
"path": "release-evidence/cpu-tests.json",
2121
"sha256": "304c6ec9aa6d5afef27a07f55e8e30622eaed8d2ccd49c1e6e19b369d3eede2e"

‎release-evidence/cpu-observations.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,7 @@ Command from the isolated working-copy root: `PYTHONDONTWRITEBYTECODE=1 OMP_NUM_
99
Seven figure redraws also passed in the new environment with outputs outside the source snapshot. Original table-value assertions passed. They used packaged aggregate results and do not establish full row-based figure reproduction.
1010

1111
The earlier hosted CPU installation exhausted disk space while installing the default CUDA-enabled PyTorch dependencies, before tests began. The CPU workflow now selects the official CPU-only wheel explicitly, checks dependencies and verifies that a CUDA wheel was not substituted. Research code, tests and release-gate requirements were unchanged. This fresh installation and test run supersedes the earlier CPU result obtained with existing dependencies. Hosted CI will rerun after the candidate is pushed; no hosted success is implied by this local validation. GPU parity and optional CUDA kernels remain separately unvalidated.
12+
13+
## README presentation update
14+
15+
The README was subsequently redesigned with the deployed project banner, resource navigation, documented model results and an explicit CPU-only validation installation example. GitHub's Markdown renderer accepted the layout; local links, navigation anchors and shell-example syntax were checked. All 118 files in `cpu-tested-files.json` still match their recorded hashes, including the research code, tests, dependency declarations, CI workflow and release-gate implementation. Fresh installation and CPU evidence are therefore retained and rebound to the updated documentation artifact. No CPU suite or GPU evaluation was repeated for this presentation-only change.

0 commit comments

Comments
 (0)