You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7df1342
Browse filesBrowse the repository at this point in the historyBrowse files
Publish the release: link the paper, models and datasets
The code, three Lev checkpoints and three datasets are now public, so the
README, resource list and cards drop the private-staging wording and link
what a reader can open: the paper on alphaXiv and as a PDF, the Hugging Face
collection, each model and dataset, and the CPU test workflow. The README
gains a dataset table and a BibTeX entry.
The cards and release status now say what has not been re-run on the
released packages (full-size GPU parity, complete data reconstruction,
all-figure reproduction) instead of listing them as pending gates.
MANIFEST.sha256 is regenerated for the edited files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
<ahref="https://www.alphaxiv.org/pdf/2610.lev-looped-decision-models"><imgsrc="https://img.shields.io/badge/Paper-alphaXiv-b31b1b?style=flat-square"alt="Paper on alphaXiv"></a>
<ahref="https://huggingface.co/collections/songdj/lev"><imgsrc="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models%20%26%20Data-ffcc4d?style=flat-square"alt="Models and data on Hugging Face"></a>
> **Release status:**GitHub source and Hugging Face model repositories are private research candidates. The paper is forthcoming; no arXiv identifier has been assigned. Three companion Dataset repositories are **privately staged**: licensed source subsets of LevBench and Lev-train-v2, plus text-free Lev-results. [Resource availability →](docs/RESOURCES.md)
32
+
> **Release:**the code, three model checkpoints and three datasets are public. Read the paper on [alphaXiv](https://www.alphaxiv.org/pdf/2610.lev-looped-decision-models) or download the [PDF](https://openlair.github.io/lev/assets/lev-paper.pdf); the arXiv listing will be linked once it is announced.
31
33
32
34
## One checkpoint, more room to think
33
35
@@ -41,15 +43,21 @@ The accuracy result uses JevBench's **231 public tasks** and fp32 readouts. Fixe
41
43
42
44
## Model family
43
45
44
-
All linked model repositories currently require authorized access. The source package contains model cards and calibration metadata; weights are staged separately.
46
+
Each Hugging Face repository holds the seed-0 LoRA adapter, decision head, tokenizer assets and per-depth temperatures. This source repository holds the model cards and calibration metadata.
45
47
46
48
| Model | Backbone | Native depths | JevBench accuracy, first → deepest | Artifacts |
These are existing study results, not new evaluations of the release candidate. The cards document backbone revisions, seed coverage, calibration and limitations.
54
+
These are existing study results, not new evaluations of the released packages. The cards document backbone revisions, seed coverage, calibration and limitations.
55
+
56
+
| Dataset | What it holds |
57
+
|---|---|
58
+
|[🤗 LevBench](https://huggingface.co/datasets/songdj/LevBench)| Development and locked-test decisions across five tiers (rights-filtered subset) |
59
+
|[🤗 Lev-train-v2](https://huggingface.co/datasets/songdj/Lev-train-v2)| S31 / S60 / S120 training mixtures, replay and order recipes (rights-filtered subset) |
60
+
|[🤗 Lev-results](https://huggingface.co/datasets/songdj/Lev-results)| Per-depth outputs, calibration and latency evidence, without question text |
53
61
54
62
## Quick start
55
63
@@ -76,15 +84,15 @@ The validated fresh CPU installation passed **187 tests**, with **14 opt-in netw
76
84
77
85
### 2. Score a request at a chosen depth
78
86
79
-
For model inference, obtain the appropriate private checkpoint and prepare a **separate CUDA-capable PyTorch environment** using the [installation notes](docs/INSTALL.md). The CPU-only test environment above is for validation. A checkpoint needs its adapter configuration, adapter weights and `head.pt`; the loader fetches the pinned backbone, so cache it first for offline use.
87
+
For model inference, download a checkpoint (`hf download songdj/Lev-2.6B --local-dir ../lev-weights/Lev-2.6B`) and prepare a **separate CUDA-capable PyTorch environment** using the [installation notes](docs/INSTALL.md). The CPU-only test environment above is for validation. A checkpoint needs its adapter configuration, adapter weights and `head.pt`; the loader fetches the pinned backbone, so cache it first for offline use.
80
88
81
89
```bash
82
90
python examples/fixed_depth.py \
83
91
--checkpoint ../lev-weights/Lev-2.6B \
84
92
--depth 2 --device cuda
85
93
```
86
94
87
-
The example scores an original two-option toy request. A chosen fixed depth executes only that many recurrent steps. `--all-depths` returns every depth's output from a full-depth pass; **it does not save computation**. Full-size GPU parity remains a release requirement.
95
+
The example scores an original two-option toy request. A chosen fixed depth executes only that many recurrent steps. `--all-depths` returns every depth's output from a full-depth pass; **it does not save computation**. Full-size GPU parity has not been re-run on the released packages.
88
96
89
97
For calibrated probabilities, verify checkpoint identity and add `--temperatures calibration/Lev-2.6B-s0.json`. Per-depth temperatures are separate from the scalar temperature in `head.pt`. Without this option, the example explicitly labels its scores as uncalibrated.
The redraws preserve the study's table-value assertions. They do not recompute confidence intervals from the complete per-question collection. Companion [Dataset repositories](docs/RESOURCES.md)now stage source-specific licensed subsets and existing text-free results privately. Complete source rights and end-to-end reconstruction remain under review; these subsets are not the full frozen corpus.
116
+
The redraws preserve the study's table-value assertions. They do not recompute confidence intervals from the complete per-question collection. The companion [datasets](docs/RESOURCES.md)publish source-specific licensed subsets and existing text-free results. Complete source rights and end-to-end reconstruction remain open; these subsets are not the full frozen corpus.
109
117
110
118
## Scope and limitations
111
119
112
120
-**Adaptive routing is offline.** The saved all-depth logits support analysis of early exits. Online hidden-state continuation and adaptive serving are not implemented. The paper's stopper is cross-fitted per workload, rather than supplied as a universal checkpoint threshold.
113
121
-**Gains are workload-dependent.** Lev-2.6B failed the locked LevBench G3 criterion in all three study seeds. The Lev-1.4B and Lev-3B headline seeds passed there; that does not establish a general advantage over fixed-depth frontiers. Greater depth can also overturn an earlier correct decision.
114
122
-**Accuracy and timing have different measurement paths.** Accuracy uses fp32 readouts; reported H200 timing uses bf16, batch one and implementation-specific kernels. Latency comparisons are not portable speed guarantees. G3 and the paper's token-weighted cost intervals are different analyses.
115
-
-**Release readiness is tracked separately.** CPU installation and tests have passed. Full-size GPU parity, final public weight identities, data rights, complete reconstruction, all-figure reproduction, public endpoints and author review remain gated. Never use the fixed locked test to choose a new rule or checkpoint.
123
+
-**What has and has not been re-run.** CPU installation and tests pass on every push. Full-size GPU parity on the released packages, complete data reconstruction and all-figure reproduction have not been re-run; see [release status](docs/RELEASE_STATUS.md). Never use the fixed locked test to choose a new rule or checkpoint.
**Lev: Thinking Fast and Deep with Looped Decision Models**
133
141
Dingjie Song, Zhiling Yan, Yuxuan Zhang, and Lichao Sun.
134
142
135
-
The paper is forthcoming. Its arXiv identifier and final citation will be added when assigned; the [project paper section](https://openlair.github.io/lev/#paper) is the current reference point.
143
+
Read the paper on [alphaXiv](https://www.alphaxiv.org/pdf/2610.lev-looped-decision-models) or download the [PDF](https://openlair.github.io/lev/assets/lev-paper.pdf). The arXiv identifier will be added when it is announced.
144
+
145
+
```bibtex
146
+
@misc{song2026lev,
147
+
title = {Lev: Thinking Fast and Deep with Looped Decision Models},
148
+
author = {Song, Dingjie and Yan, Zhiling and Zhang, Yuxuan and Sun, Lichao},
Private repository: [songdj/Lev-results](https://huggingface.co/datasets/songdj/Lev-results). This candidate packages aggregate JSON inputs under `results/` and `figures/`, not the full per-question collection. Aggregate figure redraws and re-estimation from row-level observations are different reproducibility levels.
10
+
Hub repository: [songdj/Lev-results](https://huggingface.co/datasets/songdj/Lev-results). This candidate packages aggregate JSON inputs under `results/` and `figures/`, not the full per-question collection. Aggregate figure redraws and re-estimation from row-level observations are different reproducibility levels.
11
11
12
12
Depth-row schema (version 1): `model`, `panel`, `split`, `depths`, `source_sha256`, and `rows`. Each row retains `id`, `group`, question identifier `question`, `source`, `task`, `type`, `variant`, numeric `label` and/or per-depth `correct`, `state_tokens`, `tokens`, logits as `[depth][option]`, and any `tier`, `difficulty` (`kind`, `k`), `unanswerable`, `report_only`. Question identity is `(id, question)`; bootstrap identity is `group`. No state, question instructions, criteria or option description is included. Model identity, depth/option ordering, inference dtype, temperature status and calibration provenance must accompany the final collection.
13
13
14
-
`scripts/export_results.py` projects the standard depth-row schema and refuses locked test export. It intentionally does not export JevBench's different per-depth probability files. The private Dataset uses a separate hash-allowlisted publication adapter for these files, preserves stored values and option order, and replaces textual identities with stable hashes. Its own manifest and audit cover exported native and JevBench files; the normal code exporter retains its existing restrictions.
14
+
`scripts/export_results.py` projects the standard depth-row schema and refuses locked test export. It intentionally does not export JevBench's different per-depth probability files. The Dataset uses a separate hash-allowlisted publication adapter for these files, preserves stored values and option order, and replaces textual identities with stable hashes. Its own manifest and audit cover exported native and JevBench files; the normal code exporter retains its existing restrictions.
15
15
16
16
The original cost analyses need token counts, group membership, per-depth latency fits and system/model mapping, not just ids and logits. Fitted latency tables are supplied in `results/latency/latency-fits.json`. Every row export needs an input hash, output hash and permissions review; a text-free projection does not by itself settle source terms.
17
17
18
18
19
-
The private Dataset card is the current source of package coverage, missing materials, source licenses and immutable file identities. The code candidate contains metadata only; source-specific hosted subsets have new hashes and do not replace the full original frozen inputs. See [data coverage](../docs/DATA.md#private-staging-coverage).
19
+
The Hugging Face Dataset card is the current source of package coverage, missing materials, source licenses and immutable file identities. The code candidate contains metadata only; source-specific hosted subsets have new hashes and do not replace the full original frozen inputs. See [data coverage](../docs/DATA.md#hosted-coverage).
0 commit comments