Skip to content

Commit ab3198f

Browse files
committed
[MLX] Avoid standalone target forwards between DFlash rounds
MLX DFlash decoding currently runs a standalone target forward before each draft and verification round. For eligible singleton token batches, draft directly from the uncommitted input token and let verification commit the anchor and accepted proposals. Preserve the existing path for mixed inputs, empty pending context, and context tails. This applies to both original DFlash and DFlash2. Verify greedy proposals against the target sample already drawn for each absolute position. This keeps RNG selection stable across prompt chunking and batch composition when the shortcut is enabled, and removes retained vocabulary probability arrays. The original baseline passes the schedule test; the inconsistency occurs in the shortcut without this sampling change. Native sampling, batching, and serialized DFlash suites pass, including temperature 0.8 distributions, top-k/top-p, cache lifecycle, shared-payload input offsets, and fallback paths. An input-sensitive original-DFlash fixture also exercises target-only/speculative parity, reset, clone, and rewind. A wrong-anchor mutation passes the old continuation test and fails the new slice checks for both draft formats. The same 96 schedule cases pass on the unchanged baseline, sampling-only control, and combined implementation; the shortcut before the sampling change fails for both draft formats. Lintrunner and git diff --check pass. Measured actual serving through the unmodified loopback `/v1/chat/completions` endpoint on M5 Max (64 GB), with full Q4_K_M target/draft models and FP16 activations. Baseline is the parent stack (83884a8004); production workers differ only in the MLX executor. One active request, four short prompts, 64 generated tokens, three measured repetitions per prompt after excluded warmups (288 measured requests), seed 42, top-k 0/top-p 1, `ignore_eos=true`, greedy drafts and target temperature 0 or 0.8. Fresh sessions, no prefix reuse; variant order AB/BA/AB. Model startup and cooling pauses between groups are excluded. These rates describe warmed single requests under controlled starting temperatures; sustained server capacity was not measured. Server effective decode uses `(output_tokens - 1) / native_decode_seconds`, from the native runner's first output batch to completion. Speculative bursts give multiple tokens the same timestamp, including any burst delivered with the first token. HTTP end-to-end throughput includes preparation, tokenization, scheduling, inference and streaming through `[DONE]`. Both are pooled token counts divided by pooled latency; SSE fragments are not counted as tokens. | Drafter | Temperature | Proposals | Server effective decode tok/s, before → after | HTTP end-to-end tok/s, before → after | |---|---:|---:|---:|---:| | DFlash | 0 | 3 | 48.51 → 59.85 (+23.4%) | 42.65 → 50.88 (+19.3%) | | DFlash | 0 | 7 | 60.49 → 80.53 (+33.1%) | 51.23 → 64.69 (+26.3%) | | DFlash | 0 | 15 | 84.50 → 108.88 (+28.8%) | 66.90 → 81.14 (+21.3%) | | DFlash | 0.8 | 3 | 47.33 → 59.24 (+25.2%) | 41.70 → 50.40 (+20.8%) | | DFlash | 0.8 | 7 | 60.05 → 75.39 (+25.5%) | 50.78 → 61.31 (+20.7%) | | DFlash | 0.8 | 15 | 64.43 → 91.26 (+41.6%) | 53.72 → 70.81 (+31.8%) | | DFlash2 | 0 | 3 | 48.00 → 62.00 (+29.2%) | 41.82 → 51.99 (+24.3%) | | DFlash2 | 0 | 7 | 67.08 → 94.85 (+41.4%) | 55.44 → 72.59 (+30.9%) | | DFlash2 | 0 | 15 | 87.19 → 118.54 (+35.9%) | 67.97 → 85.07 (+25.2%) | | DFlash2 | 0.8 | 3 | 45.88 → 59.07 (+28.8%) | 40.20 → 49.84 (+24.0%) | | DFlash2 | 0.8 | 7 | 62.68 → 85.47 (+36.4%) | 52.27 → 66.85 (+27.9%) | | DFlash2 | 0.8 | 15 | 86.35 → 94.84 (+9.8%) | 67.21 → 72.06 (+7.2%) | GPU frequency/activity/temperature and macOS thermal state were sampled every 200 ms. All measured request-window samples reported nominal thermal pressure, AC power and low power mode disabled, with 100.0% telemetry coverage. Maximum measured GPU temperature was 81.6 C. Per-case GPU active-time-weighted clocks ranged 1.490–1.510 GHz for baseline and 1.601–1.610 GHz for candidate (+6.3% to +8.0% paired differences). Within a fixed case and variant, repetition-average clocks varied by at most 0.8%. Clocks were monitored, not locked, so these are whole-system serving gains. An exploratory warmer run triggered fair thermal pressure and clocks around 1.0 GHz; it is excluded in full. Each group started after two seconds of nominal thermal state with GPU <=60 C / CPU <=70 C. These are experimental starting conditions, not hardware throttling thresholds. All greedy visible HTTP responses match baseline, and seeded visible-text replay is stable in each variant across all conditions. Sampled old/new responses can differ at the same seed because target RNG coupling changes, so sampled gains include different token streams and acceptance patterns. Per-prompt HTTP means below baseline in this corpus: DFlash2, T=0.8, N=15, explanation: -6.7%; DFlash2, T=0.8, N=15, planning: -4.0%. A separate earlier, unmonitored native DFlash2/N=15/T=0.8 matched-sampler control produces identical tokens and improves prefill-plus-generation throughput 24.3% (warm decode 31.6%); that control is not an HTTP measurement. These means cover a small corpus with three repetitions, and longer FP16 runs have existing near-tie differences across verification shapes. Concurrency was not benchmarked. CUDA code and interfaces are unchanged. Follow-up to #24811, tracked in #23011. Stacked on #25336. Authored with AI assistance from OpenAI Codex. ghstack-source-id: ea265ca ghstack-comment-id: 7033163531 Pull-Request: #25337
1 parent 120045c commit ab3198f

5 files changed

Lines changed: 256 additions & 71 deletions

File tree

‎examples/models/muse-glimmer/runtime/mlx_executor.cpp‎

Lines changed: 65 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -659,7 +659,6 @@ std::optional<batching::Token> MuseGlimmerMLXExecutor::sample_row(
659659
int row,
660660
batching::SessionId session,
661661
batching::Position prediction_position,
662-
std::vector<float>* probabilities,
663662
bool draft) {
664663
const auto it = sessions_.find(session);
665664
if (it == sessions_.end() || !it->second.sampling || row < 0 ||
@@ -675,18 +674,10 @@ std::optional<batching::Token> MuseGlimmerMLXExecutor::sample_row(
675674
})) {
676675
return std::nullopt;
677676
}
678-
if (draft) {
677+
if (draft || params.temperature == 0) {
679678
return muse_glimmer::argmax_index(values, spec_->vocab_size);
680679
}
681-
std::vector<float> scratch;
682-
auto& p = probabilities ? *probabilities : scratch;
683-
if (params.temperature == 0) {
684-
const auto token = muse_glimmer::argmax_index(values, spec_->vocab_size);
685-
p.assign(spec_->vocab_size, 0.0f);
686-
p[token] = 1.0f;
687-
return token;
688-
}
689-
p = muse_glimmer::sampling_probabilities(
680+
const auto p = muse_glimmer::sampling_probabilities(
690681
values,
691682
spec_->vocab_size,
692683
params.temperature,
@@ -879,8 +870,8 @@ bool MuseGlimmerMLXExecutor::draft(
879870
}
880871
} else {
881872
for (int j = 1; j < v; ++j) {
882-
auto token = sample_row(
883-
logits, i * b + j, owners[i], ends[i] + j, nullptr, true);
873+
auto token =
874+
sample_row(logits, i * b + j, owners[i], ends[i] + j, true);
884875
if (!token) {
885876
return false;
886877
}
@@ -903,7 +894,8 @@ bool MuseGlimmerMLXExecutor::draft(
903894

904895
bool MuseGlimmerMLXExecutor::speculate(
905896
const batching::BatchInput& batch,
906-
batching::BatchOutput& out) {
897+
batching::BatchOutput& out,
898+
bool input_anchors) {
907899
const int b = dflash_->verify_block;
908900
size_t cursor = 0;
909901
while (cursor < batch.inputs.size()) {
@@ -914,7 +906,8 @@ bool MuseGlimmerMLXExecutor::speculate(
914906
while (cursor < batch.inputs.size()) {
915907
const auto& input = batch.inputs[cursor];
916908
auto& state = sessions_.at(input.sid);
917-
const int end = input.position + input.offset + input.size;
909+
const int end = input.position + input.offset + input.size -
910+
static_cast<int>(input_anchors);
918911
if (!out.outputs[cursor] || end != ctl_->pos(state.seq_id) ||
919912
end > spec_->max_context_length - dflash_->block) {
920913
++cursor;
@@ -930,7 +923,10 @@ bool MuseGlimmerMLXExecutor::speculate(
930923
}
931924
context += count;
932925
owners.push_back(input.sid);
933-
anchors.push_back(out.outputs[cursor]->tokens.front());
926+
anchors.push_back(
927+
input_anchors ? (*std::get<batching::TokenInputPtr>(
928+
input.payload))[input.offset]
929+
: out.outputs[cursor]->tokens.front());
934930
indices.push_back(cursor++);
935931
}
936932
if (owners.empty()) {
@@ -940,9 +936,8 @@ bool MuseGlimmerMLXExecutor::speculate(
940936
if (!draft(owners, anchors, dflash_->context, &verification)) {
941937
return false;
942938
}
943-
Probabilities p(verification.inputs.size());
944939
batching::BatchOutput predictions;
945-
if (!execute_impl(verification, predictions, &p)) {
940+
if (!execute_impl(verification, predictions, true)) {
946941
return false;
947942
}
948943
for (size_t i = 0; i < owners.size(); ++i) {
@@ -951,28 +946,18 @@ bool MuseGlimmerMLXExecutor::speculate(
951946
const auto& input = verification.inputs[i * b];
952947
const auto& block = *std::get<batching::TokenInputPtr>(input.payload);
953948
for (int j = 0; j < b; ++j) {
954-
auto& target = p[i * b + j];
955-
const int position = input.position + j + 1;
956-
if (j + 1 < b) {
957-
const auto proposal = block[j + 1];
958-
auto rng = draw_rng(state.seed, position, 2);
959-
if (muse_glimmer::accept_with_probability(rng, target[proposal])) {
960-
output.push_back(proposal);
961-
continue;
962-
}
949+
const auto& prediction = predictions.outputs[i * b + j];
950+
if (!prediction || prediction->tokens.size() != 1) {
951+
return false;
963952
}
964-
auto rng = draw_rng(state.seed, position, j + 1 == b ? 4 : 3);
965-
batching::Token correction;
966-
if (state.sampling->temperature == 0) {
967-
correction = muse_glimmer::argmax_index(target.data(), target.size());
968-
} else if (j + 1 == b) {
969-
correction = muse_glimmer::categorical_sample(
970-
rng, target.data(), target.size());
971-
} else {
972-
correction = muse_glimmer::sample_excluding_token_in_place(
973-
rng, target, block[j + 1]);
953+
const auto token = prediction->tokens.front();
954+
const int position = input.position + j + 1;
955+
// For a greedy draft, matching a target draw accepts with p(proposal).
956+
// Reuse its per-position RNG so batching/chunking cannot change draws.
957+
output.push_back(token);
958+
if (j + 1 < b && token == block[j + 1]) {
959+
continue;
974960
}
975-
output.push_back(correction);
976961
// Leave the returned correction/bonus uncommitted.
977962
if (!rewind(state, position)) {
978963
return false;
@@ -987,13 +972,13 @@ bool MuseGlimmerMLXExecutor::speculate(
987972
bool MuseGlimmerMLXExecutor::execute(
988973
const batching::BatchInput& batch,
989974
batching::BatchOutput& out) {
990-
return execute_impl(batch, out, nullptr);
975+
return execute_impl(batch, out, false);
991976
}
992977

993978
bool MuseGlimmerMLXExecutor::execute_impl(
994979
const batching::BatchInput& batch,
995980
batching::BatchOutput& out,
996-
Probabilities* probabilities) {
981+
bool verification) {
997982
#if ET_HAS_EXCEPTIONS
998983
try {
999984
#endif
@@ -1014,6 +999,42 @@ bool MuseGlimmerMLXExecutor::execute_impl(
1014999
}
10151000
batching::BatchOutput pending;
10161001
pending.outputs.resize(batch.inputs.size());
1002+
// Verification can commit a single input token as the draft anchor.
1003+
if (dflash_ && !verification && !batch.inputs.empty()) {
1004+
std::vector<batching::SessionId> owners;
1005+
bool input_anchors = true;
1006+
for (const auto& input : batch.inputs) {
1007+
const auto& state = sessions_.at(input.sid);
1008+
const int start = input.position + input.offset;
1009+
// Older draft artifacts require at least one new context row.
1010+
if (!input.produce_output || input.size != 1 ||
1011+
!std::holds_alternative<batching::TokenInputPtr>(input.payload) ||
1012+
start != ctl_->pos(state.seq_id) || state.hidden.empty() ||
1013+
start > spec_->max_context_length - dflash_->block ||
1014+
std::find(owners.begin(), owners.end(), input.sid) !=
1015+
owners.end()) {
1016+
input_anchors = false;
1017+
break;
1018+
}
1019+
owners.push_back(input.sid);
1020+
}
1021+
if (input_anchors) {
1022+
for (size_t i = 0; i < batch.inputs.size(); ++i) {
1023+
pending.outputs[i] = batching::Output{batch.inputs[i].sid, {}};
1024+
}
1025+
if (!speculate(batch, pending, true) ||
1026+
std::any_of(
1027+
pending.outputs.begin(),
1028+
pending.outputs.end(),
1029+
[](const auto& output) {
1030+
return !output || output->tokens.empty();
1031+
})) {
1032+
return false;
1033+
}
1034+
out = std::move(pending);
1035+
return true;
1036+
}
1037+
}
10171038
size_t input_index = 0;
10181039
size_t consumed = 0;
10191040
while (input_index < batch.inputs.size()) {
@@ -1055,7 +1076,7 @@ bool MuseGlimmerMLXExecutor::execute_impl(
10551076
selectors.push_back(n - 1);
10561077
}
10571078
const auto rows = static_cast<aten::SizesType>(selectors.size());
1058-
if (dflash_ && !probabilities) {
1079+
if (dflash_ && !verification) {
10591080
for (const auto& slice : slices) {
10601081
if (!drain(slice.sid)) {
10611082
return false;
@@ -1119,20 +1140,16 @@ bool MuseGlimmerMLXExecutor::execute_impl(
11191140
const auto& input = batch.inputs[index];
11201141
const auto end = static_cast<batching::Position>(
11211142
static_cast<int64_t>(input.position) + input.offset + input.size);
1122-
const auto token = sample_row(
1123-
logits,
1124-
static_cast<int>(row),
1125-
input.sid,
1126-
end,
1127-
probabilities ? &probabilities->at(index) : nullptr);
1143+
const auto token =
1144+
sample_row(logits, static_cast<int>(row), input.sid, end);
11281145
if (!token) {
11291146
return false;
11301147
}
11311148
// This singleton prediction remains uncommitted until runner feedback.
11321149
pending.outputs[index] = batching::Output{input.sid, {*token}};
11331150
}
11341151
}
1135-
if (dflash_ && !probabilities && !speculate(batch, pending)) {
1152+
if (dflash_ && !verification && !speculate(batch, pending)) {
11361153
return false;
11371154
}
11381155
out = std::move(pending);

‎examples/models/muse-glimmer/runtime/mlx_executor.h‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,6 @@ class MuseGlimmerMLXExecutor final : public batching::Executor {
4646

4747
private:
4848
struct DFlash;
49-
using Probabilities = std::vector<std::vector<float>>;
5049
struct SessionState {
5150
int32_t seq_id;
5251
std::optional<batching::SamplingParams> sampling;
@@ -75,17 +74,19 @@ class MuseGlimmerMLXExecutor final : public batching::Executor {
7574
const std::vector<batching::Token>& anchors,
7675
int context_limit,
7776
batching::BatchInput* verification);
78-
bool speculate(const batching::BatchInput& batch, batching::BatchOutput& out);
77+
bool speculate(
78+
const batching::BatchInput& batch,
79+
batching::BatchOutput& out,
80+
bool input_anchors = false);
7981
bool execute_impl(
8082
const batching::BatchInput& batch,
8183
batching::BatchOutput& out,
82-
Probabilities* probabilities);
84+
bool verification);
8385
std::optional<batching::Token> sample_row(
8486
aten::Tensor& logits,
8587
int row,
8688
batching::SessionId session,
8789
batching::Position prediction_position,
88-
std::vector<float>* probabilities = nullptr,
8990
bool draft = false);
9091

9192
// Reverse destruction: vision before its module/mutex, then the module before

‎examples/models/muse-glimmer/tests/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,11 @@ ctest --test-dir cmake-out/examples/models/muse-glimmer-batching \
2626
Random and quantized PTE fixtures compare draft numerics against eager references.
2727
Controlled rejection and cache-lifecycle checks assert expected tokens and
2828
acceptance counts; continuation checks compare with target-only execution,
29-
including legacy artifacts.
29+
including legacy artifacts. Seeded continuations are compared across
30+
prompt chunking and mixed prefill/decode batches for both draft formats.
3031
Routing-sensitive fixtures also compare per-round tokens and acceptance counts
3132
between solo and packed execution with unequal prompts and reordered owners.
33+
Input-sensitive continuations check shared-payload slices against target-only decoding.
3234
A vendored stateless official reference separately checks DFlash2 model semantics,
3335
including sliding-window boundaries. Native sampling checks target-only,
3436
original DFlash, and DFlash2 at temperature 0.8 with top-k/top-p filtering;

‎examples/models/muse-glimmer/tests/export_mlx_dflash_fixture.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -377,7 +377,7 @@ def export_fixture(
377377
)
378378

379379
directory.mkdir(parents=True, exist_ok=True)
380-
if not controlled:
380+
if not controlled and not legacy:
381381
write_draft_reference(directory, target, draft, config, draft_config)
382382

383383
original_constants = export_dflash._dflash_constant_methods
@@ -413,6 +413,7 @@ def constants(**kwargs):
413413
write_nonconstant_selector_fixture(args.output)
414414
export_fixture(args.output / "legacy", -1, legacy=True)
415415
export_fixture(args.output / "random", -1, controlled=False)
416+
export_fixture(args.output / "random_legacy", -1, legacy=True, controlled=False)
416417
export_fixture(args.output / "owner_sensitive", -1, owner_sensitive=True)
417418
export_fixture(args.output / "single_proposal", -1, block_size=2)
418419
export_fixture(args.output / "routing", -1, routing_sensitive=True)

0 commit comments

Comments
 (0)