Add mlx-lm as a third server, and settle the quant and peg-native anomalies

README.md

@@ -50,10 +50,28 @@ Check a result file against the published schema:
 cargo run -p willitcall -- validate willitcall-result.json
 ```
 
-`--server` selects a preset (`llamacpp`, `ollama`, `lmstudio`, `vllm`,
-`custom`). The preset only supplies request defaults; the preset name is
+`--server` selects a preset (`llamacpp`, `ollama`, `mlx-lm`, `lmstudio`,
+`vllm`, `custom`). The preset only supplies request defaults; the preset name is
 recorded in the result file so results stay comparable.
 
+The `mlx-lm` preset defaults to port 8081, not mlx-lm's own default of 8080,
+because 8080 is this project's llama.cpp convention and two servers on one port
+is exactly the contention the preflight exists to catch.
+
+Two things to know before reading or adding an mlx-lm row:
+
+- **MLX rows are converted weights.** MLX does not consume GGUF, so an MLX row
+  for a model is not the same bits as the llama.cpp row for that model. The
+  trick used elsewhere in this project of serving one blob through two servers
+  to hold the weights constant does not work across this boundary. An MLX-vs-
+  GGUF difference includes the conversion.
+- **`/v1/models` on mlx-lm lists the whole local cache, not the loaded model.**
+  llama.cpp reports the model it is serving; mlx-lm enumerates everything in the
+  HuggingFace cache, and it will load whichever model your request names. Do not
+  discover the model id from that endpoint - pass the repo id you intend to
+  measure. Getting this wrong files a row under the wrong model name, which is
+  worse than having no row.
+
 ## What the scenarios test
 
 The corpus is 50 scenarios in six categories. Every scenario is plain TOML data
@@ -79,6 +97,41 @@ Read a red cell carefully: a red in `parallel` means the model did not emit
 several tool calls in one response, which is the capability that column
 measures. It does not mean the model is broken.
 
+## A cell is a property of the whole stack
+
+**The servers do not decode the same way, so a green on one server and a red on
+another is not by itself evidence about the model.**
+
+llama.cpp compiles the tool definitions you send into a GBNF grammar and
+constrains decoding with it. A tool call that names a function you did not
+supply, or whose arguments do not fit the schema, is not merely unlikely there:
+it cannot be sampled. Ollama and mlx-lm generate unconstrained text and parse a
+tool call out of it afterwards, so the model can emit a wrong function name or a
+malformed call, and the server finds out only after the fact.
+
+That difference is systematic and it favours llama.cpp in every row, on every
+model. So:
+
+- A llama.cpp-versus-Ollama delta is a property of the stack. Read it as "this
+  combination works", not as "Ollama is defective" or "this model is worse than
+  that one".
+- The comparison that isolates the model is same-server, not cross-server.
+- A red under an unconstrained server can mean the model emitted something
+  nearly right that the parser then rejected. The `unparsed_tool_call` failure
+  class exists to mark exactly that case, and the transcript shows the bytes.
+
+Each result records which side of this line its server sits on, in
+`server.quirk_flags`: `grammar_constrained_decoding` for llama.cpp,
+`unconstrained_post_hoc_parse` for Ollama and mlx-lm. LM Studio and vLLM are
+unflagged because their decode path has not been verified here; absence of a
+flag means unverified, not unconstrained.
+
+This was established the hard way. An earlier version of this project published
+a claim that Ollama discarded valid tool calls. Recovering the discarded bytes
+showed the model had emitted the tool's *description* where its name belonged,
+and Ollama's parser was right to reject it. The claim was retracted. The real
+finding is the mechanism above.
+
 ## The scenario-authoring rule
 
 **A scenario that a fully correct model could fail is a bug in the scenario.**

crates/wic-core/src/runner.rs

@@ -44,15 +44,21 @@ pub struct ServerVersionProbe {
 }
 
 impl RunConfig {
-    pub fn new(endpoint: String, model: String, timeout: Duration) -> Self {
+    pub fn new(
+        endpoint: String,
+        model: String,
+        timeout: Duration,
+        seed: u64,
+        temperature: f64,
+    ) -> Self {
         Self {
             endpoint,
             model,
             timeout,
             sampling: SamplingParams {
-                temperature: Some(0.0),
+                temperature: Some(temperature),
                 top_p: Some(1.0),
-                seed: Some(42),
+                seed: Some(seed),
                 max_tokens: Some(1024),
             },
             server: ServerConfig {
@@ -643,6 +649,8 @@ mod tests {
             "http://127.0.0.1:8080/v1".to_owned(),
             "fixture-model".to_owned(),
             std::time::Duration::from_secs(60),
+            42,
+            0.0,
         )
         .with_host_hardware_class(Some("Fixture workstation, 32GB".to_owned()));
 
@@ -659,6 +667,8 @@ mod tests {
             "http://127.0.0.1:8080/v1".to_owned(),
             "fixture-model".to_owned(),
             std::time::Duration::from_secs(60),
+            42,
+            0.0,
         );
         assert_eq!(config.declared_quant, None);
 

crates/wic-core/tests/empty_response.rs

@@ -54,6 +54,8 @@ async fn runner_classifies_only_empty_responses() {
             server.endpoint(),
             "fixture-model".to_owned(),
             Duration::from_secs(5),
+            42,
+            0.0,
         );
 
         let result = run_scenarios(
@@ -83,6 +85,8 @@ async fn empty_response_preserves_negative_trap_pass() {
         server.endpoint(),
         "fixture-model".to_owned(),
         Duration::from_secs(5),
+        42,
+        0.0,
     );
 
     let result = run_scenarios(
@@ -110,6 +114,8 @@ async fn runner_classifies_unparsed_tool_calls_without_changing_status() {
         server.endpoint(),
         "fixture-model".to_owned(),
         Duration::from_secs(5),
+        42,
+        0.0,
     );
 
     let result = run_scenarios(

crates/wic-core/tests/evidence.rs

@@ -79,6 +79,8 @@ async fn written_transcript_redacts_sensitive_request_headers() {
         server.endpoint(),
         "fixture-model".to_owned(),
         Duration::from_secs(5),
+        42,
+        0.0,
     );
     config.request_headers.insert(
         AUTHORIZATION,
@@ -124,6 +126,8 @@ async fn written_transcript_preserves_sse_body_bytes_exactly() {
         server.endpoint(),
         "fixture-model".to_owned(),
         Duration::from_secs(5),
+        42,
+        0.0,
     );
 
     let result = run_scenarios(
@@ -167,6 +171,8 @@ async fn fresh_result_paths_exist_and_hash_transcript_bytes() {
         server.endpoint(),
         "fixture-model".to_owned(),
         Duration::from_secs(5),
+        42,
+        0.0,
     );
 
     let result = run_scenarios(
@@ -208,6 +214,8 @@ async fn written_transcript_matches_checked_in_schema() {
         server.endpoint(),
         "fixture-model".to_owned(),
         Duration::from_secs(5),
+        42,
+        0.0,
     );
 
     let result = run_scenarios(

crates/willitcall/src/main.rs

@@ -32,6 +32,7 @@ const EXIT_CODE_HELP: &str = "Exit codes:\n  0  all scenarios passed\n  1  at le
 const KNOWN_INFERENCE_SERVERS: &[(u16, &str)] = &[
     (11434, "Ollama"),
     (8080, "llama.cpp"),
+    (8081, "MLX LM"),
     (1234, "LM Studio"),
     (8000, "vLLM"),
 ];
@@ -67,6 +68,10 @@ struct RunArgs {
     out: PathBuf,
     #[arg(long, default_value_t = 60, value_parser = clap::value_parser!(u64).range(1..))]
     timeout: u64,
+    #[arg(long, default_value_t = 42)]
+    seed: u64,
+    #[arg(long, default_value_t = 0.0, allow_negative_numbers = true)]
+    temperature: f64,
     #[arg(long)]
     json: bool,
     #[arg(long)]
@@ -83,6 +88,7 @@ struct RunArgs {
 enum ServerPreset {
     Llamacpp,
     Ollama,
+    MlxLm,
     Lmstudio,
     Vllm,
     Custom,
@@ -93,6 +99,7 @@ impl ServerPreset {
         match self {
             Self::Llamacpp => "llamacpp",
             Self::Ollama => "ollama",
+            Self::MlxLm => "mlx_lm",
             Self::Lmstudio => "lmstudio",
             Self::Vllm => "vllm",
             Self::Custom => "custom",
@@ -103,6 +110,7 @@ impl ServerPreset {
         match self {
             Self::Llamacpp => Some("http://127.0.0.1:8080/v1"),
             Self::Ollama => Some("http://127.0.0.1:11434/v1"),
+            Self::MlxLm => Some("http://127.0.0.1:8081/v1"),
             Self::Lmstudio => Some("http://127.0.0.1:1234/v1"),
             Self::Vllm => Some("http://127.0.0.1:8000/v1"),
             Self::Custom => None,
@@ -123,14 +131,21 @@ impl ServerPreset {
                 path: "/version",
                 field: "version",
             }),
-            Self::Lmstudio | Self::Custom => None,
+            Self::MlxLm | Self::Lmstudio | Self::Custom => None,
         }
     }
 
     fn config(self) -> ServerConfig {
         ServerConfig {
             preset_name: self.name().to_owned(),
-            quirk_flags: Vec::new(),
+            // llama.cpp builds GBNF from tools; Ollama and mlx-lm generate unconstrained
+            // text and parse afterward. In mlx-lm 0.31.3 tools only reach the chat
+            // template; there is no grammar/logit mask, and parse failures are swallowed.
+            quirk_flags: match self {
+                Self::Llamacpp => vec!["grammar_constrained_decoding".to_owned()],
+                Self::Ollama | Self::MlxLm => vec!["unconstrained_post_hoc_parse".to_owned()],
+                Self::Lmstudio | Self::Vllm | Self::Custom => Vec::new(),
+            },
             version_probe: self.version_probe(),
         }
     }
@@ -393,6 +408,11 @@ async fn execute_with_known_servers(
 ) -> Result<u8, ExecuteError> {
     match cli.command {
         Command::Run(args) => {
+            if args.temperature < 0.0 {
+                return Err(ExecuteError::Usage(
+                    "temperature must be non-negative".to_owned(),
+                ));
+            }
             let scenarios = match args.scenarios {
                 Some(path) => load_scenarios_from_dir(&path),
                 None => load_embedded_scenarios(),
@@ -419,10 +439,16 @@ async fn execute_with_known_servers(
                     "another inference server is responding on {endpoints}; {stop}, or re-run with --force"
                 )));
             }
-            let config = RunConfig::new(endpoint, args.model, Duration::from_secs(args.timeout))
-                .with_server(args.server.config())
-                .with_host_hardware_class(args.host_hardware_class)
-                .with_declared_quant(args.quant);
+            let config = RunConfig::new(
+                endpoint,
+                args.model,
+                Duration::from_secs(args.timeout),
+                args.seed,
+                args.temperature,
+            )
+            .with_server(args.server.config())
+            .with_host_hardware_class(args.host_hardware_class)
+            .with_declared_quant(args.quant);
             preflight(&config).await.map_err(ExecuteError::Preflight)?;
             let mut result = run_scenarios(&config, &scenarios, &args.out)
                 .await
@@ -848,6 +874,7 @@ mod tests {
 
     #[test]
     fn server_presets_select_defaults_and_endpoint_overrides_them() {
+        assert_eq!(ServerPreset::MlxLm.name(), "mlx_lm");
         assert_eq!(
             resolve_endpoint(ServerPreset::Llamacpp, None).expect("llamacpp endpoint"),
             "http://127.0.0.1:8080/v1"
@@ -864,6 +891,10 @@ mod tests {
             resolve_endpoint(ServerPreset::Vllm, None).expect("vllm endpoint"),
             "http://127.0.0.1:8000/v1"
         );
+        assert_eq!(
+            resolve_endpoint(ServerPreset::MlxLm, None).expect("mlx_lm endpoint"),
+            "http://127.0.0.1:8081/v1"
+        );
         assert!(resolve_endpoint(ServerPreset::Custom, None).is_err());
         assert_eq!(
             resolve_endpoint(

crates/willitcall/src/site.rs

@@ -97,6 +97,9 @@ fn render_index(results: &[ResultFile], repo_base: &str) -> String {
         .collect::<BTreeSet<_>>()
         .len();
     let case_studies_url = format!("{repo_base}/tree/main/docs/case-studies");
+    let peg_native_case_study_url = format!(
+        "{repo_base}/blob/main/docs/case-studies/2026-07-21-llamacpp-500s-on-llama-3.1-tool-calls.md"
+    );
     let uniform_environment = results
         .first()
         .and_then(|result| result.result.metadata.environment.as_ref())
@@ -135,7 +138,12 @@ fn render_index(results: &[ResultFile], repo_base: &str) -> String {
       <p>A cell measures the whole stack: model x quant x server x server version. It is not a property of the model alone.</p>
       <p>Red means the combination failed as tested, not that the weights are bad. The same weights can pass on one server and fail on another; where that is proven, the cell carries a cause annotation.</p>
       <p>Every red cell links to the full request/response transcript that produced it when the result schema supplies a transcript path. Legacy schema v1 results do not record transcript paths. See the <a href="{}">case studies under docs/case-studies/</a> for controlled comparisons.</p>
-      <p>Sample size and method: {} distinct scenarios are represented in this result set. Each published cell is one run. Findings in the case studies are replicated across at least five runs per arm before a verdict is drawn, so a cell tells you what one run measured and a case study tells you what held up under repetition.</p>
+      <p>The servers do not decode the same way. llama.cpp compiles the supplied tool definitions into a GBNF grammar and constrains decoding with it, so a call naming a function that was never supplied cannot be sampled there. Ollama and MLX LM generate unconstrained text and parse the tool call out of it afterwards. This systematically favours llama.cpp, so a llama.cpp-versus-Ollama difference is a property of the combination, not evidence of a server defect or a difference between models; the comparison that isolates the model is same-server.</p>
+      <p>Sample size and method: {} distinct scenarios are represented in this result set. Each published cell is one run. Findings in the case studies are replicated across at least five runs per arm before a verdict is drawn, so a cell tells you what one run measured and a case study tells you what held up under repetition. The current case studies cover 90 runs across 18 quantization arms, and 40 runs across 8 arms for the peg-native anomaly.</p>
+      <h2>Excluded rows</h2>
+      <ul>
+        <li>Meta-Llama-3.1-8B-Instruct on llama.cpp (Q8_0, Q4_K_M, Q3_K_M) is excluded from the quantization conclusion because llama.cpp returns HTTP 500 on 7-9 of 50 scenarios per run for this model ("does not match the expected peg-native format"). These are server errors, not model failures, and are not comparable across arms. See the <a href="{}">peg-native case study</a>.</li>
+      </ul>
 {}
     </section>
 
@@ -150,6 +158,7 @@ fn render_index(results: &[ResultFile], repo_base: &str) -> String {
             <option value="all">All</option>
             <option value="ollama">Ollama</option>
             <option value="llamacpp">llama.cpp</option>
+            <option value="mlx_lm">MLX LM</option>
           </select>
         </label>
       </div>
@@ -167,6 +176,7 @@ fn render_index(results: &[ResultFile], repo_base: &str) -> String {
 "#,
         escape_html(&case_studies_url),
         scenario_count,
+        escape_html(&peg_native_case_study_url),
         environment_statement,
         results.len()
     )
@@ -501,6 +511,8 @@ fn model_label(file_name: &str, server: &str) -> String {
 fn display_server(server: &str) -> &str {
     if server == "llamacpp" {
         "llama.cpp"
+    } else if server == "mlx_lm" {
+        "MLX LM"
     } else {
         server
     }