You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 88e6748
Browse filesBrowse the repository at this point in the historyBrowse files
[executorch][native-ggml] Share the ggml runners' engine flags
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom):
* #25381
* #25380
* #25379
* #25378
* #25377
* #25376
* #25375
* #25374
* #25373
* #25372
* #25371
* __->__ #25370
* #25369
* #25368
run_hf_llm_ggml and ggml_ptn_runner take their engine flags (--device,
--device_index, --cpu_threads, --max_cached_graphs, --print_graph_ops) from one
GgmlEngineFlags library, and the README gains a table of the engine options
that change numerics.
Both runners then accept the same flags with the same meaning and help, and a
later option is added and documented once.
Authored with assistance from Claude and Codex.
Differential Revision: D133480460
Pull Request resolved: #25370
Copy file name to clipboardExpand all lines: backends/native/README.md
+9Lines changed: 9 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -268,6 +268,15 @@ ExecuTorch with the desired GGML backend (for example, `GGML_CUDA=ON`,
268
268
`GGML_HIP=ON`, or `GGML_METAL=ON`) and set `ptn::GgmlEngineOptions::device` to
269
269
`ptn::GgmlDevice::Gpu`. CPU support is required even when selecting a GPU.
270
270
271
+
The ggml runners (`run_hf_llm_ggml`, `ggml_ptn_runner`) share their engine
272
+
flags (`runtime/engine/ggml/GgmlEngineFlags.h`). Engine options that change
273
+
numerics default to llama.cpp's behaviour; the testing ones end in
274
+
`_for_testing` and turn on eager's exact behaviour where ggml's differs.
275
+
276
+
| Option (flag) | Default, as in llama.cpp | Set | Ops | Backends |
277
+
| --- | --- | --- | --- | --- |
278
+
|`allow_activation_quant_for_weight_only`| quantizes the activations of matmuls with weight-only quantized weights in their block format | false (debugging): those weights load dequantized to F32, about 4-12x their packed size by type |`linear`| all |
279
+
271
280
The CMake installation includes the GGML libraries and package configuration.
272
281
It also copies the submodule's upstream MIT license, including its copyright
273
282
notice, to `share/licenses/executorch/ggml/LICENSE` under the installation
0 commit comments