Hi @antirez,
Thank you so much for your great work. I have run ds4 Q2 model on my Mac Studio 128GB. The only issue I came across is the prefills which make the prompt very slow. This happens when I try to ask Claude to summarize my code base and give plans. Any idea how I can improve the performance?
./ds4-server
--model ds4flash.gguf
--host 0.0.0.0
--port 8080
--ctx 65536
--kv-disk-dir ./cache/kv
--kv-disk-space-mb 8192
ds4: Metal device Apple M4 Max, 128.00 GiB RAM
ds4: Metal 4 tensor API disabled for pre-M5/pre-A19 devices
ds4: drift-patch flags hc_stable=on norm_unify=on kv_raw_f32=off rope_exp2_log2=off math_safe=off tensor_matmul=off
ds4: requesting Metal residency (may take tens of seconds)... done
ds4: warming Metal model views... done
ds4: Metal model views created in 1.617 ms, residency requested in 31074.930 ms, warmup 4.012 ms (mapped 93065.67 MiB from offset 5.09 MiB)
ds4: Metal mapped mmaped model as 2 overlapping shared buffers
ds4: metal backend initialized for graph diagnostics
0705 16:36:50 ds4-server: context buffers 1393.67 MiB (ctx=65536, backend=metal, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=16386)
0705 16:36:50 ds4-server: KV disk cache ./cache/kv (budget=8192 MiB, cross-quant=accept, min=512, cold_max=30000, continued=10000, trim=32, align=2048, hit_half_life=21600s)
0705 16:36:50 ds4-server: listening on http://0.0.0.0:8080
0705 16:42:19 ds4-server: chat ctx=0..24527:24527 TOOLS prompt start
0705 16:42:19 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 0/24527 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
0705 16:42:32 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 4096/24527 (16.7%) chunk=316.37 t/s avg=316.35 t/s 12.947s
0705 16:42:45 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 8192/24527 (33.4%) chunk=312.70 t/s avg=314.52 t/s 26.046s
0705 16:42:59 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 12288/24527 (50.1%) chunk=300.16 t/s avg=309.58 t/s 39.692s
0705 16:43:13 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 16384/24527 (66.8%) chunk=293.22 t/s avg=305.32 t/s 53.661s
0705 16:43:28 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 20102/24527 (82.0%) chunk=249.51 t/s avg=293.19 t/s 68.562s
0705 16:43:28 ds4-server: kv cache evicted reason=disk-cache-full tokens=10240 hits=0 size=157.34 MiB file=./cache/kv/be1abb4b15f7cf5863991377b9c8e9efd09fb794.kv
0705 16:43:28 ds4-server: kv cache stored tokens=20102 trimmed=4425 reason=cold key=token-text size=286.80 MiB save=34.9 ms
0705 16:43:30 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 20480/24527 (83.5%) chunk=157.21 t/s avg=288.59 t/s 70.967s
0705 16:43:30 ds4-server: kv cache evicted reason=disk-cache-full tokens=20102 hits=0 size=286.80 MiB file=./cache/kv/34ebbe2c6391c97a13048c1d08cf4b28f84a4908.kv
0705 16:43:30 ds4-server: kv cache stored tokens=20480 trimmed=0 reason=continued key=token-text size=291.79 MiB save=29.1 ms
0705 16:43:47 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 24527/24527 (100.0%) chunk=242.85 t/s avg=279.89 t/s 87.631s
0705 16:43:47 ds4-server: chat ctx=0..24527:24527 TOOLS prompt done 87.631s
0705 16:43:49 ds4-server: chat ctx=24527..24577:50 gen=50 TOOLS DSML_START decoding chunk=23.74 t/s avg=23.74 t/s 2.106s
0705 16:43:51 ds4-server: chat ctx=24577..24627:50 gen=100 TOOLS DSML_START decoding chunk=24.06 t/s avg=23.90 t/s 4.185s
0705 16:43:53 ds4-server: chat ctx=24627..24677:50 gen=150 TOOLS DSML_START decoding chunk=23.79 t/s avg=23.86 t/s 6.287s
0705 16:43:55 ds4-server: chat ctx=24677..24727:50 gen=200 TOOLS DSML_START decoding chunk=23.81 t/s avg=23.85 t/s 8.387s
0705 16:43:55 ds4-server: chat ctx=24727..24732:5 gen=205 TOOLS DSML_START DSML_END decoding chunk=23.76 t/s avg=23.85 t/s 8.597s
0705 16:43:55 ds4-server: tool calls ctx=0..24527:24527 n=3 raw_dsml=1 ids=[toolu_c328cf6a3554972802794e57c9042e6e,toolu_bef80c046ceba19c46b6bba0b13b8770,toolu_fb6e0342892d21c8dd6c8aa74cdb097e] names=[Read,Read,Read]
0705 16:43:55 ds4-server: chat ctx=0..24527:24527 gen=205 TOOLS DSML_START DSML_END finish=tool_calls 96.229s
0705 16:43:55 ds4-server: anthropic live continuation match=tool-output-ids ids=3 cached=24732 prompt=39299
0705 16:43:55 ds4-server: chat ctx=24732..39299:14567 TOOLS prompt start
0705 16:43:55 ds4-server: chat ctx=24732..39299:14567 TOOLS prefill chunk 0/14567 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.000s
0705 16:44:10 ds4-server: chat ctx=24732..39299:14567 TOOLS prefill chunk 3940/14567 (27.0%) chunk=265.76 t/s avg=265.75 t/s 14.826s
0705 16:44:25 ds4-server: chat ctx=24732..39299:14567 TOOLS prefill chunk 8036/14567 (55.2%) chunk=266.76 t/s avg=266.27 t/s 30.180s
0705 16:44:41 ds4-server: chat ctx=24732..39299:14567 TOOLS prefill chunk 12132/14567 (83.3%) chunk=259.29 t/s avg=263.87 t/s 45.977s
0705 16:44:52 ds4-server: chat ctx=24732..39299:14567 TOOLS prefill chunk 14567/14567 (100.0%) chunk=221.09 t/s avg=255.60 t/s 56.991s
0705 16:44:52 ds4-server: chat ctx=24732..39299:14567 TOOLS prompt done 56.991s
Hi @antirez,
Thank you so much for your great work. I have run ds4 Q2 model on my Mac Studio 128GB. The only issue I came across is the prefills which make the prompt very slow. This happens when I try to ask Claude to summarize my code base and give plans. Any idea how I can improve the performance?
./ds4-server
--model ds4flash.gguf
--host 0.0.0.0
--port 8080
--ctx 65536
--kv-disk-dir ./cache/kv
--kv-disk-space-mb 8192
ds4: Metal device Apple M4 Max, 128.00 GiB RAM
ds4: Metal 4 tensor API disabled for pre-M5/pre-A19 devices
ds4: drift-patch flags hc_stable=on norm_unify=on kv_raw_f32=off rope_exp2_log2=off math_safe=off tensor_matmul=off
ds4: requesting Metal residency (may take tens of seconds)... done
ds4: warming Metal model views... done
ds4: Metal model views created in 1.617 ms, residency requested in 31074.930 ms, warmup 4.012 ms (mapped 93065.67 MiB from offset 5.09 MiB)
ds4: Metal mapped mmaped model as 2 overlapping shared buffers
ds4: metal backend initialized for graph diagnostics
0705 16:36:50 ds4-server: context buffers 1393.67 MiB (ctx=65536, backend=metal, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=16386)
0705 16:36:50 ds4-server: KV disk cache ./cache/kv (budget=8192 MiB, cross-quant=accept, min=512, cold_max=30000, continued=10000, trim=32, align=2048, hit_half_life=21600s)
0705 16:36:50 ds4-server: listening on http://0.0.0.0:8080
0705 16:42:19 ds4-server: chat ctx=0..24527:24527 TOOLS prompt start
0705 16:42:19 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 0/24527 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
0705 16:42:32 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 4096/24527 (16.7%) chunk=316.37 t/s avg=316.35 t/s 12.947s
0705 16:42:45 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 8192/24527 (33.4%) chunk=312.70 t/s avg=314.52 t/s 26.046s
0705 16:42:59 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 12288/24527 (50.1%) chunk=300.16 t/s avg=309.58 t/s 39.692s
0705 16:43:13 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 16384/24527 (66.8%) chunk=293.22 t/s avg=305.32 t/s 53.661s
0705 16:43:28 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 20102/24527 (82.0%) chunk=249.51 t/s avg=293.19 t/s 68.562s
0705 16:43:28 ds4-server: kv cache evicted reason=disk-cache-full tokens=10240 hits=0 size=157.34 MiB file=./cache/kv/be1abb4b15f7cf5863991377b9c8e9efd09fb794.kv
0705 16:43:28 ds4-server: kv cache stored tokens=20102 trimmed=4425 reason=cold key=token-text size=286.80 MiB save=34.9 ms
0705 16:43:30 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 20480/24527 (83.5%) chunk=157.21 t/s avg=288.59 t/s 70.967s
0705 16:43:30 ds4-server: kv cache evicted reason=disk-cache-full tokens=20102 hits=0 size=286.80 MiB file=./cache/kv/34ebbe2c6391c97a13048c1d08cf4b28f84a4908.kv
0705 16:43:30 ds4-server: kv cache stored tokens=20480 trimmed=0 reason=continued key=token-text size=291.79 MiB save=29.1 ms
0705 16:43:47 ds4-server: chat ctx=0..24527:24527 TOOLS prefill chunk 24527/24527 (100.0%) chunk=242.85 t/s avg=279.89 t/s 87.631s
0705 16:43:47 ds4-server: chat ctx=0..24527:24527 TOOLS prompt done 87.631s
0705 16:43:49 ds4-server: chat ctx=24527..24577:50 gen=50 TOOLS DSML_START decoding chunk=23.74 t/s avg=23.74 t/s 2.106s
0705 16:43:51 ds4-server: chat ctx=24577..24627:50 gen=100 TOOLS DSML_START decoding chunk=24.06 t/s avg=23.90 t/s 4.185s
0705 16:43:53 ds4-server: chat ctx=24627..24677:50 gen=150 TOOLS DSML_START decoding chunk=23.79 t/s avg=23.86 t/s 6.287s
0705 16:43:55 ds4-server: chat ctx=24677..24727:50 gen=200 TOOLS DSML_START decoding chunk=23.81 t/s avg=23.85 t/s 8.387s
0705 16:43:55 ds4-server: chat ctx=24727..24732:5 gen=205 TOOLS DSML_START DSML_END decoding chunk=23.76 t/s avg=23.85 t/s 8.597s
0705 16:43:55 ds4-server: tool calls ctx=0..24527:24527 n=3 raw_dsml=1 ids=[toolu_c328cf6a3554972802794e57c9042e6e,toolu_bef80c046ceba19c46b6bba0b13b8770,toolu_fb6e0342892d21c8dd6c8aa74cdb097e] names=[Read,Read,Read]
0705 16:43:55 ds4-server: chat ctx=0..24527:24527 gen=205 TOOLS DSML_START DSML_END finish=tool_calls 96.229s
0705 16:43:55 ds4-server: anthropic live continuation match=tool-output-ids ids=3 cached=24732 prompt=39299
0705 16:43:55 ds4-server: chat ctx=24732..39299:14567 TOOLS prompt start
0705 16:43:55 ds4-server: chat ctx=24732..39299:14567 TOOLS prefill chunk 0/14567 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.000s
0705 16:44:10 ds4-server: chat ctx=24732..39299:14567 TOOLS prefill chunk 3940/14567 (27.0%) chunk=265.76 t/s avg=265.75 t/s 14.826s
0705 16:44:25 ds4-server: chat ctx=24732..39299:14567 TOOLS prefill chunk 8036/14567 (55.2%) chunk=266.76 t/s avg=266.27 t/s 30.180s
0705 16:44:41 ds4-server: chat ctx=24732..39299:14567 TOOLS prefill chunk 12132/14567 (83.3%) chunk=259.29 t/s avg=263.87 t/s 45.977s
0705 16:44:52 ds4-server: chat ctx=24732..39299:14567 TOOLS prefill chunk 14567/14567 (100.0%) chunk=221.09 t/s avg=255.60 t/s 56.991s
0705 16:44:52 ds4-server: chat ctx=24732..39299:14567 TOOLS prompt done 56.991s