Merge pull request #145 from Raphaeal19/feat/android-benchmark-submission-block

android: add an on-device benchmark that prints the community submiss…

Author: shuk777

android/README.md

@@ -111,7 +111,229 @@ adb shell am instrument -w -e class dev.edge0.runtime.app.LlamaRuntimeTest \
 Coverage: 8B/35B smoke, 35B↔8B in-process switching, prefix-reuse fidelity,
 thinking on/off × system-prompt quadrants (8B gate + 35B off leak probe), and
 identity retention across multi-turn rendering. Host-side logic tests:
-`./gradlew :app:testDebugUnitTest` (32 tests, no device needed).
+`./gradlew :app:testDebugUnitTest` (no device needed).
+
+### 1.7 Benchmark (community submission block)
+
+`BenchTest` measures decode speed, prefill speed and peak memory of a model on a real phone and
+prints the result block used by the
+[Hardware Benchmarks](https://github.com/Edge0-AI/Edge0/discussions/142) Discussion
+(`runtime: android`). Besides the speed and memory numbers, the block records the device model, SoC,
+RAM and Android version and the engine commit, so it is ready to paste.
+
+#### What you need
+
+- A phone connected over USB with USB debugging on (`adb devices` lists it as `device`).
+- The app and the test APK installed (§1.3) and the model staged on the phone (§1.4).
+- A tier that fits the phone: `8b` runs on any 8 GB+ phone, `35b` needs 12 GB+.
+
+#### Quick start
+
+```bash
+./gradlew :app:installDebug :app:installDebugAndroidTest
+bash tools/model/push_models.sh --8b          # or --35b; restage if the app's data was cleared
+bash tools/bench/run_bench.sh --8b --reporter @your-handle
+```
+
+The last command prints the block on stdout and nothing else, so it can be redirected:
+`bash tools/bench/run_bench.sh --8b > block.txt`.
+
+#### The runner script: `tools/bench/run_bench.sh`
+
+```
+bash tools/bench/run_bench.sh --8b | --35b [--reporter @handle] [--notes "text"] [--temperature 0.2]
+```
+
+| Option | Required | Meaning |
+|---|---|---|
+| `--8b` / `--35b` | yes | The tier to run. If both are given, the last one wins. |
+| `--reporter @handle` | no | Your handle, written to the `reporter` field. The `@` is added if missing. |
+| `--notes "text"` | no | Free text appended to the `notes` field (for example `case on, charging`). Quotes and spaces are safe. |
+| `--temperature 0.2` | no | Sampling temperature, 0 (greedy) to 2. The default, 0.2, is the app's own default. See "Sampling". |
+
+| Environment | Meaning |
+|---|---|
+| `ADB_SERIAL` | Which device to use. Required when more than one is attached (`adb devices` shows the serials). |
+
+What it does, in order:
+
+1. Checks that `adb` sees a usable device. If not, it stops with exit code 1 and tells you to check
+   `adb devices` or set `ADB_SERIAL`.
+2. Starts the instrumented test with `adb shell am instrument -w`, passing the tier, reporter and
+   notes. It prints one line to stderr, then stays silent until the run ends (see "Following
+   progress").
+3. Reads the block out of the instrumentation output and prints it to stdout.
+4. If no block was produced, or the test failed, it prints the full instrumentation output to
+   stderr and exits with code 1. The failure text from the test is in that output.
+
+Exit codes: `0` block printed, `1` no device or no block (including a failed test), `2` bad
+arguments.
+
+The script does not check beforehand that the APKs and the model are on the phone. The test
+reports it: a missing model fails with `edge0-<tier>.gguf not found under files/models`, and a
+missing test APK fails with `INSTRUMENTATION_FAILED` and "Unable to find instrumentation info".
+
+Without the script, the same run is:
+
+```bash
+adb shell am instrument -w -e class dev.edge0.runtime.app.BenchTest \
+  -e tier 8b -e reporter @your-handle \
+  dev.edge0.runtime.app.test/androidx.test.runner.AndroidJUnitRunner
+```
+
+Besides the instrumentation output, the test logs the block under the tag `E0Bench` and saves a
+copy to `/sdcard/Android/data/dev.edge0.runtime.app/files/bench-edge0-<tier>.txt` (`adb pull`).
+`BenchTest` does nothing unless `-e tier` is given, so other instrumented runs are unaffected.
+
+#### Running through Gradle
+
+The same test can be run as a Gradle connected test. Install the app and stage the model first, then
+name the test class and pass the arguments as Gradle properties:
+
+```bash
+./gradlew :app:installDebug                 # the app must exist before the model can be staged (§1.4)
+./gradlew :app:connectedDebugAndroidTest \
+  -Pandroid.testInstrumentationRunnerArguments.class=dev.edge0.runtime.app.BenchTest \
+  -Pandroid.testInstrumentationRunnerArguments.tier=8b \
+  -Pandroid.testInstrumentationRunnerArguments.reporter=@your-handle
+```
+
+- Always pass `class=`: without it Gradle runs every instrumented test, including
+  `LlamaRuntimeTest`, which needs both models. Without `tier=`, `BenchTest` skips itself.
+- Gradle reinstalls the app in place, so the model you staged is still there for the run. When the run
+  ends it **uninstalls both APKs, which removes the staged model**, so stage it again for the next run.
+- The console only shows pass or fail. The block is recorded in
+  `app/build/outputs/androidTest-results/connected/debug/<device>/` (`test-result.textproto`, key
+  `submission_block`) and in the logcat attachment next to it, which prints it with:
+
+  ```bash
+  sed -n 's/.*E0Bench *: //p' app/build/outputs/androidTest-results/connected/debug/*/logcat-*BenchTest*.txt
+  ```
+
+- Gradle warns that runner arguments given on the command line are not compatible with its
+  configuration cache. The arguments still reach the test; the run just does not use the cache.
+
+#### What the benchmark does
+
+- **Prompt:** one fixed prompt of about 3.3k tokens, built from the `BENCH_LONG=1` paragraphs of
+  `python/examples/bench.py` plus an instruction that asks for a long answer. Thinking is off.
+- **Protocol:** one cold warmup run, then 2 measured runs with a 5 s pause between them. Each run
+  starts from scratch, so every run pays a full prefill. The block reports the mean of the 2
+  measured runs. The warmup is not averaged, but its numbers and each measured run's are listed in
+  `notes`.
+- **Decode:** exactly 200 new tokens, so every run measures an extended decode, sampled at
+  temperature 0.2 by default (the app's own default; `--temperature 0` is greedy). See "Sampling".
+  The engine cannot be told to ignore end-of-sequence, so a run where the model stops earlier fails
+  instead of reporting a shorter window.
+- **Timing:** prefill and decode speeds come from the engine's own timers, as in the app's
+  metrics line.
+- **Peak memory:** the process's peak resident set (`VmHWM`). The benchmark resets that counter
+  before every run (by writing `5` to `/proc/self/clear_refs`), so each run reports its own peak, and
+  the block shows the highest of the measured runs. If the phone refuses the reset, the number is the
+  whole-process maximum and `notes` says so. It includes memory-mapped model pages that are resident
+  at the time, so it is not the model's size and not comparable to MLX's allocator peak.
+- **Comparable runs only:** a run fails instead of reporting if the prompt is outside 2.8k-3.8k
+  tokens, the prefix cache was reused, or decode did not produce exactly 200 tokens.
+
+Differences from `python/examples/bench.py`: the Android run reports a mean of 2 runs after a cold
+warmup (the Python bench reports its cold and warm runs) and has no 10-token untimed decode warmup
+inside a run; the model may end its answer early, which fails the run (the Python bench always
+decodes 200 tokens); only the temperature is applied when sampling (next section); and peak memory
+is the process's resident peak, not MLX's allocator peak.
+
+#### Sampling
+
+| Setting | `python/examples/bench.py` | Android benchmark |
+|---|---|---|
+| Temperature | `BENCH_TEMP`, default 0.7 | `--temperature`, default 0.2 (the app's default); 0 is greedy |
+| Top-k | model config, default 64 | Not applied. With temperature above 0 the engine always uses top-k 40. |
+| Top-p | model config, default 0.95 | Not applied. The engine has no top-p. |
+| Repetition penalty | model config, default off | Not applied. |
+| Seed | `BENCH_SEED`, optional | Fixed inside the engine. |
+
+The app's Settings screen has top-k and top-p sliders, but the llama path forwards only the
+temperature to the engine (`LlamaRuntime.kt`), so they have no effect there either. Applying them
+needs a change to the engine's decode loop (`llama_chat.c`), which the benchmark does not touch.
+
+#### The block
+
+```
+tier:            edge0-8b
+runtime:         android
+device:          Samsung SM-S911U
+soc:             Snapdragon 8 Gen 2
+ram_gb:          8
+os:              Android 16 (API 36)
+decode_tok_s:    1.14
+prefill_tok_s:   10.3
+peak_active_gib: 3.40
+engine_commit:   7ab4ee7baad2d920464cbacfad4f4b07cf111fd2
+reporter:        @your-handle
+notes:           cold run: decode 1.80 tok/s, prefill 10.4 tok/s, peak 3.54 GiB; run 1: decode 1.56 tok/s, prefill 10.0 tok/s, peak 3.33 GiB; run 2: decode 0.72 tok/s, prefill 10.7 tok/s, peak 3.40 GiB; 3301-token prompt, 200 decoded, mean of 2 reps, temperature 0.2
+```
+
+The block above is the output of one run of the 8B tier on an 8 GB phone, where the 5 GB model does
+not fit in free memory (see below). The screen turned off during run 2, which decoded at less than
+half the speed of run 1; the notes list every run so that an outlier like this stays visible.
+
+| Field | Who fills it |
+|---|---|
+| `tier`, `runtime` | The benchmark. |
+| `decode_tok_s`, `prefill_tok_s`, `peak_active_gib` | The benchmark (means of the 2 measured runs; peak is the highest of the two). |
+| `engine_commit` | The benchmark: the pinned engine commit from `vendor.llama.pin`. |
+| `reporter` | `--reporter`. |
+| `notes` | The benchmark adds the cold run and each measured run (decode, prefill and, when the peak could be reset per run, peak), the prompt length, tokens decoded, run count and temperature; `--notes` is appended. |
+| `device`, `soc`, `ram_gb`, `os` | The benchmark, from the Android framework. `device` is manufacturer and model. `soc` is the chip: known Qualcomm codes are mapped to marketing names (`SM8550` becomes Snapdragon 8 Gen 2), any other chip is shown as vendor and model code. `ram_gb` is total memory rounded up to the marketed size (a phone that reports 7.4 GiB shows 8). `os` is the Android release and API level. |
+
+#### Following progress
+
+The runner prints nothing until the run finishes. To follow progress, watch the engine log in a
+second terminal:
+
+```bash
+adb logcat -s E0LLAMA
+```
+
+The benchmark sends three requests: the warmup, then runs 1 and 2. Each logs a `REND` line when it
+starts, `PRE fill ...` when its prefill is done, and `OK gen=... tps=... prefill_ms=...` when it
+finishes, so the third `OK gen=` line means the run is nearly over.
+
+#### Getting representative numbers
+
+The model file is memory-mapped. When it is larger than the free memory on the phone, the weights
+are read back from flash again and again, and speed drops sharply (an 8 GB phone running the 5 GB
+8B model reaches only a few tokens per second, and the warmup alone can take several minutes). For a
+representative number, reboot, close other apps, keep the screen on for the whole run, and note in
+`--notes` whether the phone was charging and how much memory was free. The notes list every run, so a
+run that was disturbed (for example the screen turning off partway) shows up as an outlier instead of
+being hidden in the mean.
+
+#### Troubleshooting
+
+| Message | Meaning and fix |
+|---|---|
+| `no usable device` | `adb devices` shows nothing usable. Unlock the phone, accept the USB debugging prompt, or set `ADB_SERIAL`. |
+| `INSTRUMENTATION_FAILED` / `Unable to find instrumentation info` | The test APK is not installed. Run `./gradlew :app:installDebugAndroidTest`. |
+| `edge0-<tier>.gguf not found under files/models` | The model is not staged, or the app's data was cleared. Stage it again (§1.4). |
+| `load failed: ...` | The engine could not load the model. The text after it is the engine's reason. |
+| `prefix cache reused N tokens` | The run did not pay a full prefill, so the prefill speed would be wrong. Re-run. |
+| `prompt is N tokens, expected 2800..3800` | The tokenizer produced a different prompt length than the benchmark was designed for. Report it. |
+| `decode stopped after N tokens, need exactly 200` | The model ended its answer early, or the engine hit a decode error (for example a full KV cache). The engine reports such an error as a normal, shorter answer, so the two look the same. Re-run; if it repeats, report it with the tier and the number of tokens. |
+
+#### Code layout
+
+| File | Role |
+|---|---|
+| `app/src/main/java/dev/edge0/runtime/app/bench/Bench.kt` | Tiers, instrumentation arguments, the fixed prompt, the measurement protocol and its results. |
+| `app/src/main/java/dev/edge0/runtime/app/bench/BenchBlock.kt` | Renders the submission block and the per-run notes; holds the device-name, SoC and RAM helpers. |
+| `app/src/main/java/dev/edge0/runtime/app/bench/DeviceProbe.kt` | Reads the device model, SoC, RAM and Android version from the Android framework, and resets the peak-memory counter. |
+| `app/src/androidTest/java/dev/edge0/runtime/app/BenchTest.kt` | The instrumented test: reads the arguments, loads the model, runs the protocol, prints the block. |
+| `app/src/test/java/dev/edge0/runtime/app/bench/` | JVM unit tests for the protocol and the block (no device needed). |
+| `tools/bench/run_bench.sh` | The runner script described above. |
+
+The protocol only depends on the app's `Runtime` interface, so it is unit-tested with a scripted
+runtime. `app/build.gradle.kts` adds a `BuildConfig.ENGINE_COMMIT` field read from
+`../vendor.llama.pin`.
 
 ---
 
@@ -179,6 +401,7 @@ app/                     Android app: Compose UI (src/main/java), JNI shell (src
                          instrumented + unit tests (src/androidTest, src/test)
 tools/llama/             build_vendor_libs.sh — engine rebuild from the pinned tree + bands
 tools/model/             push_models.sh — md5-gated model staging to devices
+tools/bench/             run_bench.sh — on-device benchmark runner (§1.7)
 ../vendor/llama.cpp/     materialized by the build scripts from vendor.llama.pin (gitignored; never patched in place)
 ../patches/llama.cpp/    common(6) + android(14) hook-point bands + ledger README
 ../windows/              desktop companion — hosts the MLX→GGUF converter used in §1.4

android/app/build.gradle.kts

@@ -32,6 +32,11 @@ android {
         testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
         buildConfigField("String", "GIT_SHA",
             "\"${System.getenv("E0_GIT_SHA") ?: "unknown"}\"")
+        // engine provenance for the benchmark submission block (repo-root vendor.llama.pin)
+        val enginePin = providers.fileContents(
+            rootProject.layout.projectDirectory.dir("..").file("vendor.llama.pin")
+        ).asText.map { it.trim().substringBefore(' ') }.orElse("unknown")
+        buildConfigField("String", "ENGINE_COMMIT", "\"${enginePin.get()}\"")
     }
     buildTypes {
         debug { isMinifyEnabled = false }

android/app/src/androidTest/java/dev/edge0/runtime/app/BenchTest.kt

@@ -0,0 +1,88 @@
+// BenchTest - on-device benchmark that prints the community submission block.
+// Skipped unless `-e tier 8b|35b` is given, so other instrumented runs are unaffected.
+// Usage and model staging: android/README.md, "Benchmark".
+package dev.edge0.runtime.app
+
+import android.content.Context
+import android.os.Bundle
+import android.util.Log
+import androidx.test.platform.app.InstrumentationRegistry
+import dev.edge0.runtime.app.bench.BenchArgs
+import dev.edge0.runtime.app.bench.BenchConfig
+import dev.edge0.runtime.app.bench.BenchRunner
+import dev.edge0.runtime.app.bench.buildPrompt
+import dev.edge0.runtime.app.bench.describeRuns
+import dev.edge0.runtime.app.bench.findModel
+import dev.edge0.runtime.app.bench.formatSubmissionBlock
+import dev.edge0.runtime.app.bench.readDeviceInfo
+import dev.edge0.runtime.app.bench.resetPeakMemory
+import dev.edge0.runtime.app.runtime.LlamaRuntime
+import java.io.File
+import kotlinx.coroutines.runBlocking
+import org.junit.Assert.assertNotNull
+import org.junit.Assert.assertTrue
+import org.junit.Assume.assumeTrue
+import org.junit.Test
+
+class BenchTest {
+
+    @Test
+    fun submissionBlock() = runBlocking {
+        val instrumentation = InstrumentationRegistry.getInstrumentation()
+        val raw = InstrumentationRegistry.getArguments()
+        val rawArgs = raw.keySet().associateWith { raw.getString(it) }
+        assumeTrue("pass -e ${BenchArgs.KEY_TIER} 8b|35b to run the benchmark",
+            BenchArgs.KEY_TIER in rawArgs)
+        val args = BenchArgs.parse(rawArgs)
+        val context = instrumentation.targetContext
+
+        val model = findModel(modelRoots(context), args.tier)
+        assertNotNull("${args.tier.modelFile} not found under files/models (see README, Benchmark)", model)
+
+        val runtime = LlamaRuntime()
+        runtime.ensureLoaded(model!!.absolutePath)
+        assertTrue("load failed: ${runtime.state.value}", runtime.activeModelDir != null)
+
+        val summary = try {
+            BenchRunner(runtime, BenchConfig(temperature = args.temperature), resetPeak = ::resetPeakMemory)
+                .run(buildPrompt())
+        } finally {
+            runtime.unloadActive()
+        }
+
+        val shape = summary.reps.first()
+        val block = formatSubmissionBlock(
+            tier = args.tier,
+            device = readDeviceInfo(context),
+            summary = summary,
+            engineCommit = BuildConfig.ENGINE_COMMIT,
+            reporter = args.reporter,
+            notes = describeRuns(summary) + listOf(
+                "${shape.promptTokens}-token prompt, ${shape.newTokens} decoded, mean of ${summary.reps.size} reps, " +
+                    "temperature ${args.temperature}",
+                args.notes,
+            ),
+        )
+        publish(context.getExternalFilesDir(null) ?: context.filesDir, args.tier.id, block)
+    }
+
+    private fun modelRoots(context: Context): List<File?> = listOf(
+        File(context.filesDir, "models"),
+        context.getDir("models", 0),
+        File(context.cacheDir, "models"),
+        context.getExternalFilesDir(null)?.let { File(it, "models") },
+    )
+
+    /** Prints through `am instrument` and logcat, then saves a pullable copy; the copy is best effort. */
+    private fun publish(outputDir: File, tierId: String, block: String) {
+        Log.i(TAG, "\n$block")
+        val instrumentation = InstrumentationRegistry.getInstrumentation()
+        instrumentation.sendStatus(0, Bundle().apply { putString("submission_block", "\n$block") })
+        // "stream" is the standard key for a test's console output, which Gradle's connected-test report shows
+        instrumentation.sendStatus(0, Bundle().apply { putString("stream", "\n$block\n") })
+        runCatching { File(outputDir, "bench-$tierId.txt").writeText(block + "\n") }
+            .onFailure { Log.w(TAG, "could not save the block to $outputDir", it) }
+    }
+
+    private companion object { const val TAG = "E0Bench" }
+}