Skip to content

Commit dbf02a4

Browse files
committed
[DOC] Add Kotlin examples for the Android LLM runner
Mirrors the Java snippets in run-on-android.md with idiomatic Kotlin equivalents (object expression for the LlmCallback, IntArray/FloatArray for primitive arrays, Float.SIZE_BYTES, ByteBuffer.apply { ... }, trailing commas on multi-line calls). Follows the run-on-ios.md convention of putting a language label above each fenced block rather than tab-sets. Requested in review on the original PR (#19611).
1 parent 3aaad7d commit dbf02a4

1 file changed

Lines changed: 149 additions & 1 deletion

File tree

docs/source/llm/run-on-android.md

Lines changed: 149 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,17 +10,26 @@ To add the `executorch-android` library to your app, see [Using ExecuTorch on An
1010

1111
## Runtime API
1212

13-
Once the `executorch-android` AAR is on your classpath, you can import the LLM runner classes from the `org.pytorch.executorch.extension.llm` package.
13+
Once the `executorch-android` AAR is on your classpath, you can import the LLM runner classes from the `org.pytorch.executorch.extension.llm` package. The runner is callable from both Java and Kotlin; the rest of this guide shows both side by side.
1414

1515
### Importing
1616

17+
Java:
1718
```java
1819
import org.pytorch.executorch.extension.llm.LlmModule;
1920
import org.pytorch.executorch.extension.llm.LlmModuleConfig;
2021
import org.pytorch.executorch.extension.llm.LlmGenerationConfig;
2122
import org.pytorch.executorch.extension.llm.LlmCallback;
2223
```
2324

25+
Kotlin:
26+
```kotlin
27+
import org.pytorch.executorch.extension.llm.LlmModule
28+
import org.pytorch.executorch.extension.llm.LlmModuleConfig
29+
import org.pytorch.executorch.extension.llm.LlmGenerationConfig
30+
import org.pytorch.executorch.extension.llm.LlmCallback
31+
```
32+
2433
### LlmModule
2534

2635
The `LlmModule` class provides a simple Java interface for loading a text-generation model, configuring its tokenizer, generating token streams, and stopping execution. It also supports multimodal models that accept image and audio inputs alongside a text prompt.
@@ -31,15 +40,26 @@ This API is experimental and subject to change.
3140

3241
Create an `LlmModule` by specifying paths to your serialized model (`.pte`) and tokenizer files. For text-only models, the simple constructor is enough:
3342

43+
Java:
3444
```java
3545
LlmModule module = new LlmModule(
3646
"/data/local/tmp/llama-3.2-instruct.pte",
3747
"/data/local/tmp/tokenizer.model",
3848
0.8f);
3949
```
4050

51+
Kotlin:
52+
```kotlin
53+
val module = LlmModule(
54+
"/data/local/tmp/llama-3.2-instruct.pte",
55+
"/data/local/tmp/tokenizer.model",
56+
0.8f,
57+
)
58+
```
59+
4160
For finer control (multimodal model type, BOS/EOS handling, supplementary data files, load mode), use `LlmModuleConfig` with the fluent builder:
4261

62+
Java:
4363
```java
4464
LlmModuleConfig config = LlmModuleConfig.create()
4565
.modulePath("/data/local/tmp/llama-3.2-instruct.pte")
@@ -52,6 +72,19 @@ LlmModuleConfig config = LlmModuleConfig.create()
5272
LlmModule module = new LlmModule(config);
5373
```
5474

75+
Kotlin:
76+
```kotlin
77+
val config = LlmModuleConfig.create()
78+
.modulePath("/data/local/tmp/llama-3.2-instruct.pte")
79+
.tokenizerPath("/data/local/tmp/tokenizer.model")
80+
.temperature(0.8f)
81+
.modelType(LlmModuleConfig.MODEL_TYPE_TEXT)
82+
.loadMode(LlmModuleConfig.LOAD_MODE_MMAP)
83+
.build()
84+
85+
val module = LlmModule(config)
86+
```
87+
5588
Available load modes are `LOAD_MODE_FILE`, `LOAD_MODE_MMAP` (default), `LOAD_MODE_MMAP_USE_MLOCK`, and `LOAD_MODE_MMAP_USE_MLOCK_IGNORE_ERRORS`. Available model types are `MODEL_TYPE_TEXT` and `MODEL_TYPE_TEXT_VISION` (the `MODEL_TYPE_MULTIMODAL` constant is currently an alias for `MODEL_TYPE_TEXT_VISION` and selects the same runtime path).
5689

5790
Construction itself is lightweight and does not load the program data immediately.
@@ -60,19 +93,29 @@ Construction itself is lightweight and does not load the program data immediatel
6093

6194
Explicitly load the model before generation to avoid paying the load cost during your first `generate` call.
6295

96+
Java:
6397
```java
6498
int status = module.load();
6599
if (status != 0) {
66100
// Handle load failure (status is an ExecuTorch runtime error code).
67101
}
68102
```
69103

104+
Kotlin:
105+
```kotlin
106+
val status = module.load()
107+
if (status != 0) {
108+
// Handle load failure (status is an ExecuTorch runtime error code).
109+
}
110+
```
111+
70112
If you skip this step, the model is loaded lazily on the first `generate` call.
71113

72114
#### Generating
73115

74116
Generate tokens from a text prompt by passing an `LlmCallback` that receives each token as it is produced. The same callback also receives a JSON-encoded statistics string when generation completes.
75117

118+
Java:
76119
```java
77120
LlmCallback callback = new LlmCallback() {
78121
@Override
@@ -97,8 +140,31 @@ LlmCallback callback = new LlmCallback() {
97140
module.generate("Once upon a time", callback);
98141
```
99142

143+
Kotlin:
144+
```kotlin
145+
val callback = object : LlmCallback {
146+
override fun onResult(token: String) {
147+
// Called once per generated token. Append to your UI buffer here.
148+
print(token)
149+
}
150+
151+
override fun onStats(statsJson: String) {
152+
// Called once when generation finishes. See extension/llm/runner/stats.h
153+
// for the field definitions.
154+
println("\n$statsJson")
155+
}
156+
157+
override fun onError(errorCode: Int, message: String) {
158+
// Called if the runtime reports an error during generation.
159+
}
160+
}
161+
162+
module.generate("Once upon a time", callback)
163+
```
164+
100165
For full control over generation parameters, use `LlmGenerationConfig`:
101166

167+
Java:
102168
```java
103169
LlmGenerationConfig genConfig = LlmGenerationConfig.create()
104170
.seqLen(2048)
@@ -109,26 +175,49 @@ LlmGenerationConfig genConfig = LlmGenerationConfig.create()
109175
module.generate("Once upon a time", genConfig, callback);
110176
```
111177

178+
Kotlin:
179+
```kotlin
180+
val genConfig = LlmGenerationConfig.create()
181+
.seqLen(2048)
182+
.temperature(0.8f)
183+
.echo(false)
184+
.build()
185+
186+
module.generate("Once upon a time", genConfig, callback)
187+
```
188+
112189
`LlmGenerationConfig` exposes `echo`, `maxNewTokens`, `seqLen`, `temperature`, `numBos`, `numEos`, and `warming`. Defaults match the C++ `GenerationConfig` documented in [Running LLMs with C++](run-with-c-plus-plus.md).
113190

114191
#### Stopping Generation
115192

116193
If you need to interrupt a long-running generation, call `stop()` from another thread (or from inside the `onResult` callback):
117194

195+
Java:
118196
```java
119197
module.stop();
120198
```
121199

200+
Kotlin:
201+
```kotlin
202+
module.stop()
203+
```
204+
122205
Generation also runs synchronously on the calling thread, so make sure you invoke `generate()` off the main thread (for example, on a `HandlerThread` or via a `java.util.concurrent.Executor`).
123206

124207
#### Resetting
125208

126209
To clear the prefilled tokens from the KV cache and reset the start position to 0, call:
127210

211+
Java:
128212
```java
129213
module.resetContext();
130214
```
131215

216+
Kotlin:
217+
```kotlin
218+
module.resetContext()
219+
```
220+
132221
This is the equivalent of `reset()` on the iOS runner and `reset()` on the C++ `IRunner`.
133222

134223
### Multimodal Inputs
@@ -139,6 +228,7 @@ For models declared as `MODEL_TYPE_TEXT_VISION` or `MODEL_TYPE_MULTIMODAL`, imag
139228

140229
Raw uint8 pixel data in CHW order can be supplied as an `int[]`, or as a direct `ByteBuffer` to avoid JNI array copies:
141230

231+
Java:
142232
```java
143233
// As int[]
144234
int[] pixels = ...; // length == channels * height * width
@@ -150,8 +240,23 @@ buffer.put(rawBytes).rewind();
150240
module.prefillImages(buffer, 336, 336, 3);
151241
```
152242

243+
Kotlin:
244+
```kotlin
245+
// As IntArray
246+
val pixels: IntArray = ... // length == channels * height * width
247+
module.prefillImages(pixels, /* width = */ 336, /* height = */ 336, /* channels = */ 3)
248+
249+
// As direct ByteBuffer (preferred for large images)
250+
val buffer = ByteBuffer.allocateDirect(3 * 336 * 336).apply {
251+
put(rawBytes)
252+
rewind()
253+
}
254+
module.prefillImages(buffer, 336, 336, 3)
255+
```
256+
153257
Pre-normalized float pixel data is also supported, both as a `float[]` and as a direct `ByteBuffer` in native byte order:
154258

259+
Java:
155260
```java
156261
float[] normalized = ...; // length == channels * height * width
157262
module.prefillImages(normalized, 336, 336, 3);
@@ -163,31 +268,63 @@ ByteBuffer floatBuffer = ByteBuffer
163268
module.prefillNormalizedImage(floatBuffer, 336, 336, 3);
164269
```
165270

271+
Kotlin:
272+
```kotlin
273+
val normalized: FloatArray = ... // length == channels * height * width
274+
module.prefillImages(normalized, 336, 336, 3)
275+
276+
val floatBuffer: ByteBuffer = ByteBuffer
277+
.allocateDirect(3 * 336 * 336 * Float.SIZE_BYTES)
278+
.order(ByteOrder.nativeOrder())
279+
// fill floatBuffer with normalized values, then:
280+
module.prefillNormalizedImage(floatBuffer, 336, 336, 3)
281+
```
282+
166283
#### Audio
167284

168285
Preprocessed audio features (for example mel spectrograms produced by a Whisper preprocessor) can be supplied as `byte[]` or `float[]`:
169286

287+
Java:
170288
```java
171289
module.prefillAudio(features, /*batchSize=*/1, /*nBins=*/128, /*nFrames=*/3000);
172290
```
173291

292+
Kotlin:
293+
```kotlin
294+
module.prefillAudio(features, /* batchSize = */ 1, /* nBins = */ 128, /* nFrames = */ 3000)
295+
```
296+
174297
Raw audio samples can be supplied with `prefillRawAudio`:
175298

299+
Java:
176300
```java
177301
module.prefillRawAudio(samples, /*batchSize=*/1, /*nChannels=*/1, /*nSamples=*/16000);
178302
```
179303

304+
Kotlin:
305+
```kotlin
306+
module.prefillRawAudio(samples, /* batchSize = */ 1, /* nChannels = */ 1, /* nSamples = */ 16000)
307+
```
308+
180309
#### Generating with Multimodal Prefill
181310

182311
After prefilling each modality, run `generate()` with the text prompt as usual:
183312

313+
Java:
184314
```java
185315
module.prefillImages(pixels, 336, 336, 3);
186316
module.generate("What's in this image?", callback);
187317
```
188318

319+
Kotlin:
320+
```kotlin
321+
module.prefillImages(pixels, 336, 336, 3)
322+
module.generate("What's in this image?", callback)
323+
```
324+
189325
For text-vision models, a convenience overload accepts the image and prompt together:
190326

327+
Java:
191328
```java
192329
module.generate(
193330
pixels, /*width=*/336, /*height=*/336, /*channels=*/3,
@@ -197,6 +334,17 @@ module.generate(
197334
/*echo=*/false);
198335
```
199336

337+
Kotlin:
338+
```kotlin
339+
module.generate(
340+
pixels, /* width = */ 336, /* height = */ 336, /* channels = */ 3,
341+
"What's in this image?",
342+
/* seqLen = */ 768,
343+
callback,
344+
/* echo = */ false,
345+
)
346+
```
347+
200348
## Demo
201349

202350
See the [Llama Android demo app](https://github.com/meta-pytorch/executorch-examples/tree/main/llm/android/LlamaDemo) in `executorch-examples` for an end-to-end project that wires `LlmModule`, `LlmCallback`, and a `HandlerThread` into a chat UI.

0 commit comments

Comments
 (0)