You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[DOC] Add Kotlin examples for the Android LLM runner
Mirrors the Java snippets in run-on-android.md with idiomatic Kotlin
equivalents (object expression for the LlmCallback, IntArray/FloatArray
for primitive arrays, Float.SIZE_BYTES, ByteBuffer.apply { ... },
trailing commas on multi-line calls). Follows the run-on-ios.md
convention of putting a language label above each fenced block rather
than tab-sets.
Requested in review on the original PR (#19611).
Copy file name to clipboardExpand all lines: docs/source/llm/run-on-android.md
+149-1Lines changed: 149 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,17 +10,26 @@ To add the `executorch-android` library to your app, see [Using ExecuTorch on An
10
10
11
11
## Runtime API
12
12
13
-
Once the `executorch-android` AAR is on your classpath, you can import the LLM runner classes from the `org.pytorch.executorch.extension.llm` package.
13
+
Once the `executorch-android` AAR is on your classpath, you can import the LLM runner classes from the `org.pytorch.executorch.extension.llm` package. The runner is callable from both Java and Kotlin; the rest of this guide shows both side by side.
The `LlmModule` class provides a simple Java interface for loading a text-generation model, configuring its tokenizer, generating token streams, and stopping execution. It also supports multimodal models that accept image and audio inputs alongside a text prompt.
@@ -31,15 +40,26 @@ This API is experimental and subject to change.
31
40
32
41
Create an `LlmModule` by specifying paths to your serialized model (`.pte`) and tokenizer files. For text-only models, the simple constructor is enough:
33
42
43
+
Java:
34
44
```java
35
45
LlmModule module =newLlmModule(
36
46
"/data/local/tmp/llama-3.2-instruct.pte",
37
47
"/data/local/tmp/tokenizer.model",
38
48
0.8f);
39
49
```
40
50
51
+
Kotlin:
52
+
```kotlin
53
+
val module =LlmModule(
54
+
"/data/local/tmp/llama-3.2-instruct.pte",
55
+
"/data/local/tmp/tokenizer.model",
56
+
0.8f,
57
+
)
58
+
```
59
+
41
60
For finer control (multimodal model type, BOS/EOS handling, supplementary data files, load mode), use `LlmModuleConfig` with the fluent builder:
Available load modes are `LOAD_MODE_FILE`, `LOAD_MODE_MMAP` (default), `LOAD_MODE_MMAP_USE_MLOCK`, and `LOAD_MODE_MMAP_USE_MLOCK_IGNORE_ERRORS`. Available model types are `MODEL_TYPE_TEXT` and `MODEL_TYPE_TEXT_VISION` (the `MODEL_TYPE_MULTIMODAL` constant is currently an alias for `MODEL_TYPE_TEXT_VISION` and selects the same runtime path).
56
89
57
90
Construction itself is lightweight and does not load the program data immediately.
@@ -60,19 +93,29 @@ Construction itself is lightweight and does not load the program data immediatel
60
93
61
94
Explicitly load the model before generation to avoid paying the load cost during your first `generate` call.
62
95
96
+
Java:
63
97
```java
64
98
int status = module.load();
65
99
if (status !=0) {
66
100
// Handle load failure (status is an ExecuTorch runtime error code).
67
101
}
68
102
```
69
103
104
+
Kotlin:
105
+
```kotlin
106
+
val status = module.load()
107
+
if (status !=0) {
108
+
// Handle load failure (status is an ExecuTorch runtime error code).
109
+
}
110
+
```
111
+
70
112
If you skip this step, the model is loaded lazily on the first `generate` call.
71
113
72
114
#### Generating
73
115
74
116
Generate tokens from a text prompt by passing an `LlmCallback` that receives each token as it is produced. The same callback also receives a JSON-encoded statistics string when generation completes.
75
117
118
+
Java:
76
119
```java
77
120
LlmCallback callback =newLlmCallback() {
78
121
@Override
@@ -97,8 +140,31 @@ LlmCallback callback = new LlmCallback() {
97
140
module.generate("Once upon a time", callback);
98
141
```
99
142
143
+
Kotlin:
144
+
```kotlin
145
+
val callback =object:LlmCallback {
146
+
overridefunonResult(token:String) {
147
+
// Called once per generated token. Append to your UI buffer here.
148
+
print(token)
149
+
}
150
+
151
+
overridefunonStats(statsJson:String) {
152
+
// Called once when generation finishes. See extension/llm/runner/stats.h
module.generate("Once upon a time", genConfig, callback);
110
176
```
111
177
178
+
Kotlin:
179
+
```kotlin
180
+
val genConfig =LlmGenerationConfig.create()
181
+
.seqLen(2048)
182
+
.temperature(0.8f)
183
+
.echo(false)
184
+
.build()
185
+
186
+
module.generate("Once upon a time", genConfig, callback)
187
+
```
188
+
112
189
`LlmGenerationConfig` exposes `echo`, `maxNewTokens`, `seqLen`, `temperature`, `numBos`, `numEos`, and `warming`. Defaults match the C++ `GenerationConfig` documented in [Running LLMs with C++](run-with-c-plus-plus.md).
113
190
114
191
#### Stopping Generation
115
192
116
193
If you need to interrupt a long-running generation, call `stop()` from another thread (or from inside the `onResult` callback):
117
194
195
+
Java:
118
196
```java
119
197
module.stop();
120
198
```
121
199
200
+
Kotlin:
201
+
```kotlin
202
+
module.stop()
203
+
```
204
+
122
205
Generation also runs synchronously on the calling thread, so make sure you invoke `generate()` off the main thread (for example, on a `HandlerThread` or via a `java.util.concurrent.Executor`).
123
206
124
207
#### Resetting
125
208
126
209
To clear the prefilled tokens from the KV cache and reset the start position to 0, call:
127
210
211
+
Java:
128
212
```java
129
213
module.resetContext();
130
214
```
131
215
216
+
Kotlin:
217
+
```kotlin
218
+
module.resetContext()
219
+
```
220
+
132
221
This is the equivalent of `reset()` on the iOS runner and `reset()` on the C++ `IRunner`.
133
222
134
223
### Multimodal Inputs
@@ -139,6 +228,7 @@ For models declared as `MODEL_TYPE_TEXT_VISION` or `MODEL_TYPE_MULTIMODAL`, imag
139
228
140
229
Raw uint8 pixel data in CHW order can be supplied as an `int[]`, or as a direct `ByteBuffer` to avoid JNI array copies:
See the [Llama Android demo app](https://github.com/meta-pytorch/executorch-examples/tree/main/llm/android/LlamaDemo) in `executorch-examples` for an end-to-end project that wires `LlmModule`, `LlmCallback`, and a `HandlerThread` into a chat UI.
0 commit comments