diff --git a/.claude/skills/forge-error-catalogue/SKILL.md b/.claude/skills/forge-error-catalogue/SKILL.md index e8e19c71..9e519dda 100644 --- a/.claude/skills/forge-error-catalogue/SKILL.md +++ b/.claude/skills/forge-error-catalogue/SKILL.md @@ -93,7 +93,9 @@ instead of re-deriving it. the NDK toolchain pins `CMAKE_SYSTEM_VERSION` to 1; read `ANDROID_PLATFORM_LEVEL`**. - **Runtime failures** (device/emulator/simulator) — **the Flet 0.86 Android `sitepackages.zip` class** (its umbrella entry explains "why only now"): - `NotADirectoryError` on a bundled data file → **`extract_packages`** meta field; + `NotADirectoryError` on a bundled data file → **`extract_packages`** meta field, + or, when the package's own import reads it (xgboost `VERSION`), **an + `importlib.resources` patch** — a meta-only fix passes CI and breaks consumers; untagged native `.so` `ModuleNotFoundError`/"circular import" → **forge `fix_wheel` ABI-tags `PyInit_*` `.so`**; pycryptodome `Cannot load native module …` → **`importlib.find_spec().origin`**; llama `FileNotFoundError` / ctypes diff --git a/.claude/skills/forge-error-catalogue/references/failure-catalogue.md b/.claude/skills/forge-error-catalogue/references/failure-catalogue.md index 928ef3cd..edf91eeb 100644 --- a/.claude/skills/forge-error-catalogue/references/failure-catalogue.md +++ b/.claude/skills/forge-error-catalogue/references/failure-catalogue.md @@ -1399,6 +1399,18 @@ because `copyOpt` copies only `**/*.so`); the fix there is to ship the data file **inside the python package's own wheel** + load it from memory — see the dedicated python-magic entry below. +**When the package itself reads the file on every import, patch the read instead.** +`extract_packages` in a recipe's `meta.yaml` reaches only mobile-forge's test app, so +the CI run goes green while every consumer app still crashes until its author adds the +same entry to their own `pyproject.toml`. If the read is one small file in the package's +own code, swap the `open(os.path.join(os.path.dirname(__file__), …))` for +`importlib.resources.files(__package__).joinpath(…).read_text()`, which zipimport +serves. xgboost 3.4 (`_c_api._py_version()` opens `xgboost/VERSION`, called from +`__init__` and again from `_load_lib`) died on `import xgboost` with +`…/sitepackages.zip/xgboost/VERSION`; `mobile.patch` hunk 4 fixes it with no consumer +configuration. The 3.3.0 recipe predated 0.86 and never met it. Reproduce on a desktop +in seconds: put the package in a zip on `sys.path` and import it. + --- ### `RuntimeError: Failed to load shared library '...//lib/lib.so'` / `dlopen failed: library "..." not found` (Android, at import — a **ctypes** loader that raises before its own jniLibs fallback) @@ -1439,7 +1451,12 @@ Android** build, while the *same recipe passes on py3.12*. llama-cpp-python hit Unsupported platform` at import on 3.13, before its jniLibs bare-soname fallback could run. **Fix:** add `or sys.platform == "android"` wherever `"linux"` is special-cased (Android is a -Linux kernel, so the linux path is almost always what you want), and bump the build. This is a +Linux kernel, so the linux path is almost always what you want), and bump the build. +**iOS is different: flet's runtime reports `"ios"` on every version, 3.12 included** +(`Py_GetPlatform()` in its 3.12.13 `Python.framework` returns the literal `"ios"`, and its +`_sysconfigdata__ios_*` sets `MACHDEP` to `ios`). So an `ios` branch is enough, and the +`darwin` branch can stay byte-identical to upstream (xgboost). A comment in +llama-cpp-python's `mobile.patch` says 3.12 iOS reports `"darwin"`; it does not. This is a **general py3.12→3.13 tell** — grep a recipe's patches/loaders for `sys.platform` before trusting a 3.13 build. Distinct from the 3.13/3.14 Android **x86_64 `SIGSYS`/seccomp `open()`** crash, which is a *native* abort from python-build's mimalloc (fixed in the `20260712` snapshot), not a @@ -1693,8 +1710,12 @@ misses. The bundled libs (llama-cpp-python's libllama/libggml*) were relocated t **Fix:** after the on-disk probes miss, fall back to `ctypes.CDLL("lib.so")` (bare soname → the Android linker resolves it from jniLibs), for both the dependency preload and the main lib; load `RTLD_GLOBAL` so preloaded deps satisfy the main -lib's DT_NEEDED. Verified: llama-cpp-python (android). See also "Unable to find -… shared library" (the `find_library`-returns-None sibling). +lib's DT_NEEDED. Verified: llama-cpp-python (android). xgboost's `libpath.py` has +the same `os.path.exists` gate (its error would be `XGBoostLibraryNotFound: Cannot +find XGBoost Library in the candidate path`); its patch returns `["libxgboost.so"]` +when nothing exists and `sys.getandroidapilevel` does — present on 3.12, where +`sys.platform` still says `"linux"`. Verified on an arm64 emulator, Flet 1.0.3. +See also "Unable to find … shared library" (the `find_library`-returns-None sibling). --- diff --git a/.claude/skills/local-recipe-testing/SKILL.md b/.claude/skills/local-recipe-testing/SKILL.md index dba10d76..0adf49e6 100644 --- a/.claude/skills/local-recipe-testing/SKILL.md +++ b/.claude/skills/local-recipe-testing/SKILL.md @@ -118,7 +118,7 @@ carries every control value: `xcrun simctl spawn "$UDID" log show --last 2m --st 1. **Use forge's `dist/` wheel, NOT `build/.../target/wheels/`.** The latter is maturin's raw output — **unstripped**. For polars that meant a **1.27 GB** `.so` (vs 130 MB stripped); it blows up install space and may not load. forge strips + repacks into `dist/`. Always test the `dist/` wheel. -2. **Build the recipe against the SAME Python `flet build` bundles — and know which one that is.** `flet build` **0.86.5 defaults to Python 3.14** (measured 2026-09-14: the app's staged site-packages hold `_cffi_backend.cpython-314-*.so`); the snippets above pass `--python-version 3.12` to pin it, and CI's recipe-tester does the same. A **consumer** example app built the plain way (`flet build apk`, no flag) therefore gets 3.14, so build the recipe for 3.14 (`source ./setup.sh 3.14`) before that pass, or pass the flag. `flet-lib*` build.sh recipes are `py3-none-` and version-independent — only the Python package needs the extra build. +2. **Build the recipe against the SAME Python `flet build` bundles — and know which one that is.** `flet build` **0.86.5 defaults to Python 3.14** (measured 2026-09-14: the app's staged site-packages hold `_cffi_backend.cpython-314-*.so`), and so does 1.0.3 (3.14.7; a forge build against the 3.14.6 support tree loads fine); the snippets above pass `--python-version 3.12` to pin it, and CI's recipe-tester does the same. A **consumer** example app built the plain way (`flet build apk`, no flag) therefore gets 3.14, so build the recipe for 3.14 (`source ./setup.sh 3.14`) before that pass, or pass the flag. `flet-lib*` build.sh recipes are `py3-none-` and version-independent — only the Python package needs the extra build. Two different failure shapes if you get it wrong. For a **compiled** package: forge's Android Rust `.so` hard-links `libpythonX.Y.so` (`DT_NEEDED`) — so the **`abi3` wheel tag is misleading**; it still needs the matching `libpython` at `dlopen`. A 3.14-built wheel in a 3.12 app fails: `dlopen … libpython3.14.so` missing → the package reports its "binary missing" (e.g. polars `NameError: PySeries`). Verify with `llvm-readelf -d | grep NEEDED`. If you only have a different support tree, you can retag a wheel for flet's python with `uvx --from wheel wheel tags --python-tag cp312 --abi-tag abi3 --remove `, but the underlying `libpython` link still has to match — so really, build on the right python. For a **pure-Python** package with a mobile patch there is no link error at all — pip simply finds no `cp3XX` match, silently installs PyPI's `py3-none-any` wheel, and the app fails on device with the unpatched loader's own error. Tell: the app's `build/site-packages//-*.dist-info/METADATA` is missing the `flet-lib*` `Requires-Dist` the recipe promotes. @@ -174,11 +174,14 @@ carries every control value: `xcrun simctl spawn "$UDID" log show --last 2m --st # config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'] = '15.0' sed -i '' 's/IPHONEOS_DEPLOYMENT_TARGET = 13.0;/IPHONEOS_DEPLOYMENT_TARGET = 15.0;/' Runner.xcodeproj/project.pbxproj cd .. && SERIOUS_PYTHON_APP=$PWD/../python-app SERIOUS_PYTHON_SITE_PACKAGES=$PWD/../site-packages \ - SERIOUS_PYTHON_VERSION=3.14 SP_NATIVE_SET=$(cat ../.serious_python_spm_key) \ + SERIOUS_PYTHON_VERSION=3.14 SERIOUS_PYTHON_BUNDLE_ID=com.flet. \ + SP_NATIVE_SET=$(cat ../.serious_python_spm_key) \ flutter build ios --simulator --build-name 1.0.0 # VERSION = the Python flet built for # → build/flutter/build/ios/iphonesimulator/.app ``` A fresh `flet build` regenerates the template, so repeat the patch after every one. + The env is flet-cli's `_flutter_path_env` (`build_base.py`); without + `SERIOUS_PYTHON_BUNDLE_ID` the embedded frameworks get `org.python.*` identifiers. 11. **Two booted simulators make `simctl booted` ambiguous.** With more than one sim booted, `simctl install booted …` targets one device and your subsequent `get_app_container booted …` may query the OTHER — the app "isn't installed" / the container is empty despite a successful install. Use the explicit `$UDID` for every simctl call (as the loop above does); never rely on `booted` unless you've verified exactly one device is booted (`xcrun simctl list devices | grep -c Booted`). diff --git a/.claude/skills/new-mobile-recipe/SKILL.md b/.claude/skills/new-mobile-recipe/SKILL.md index 115497cf..01eb80c2 100644 --- a/.claude/skills/new-mobile-recipe/SKILL.md +++ b/.claude/skills/new-mobile-recipe/SKILL.md @@ -74,6 +74,7 @@ Match the package to one of these shapes. Each maps to a template in `templates/ | C-ext needing patches | Upstream `setup.py` reaches for host `/usr/include`, has `if sys.platform` branches, or hardcodes paths | `templates/meta-with-patches.yaml` + patch file in `patches/` | | meson-python build backend | `[build-system].build-backend = "mesonpy"` (scipy, scikit-image, scikit-learn, pywavelets, contourpy) | No template — copy from `recipes/pywavelets/` (on main): `build.backend-args:` list with `-Csetup-args=--cross-file` + `-Csetup-args={MESON_CROSS_FILE}`; extra meson options ride the same way (`-Csetup-args=-D=`) | | scikit-build-core / CMake with sdist | `[build-system].build-backend = "scikit_build_core.build"` (duckdb, onnx, ml-dtypes) | No template — copy from `recipes/duckdb/` (on main) or `recipes/onnx/` (branch `machine/onnx-insightface`); CMake args via `script_env` `CMAKE_ARGS` (skbuild appends it after its own defaults) | +| Package that **ctypes-loads its own C API library** from `/lib/` (no extension module) | xgboost, lightgbm: CMake builds `lib.so`, the Python side finds it by a `__file__` path | Copy `recipes/xgboost/` (scikit-build-core lanes + one `mobile.patch`): serious_python framework-izes the lib (`.so` or `.dylib`) behind a `.fwork` pointer on iOS, so teach the loader `ios`/`android` `sys.platform` (both iOS 3.12+ and Android 3.13+; Android 3.12 says `linux`) + that pointer + an Android bare-soname fallback, drop `SOVERSION`, and grep for every other `__file__` read — Android site-packages is a zip | | C-ext consuming an existing flet-lib | Already-built `flet-libX` covers the C dep (libxml2, libcurl, libssl via openssl, etc.) | Adapt `templates/meta-with-patches.yaml` + add `requirements.host` | | Native library itself (flet-lib*), **static** | New C library a Python C-extension links at build time (libxml2, libcurl, libgeos…) | `templates/meta-flet-lib.yaml` + `templates/build-flet-lib.sh` | | Native library, **ctypes-loaded (shared)** | A pure-Python wrapper `dlopen`s the lib at runtime via `ctypes` (pyzbar→libzbar, python-magic→libmagic) | `templates/meta-flet-lib.yaml` + `templates/build-flet-lib-shared.sh`; see Pattern H | diff --git a/recipes/xgboost/README.md b/recipes/xgboost/README.md new file mode 100644 index 00000000..fa0ceec7 --- /dev/null +++ b/recipes/xgboost/README.md @@ -0,0 +1,239 @@ +# xgboost + +[XGBoost](https://xgboost.readthedocs.io/en/stable/) is gradient-boosted decision trees — the +model family that still wins most problems shaped like a spreadsheet: rows of measurements, +one label to predict. On a phone its job is usually the second half of a workflow that starts +on a laptop: train there, save the model, and ship the file inside the app so predictions run +on the device, offline, in milliseconds. + +## Install + +```toml +dependencies = [ + "flet", + "xgboost", + "joblib", +] + +[tool.flet.android] +target_arch = ["arm64-v8a", "x86_64"] +``` + +`joblib` is only there to read `.joblib` files; leave it out if you load models some other +way. + +**The [`target_arch`](https://flet.dev/docs/publish/android/#supported-target-architectures) +line is required.** XGBoost assumes a 64-bit `size_t` and does not compile for 32-bit ARM, so +no `armeabi-v7a` wheel exists, and a default `flet build apk` targets that ABI too and fails +with `No matching distribution found for xgboost`. Spell the ABI names out in full: short +forms such as `arm64` or `x64` are not Android ABI names, and Flet rejects them. + +**If scikit-learn is in the app at all** — because your file holds a `Pipeline`, a scaler or +an encoder, or because another dependency pulls it in — Android also needs it +[extracted](https://flet.dev/docs/publish/android/#extract-packages): + +```toml +[tool.flet.android] +target_arch = ["arm64-v8a", "x86_64"] +extract_packages = ["sklearn"] +``` + +`import xgboost` imports scikit-learn whenever it is installed, and scikit-learn reads a file +beside its own code through a plain path, which Flet's zipped Android site-packages cannot +serve. Without the entry, `import xgboost` itself fails with a `NotADirectoryError` ending in +`sklearn/utils/_repr_html/estimator.css`. + +## Examples + +See runnable Flet apps in [`examples/`](examples): + +- [`iris-explainer`](examples/iris-explainer) — loads a desktop-trained `.joblib` + classifier, predicts from four sliders, and shows which measurements drove the answer. + +## Usage in a Flet app + +Put the model file in your app's `src/assets/`, load it from +[`FLET_ASSETS_DIR`](https://flet.dev/docs/reference/environment-variables/#flet_assets_dir) +on a worker thread — the first load also imports xgboost and scipy — and show the answer: + +```python +import os + +import flet as ft +import joblib + +result = ft.Text() + +def load_and_predict(): + path = os.path.join(os.getenv("FLET_ASSETS_DIR"), "model.joblib") + model = joblib.load(path) + proba = model.predict_proba([[5.8, 3.0, 4.4, 1.3]])[0] + result.value = f"class {proba.argmax()} · {proba.max():.0%}" + page.update() # a background thread needs this explicitly + +page.add(result) +page.run_thread(load_and_predict) +``` + +A bare [`XGBClassifier`](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.XGBClassifier) +or `XGBRegressor` unpickles and predicts without scikit-learn installed, even though +scikit-learn was needed to train it. Without it, use only `predict`, `predict_proba`, `apply` +and `get_booster()`: `classes_` raises `ModuleNotFoundError: No module named 'sklearn'`, +and `get_params()`/`set_params()`, `fit`, `score` and the wrapper's own `save_model` all +fail. Classes are always encoded `0..n-1`, so keep your own list of names in the app. + +[`pred_contribs=True`](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.Booster.predict) +is XGBoost's built-in TreeSHAP — one contribution per feature, plus a bias, which together +sum to the model's raw score. It is the way to explain a prediction on device: the `shap` +package depends on numba, a JIT compiler with no build for Flet's mobile targets. + +```python +import xgboost as xgb + +shap = model.get_booster().predict(xgb.DMatrix(row), pred_contribs=True) +``` + +For a multi-class model the result has one block per class, `shap[row, class, feature]`. + +### Storage + +Assets ship inside the app and are replaced by every update. A model the app downloads or +retrains belongs in +[`FLET_APP_STORAGE_DATA`](https://flet.dev/docs/reference/environment-variables/#flet_app_storage_data) +instead; save it there with `model.get_booster().save_model(path)`, which needs no +scikit-learn. + +### Threading + +**On Android XGBoost uses every core by default; iOS uses one.** The Android wheel is built +with OpenMP and links the NDK's runtime. iOS has no OpenMP runtime, so that build is +single-threaded — same results, different wall clock. A pickled model keeps the thread +count it was trained with, so a model trained with `n_jobs=1` stays on one core on the phone +too. Override it on the booster, which works with or without scikit-learn (`0` means every +core): + +```python +model.get_booster().set_param({"nthread": 2}) +``` + +Predicting a few rows with a small model takes milliseconds and can stay in an event handler. +Loading, training and large batches belong in +[`page.run_thread(...)`](https://flet.dev/docs/controls/page/#flet.Page.run_thread), ending +with an explicit [`page.update()`](https://flet.dev/docs/controls/page/#flet.Page.update). +XGBoost is called through `ctypes`, which releases the GIL for each native call, so the UI +keeps running meanwhile. + +### Model files + +A `.joblib` file is a pickle, and loading a pickle runs code. Load only files your own +build produced, never one fetched over the network. + +A pickle is also tied to the versions that wrote it. Files from older XGBoost releases +[still load](https://xgboost.readthedocs.io/en/stable/tutorials/saving_model.html#loading-pickled-files-or-rds-files) +— a 2.1 model predicts identically here, after a warning — but nothing promises the reverse. +scikit-learn objects are stricter: from another scikit-learn version they unpickle with an +`InconsistentVersionWarning` and can still fail when called (a 1.7 `SimpleImputer` raises +`AttributeError` inside `predict` on 1.9). So train with the versions the app gets — the +XGBoost you pin, and, if the file holds scikit-learn objects, the scikit-learn release +pypi.flet.dev carries, pinned on the desktop as the example's `train` group does. + +For a model that has to outlive version bumps, save it in XGBoost's +[own format](https://xgboost.readthedocs.io/en/stable/tutorials/saving_model.html#a-note-on-backward-compatibility-of-models-and-memory-snapshots) +instead and load it into a +[`Booster`](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.Booster), +which needs no scikit-learn: + +```python +model.save_model("model.ubj") # on the desktop + +booster = xgb.Booster(model_file=path) # in the app +scores = booster.predict(xgb.DMatrix(rows)) +``` + +For a multi-class model `scores` matches `predict_proba`; for a binary one it is a 1-D array +of the positive class's probability. `XGBClassifier().load_model(path)` reads the same file, +but constructing an `XGBClassifier` requires scikit-learn. + +### App size + +XGBoost itself is about 2.5 MB compressed and 8–9 MB unpacked per architecture; most of +what it brings is [scipy](https://scipy.org), which `import xgboost` loads unconditionally. +Budget roughly 40 MB compressed and 115–135 MB unpacked per architecture for xgboost, scipy +and numpy together, and 8 MB / 28–32 MB more for scikit-learn if your model needs it. An APK +carries close to the unpacked figure, because Flet stores Python packages and native +libraries uncompressed so it can memory-map them; an IPA, or a Play Store download built +from an app bundle, comes closer to the compressed one. + +About 24 MB of the unpacked total is scipy's and numpy's own test suites, which your app +never imports. Flet's cleanup does not remove test suites by default, so +[name them](https://flet.dev/docs/cli/flet-build/#--cleanup-package-files): + +```toml +[tool.flet.cleanup] +package_files = ["scipy/**/tests", "numpy/tests", "numpy/*/tests"] +``` + +On Android, also use an app bundle or split APKs. + +### Other considerations + +A desktop `flet run` uses PyPI's XGBoost wheel, which is multi-threaded on every desktop +platform. A timing measured at your desk transfers poorly to Android and not at all to iOS, +so measure on a device or emulator/simulator. + +## Things to know + +- **No GPU.** Both builds are CPU-only. A model saved with `device="cuda"` loads and + predicts on the CPU. + +## Build notes (maintainers) + +### Recipe shape + +scikit-build-core + CMake, the [`soxr`](../soxr) / [`duckdb`](../duckdb) lanes: XGBoost 3.4 +replaced the custom hatchling backend of earlier releases, so `CMAKE_ARGS` is honoured. +`libxgboost` is a plain shared library loaded through `ctypes`, not an extension module, and +the patch is about where each platform's packager puts it; its preamble has the detail. + +An earlier draft of this recipe targeted 3.3.0 and had to patch the hatchling backend in +three places — to accept CMake arguments at all, to stop it naming the library after the +build host, and to stop it silently retrying without OpenMP when the cross configure failed. +None of that survives in 3.4. + +### Upgrade hazards + +- **The library name is decided in two places that must agree**: `_XGB_LIBNAME` in the root + `CMakeLists.txt` and the candidate list in `xgboost/libpath.py`. +- **The sdist's `pyproject.toml` is generated per variant** (`pypi_variants.py`). Check its + `dependencies` on every bump: the variant published as the sdist has no `nvidia-nccl-cu*` + dependency (cu13 as of 3.4, cu12 before), but the wheel variants carry one. +- **Any new `__file__`-relative read breaks `import xgboost` on Android**, where Flet ships + site-packages as a zip. `grep -rn __file__ xgboost/` on the new sdist; today only + `libpath.py` and `_c_api._py_version()` matter, and both are patched. A desktop run will not + show it: put the package in a zip on `sys.path` and import it. + +### Re-verification checklist + +- Each wheel carries `xgboost/lib/libxgboost.so` and nothing versioned beside it. +- Android: `DT_NEEDED` limited to bionic, `libc++_shared.so` and `libomp.so`; `SONAME` + `libxgboost.so`; every `LOAD` segment aligned `0x4000`. +- iOS: `otool -L` shows no `libomp`; `LC_BUILD_VERSION` platform 2 on device, 7 on the + simulators. +- METADATA: `Requires-Dist` numpy and scipy; `flet-libcpp-shared` and `flet-libomp` on + Android only. +- Sizes: re-measure in decimal MB from the wheels (sum of `zipfile` entry sizes, not `du`), + including scipy, numpy and scikit-learn as pypi.flet.dev serves them; and confirm + `import xgboost` still pulls in scipy and still imports scikit-learn when it is present. +- In a venv without scikit-learn, re-check the list in Usage: what works on an unpickled + model and what fails, and how. +- `tests/iris_classifier.joblib` was written by the version in the test file's header + comment. It keeps working across bumps; regenerate it (the recipe is in that comment) + when the version gap starts producing warnings. + +### Coverage gaps + +The device tests cover native training and prediction, UBJSON round-trips, TreeSHAP, a +desktop-trained `.joblib` classifier with a booster thread cap, and the OpenMP build flag. +They do not cover the scikit-learn wrapper's own `fit`, a scikit-learn `Pipeline`, +categorical features, external memory (`DataIter`), or any multi-threaded speed-up — the +OpenMP test checks the build flag, not that work spreads across cores. diff --git a/recipes/xgboost/examples/iris-explainer/.gitignore b/recipes/xgboost/examples/iris-explainer/.gitignore new file mode 100644 index 00000000..429a8307 --- /dev/null +++ b/recipes/xgboost/examples/iris-explainer/.gitignore @@ -0,0 +1,7 @@ +.venv/ +.flet/ +build/ +__pycache__/ +.pytest_cache/ +.ruff_cache/ +uv.lock diff --git a/recipes/xgboost/examples/iris-explainer/README.md b/recipes/xgboost/examples/iris-explainer/README.md new file mode 100644 index 00000000..22da72cf --- /dev/null +++ b/recipes/xgboost/examples/iris-explainer/README.md @@ -0,0 +1,47 @@ +# xgboost iris explainer + +A gradient-boosted classifier trained on a desktop and shipped inside the app as a `.joblib` +file. Four sliders set a flower's sepal and petal measurements; the app names the species, +shows the probability of each, and breaks the winning score down by measurement. The header +line says which XGBoost build is running and whether it has OpenMP. + +What it demonstrates: + +- **The train-on-desktop, predict-on-device workflow.** [`train.py`](train.py) fits an + `XGBClassifier` on scikit-learn's iris data and writes + [`src/assets/iris_xgb.joblib`](src/assets/iris_xgb.joblib); the app reads it from + [`FLET_ASSETS_DIR`](https://flet.dev/docs/reference/environment-variables/#flet_assets_dir) + with [`joblib.load`](https://joblib.readthedocs.io/en/stable/generated/joblib.load.html). +- **No scikit-learn on the device.** It was needed to train the model, but the app's + dependencies are XGBoost and joblib alone — a bare `XGBClassifier` predicts without it. + The species names live in the app because the model's `classes_` would need scikit-learn. +- **Explaining a prediction.** + [`pred_contribs=True`](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.Booster.predict) + returns XGBoost's TreeSHAP values; the bars are the winning class's, green where a + measurement raised its score and red where it lowered it. +- **Load off the UI thread, predict on it.** Loading runs in + [`page.run_thread(...)`](https://flet.dev/docs/controls/page/#flet.Page.run_thread), which + needs the explicit [`page.update()`](https://flet.dev/docs/controls/page/#flet.Page.update) + it ends with. A prediction from a model this size takes milliseconds, so it runs directly + in the sliders' change handler; the footer shows the time. +- **Shedding scipy's test suites.** `import xgboost` loads scipy, and the + `[tool.flet.cleanup]` table in [`pyproject.toml`](pyproject.toml) keeps its tests, and + numpy's, out of the app. + +To retrain, run `uv run --group train python train.py` with the XGBoost version the app +pins. + +## Try it + +[Build](https://flet.dev/docs/publish/) the app, then install it on a device or emulator/simulator: + +```bash +# Android +uv run flet build apk + +# iOS +uv run flet build ipa + +# iOS-Simulator +uv run flet build ios-simulator +``` diff --git a/recipes/xgboost/examples/iris-explainer/pyproject.toml b/recipes/xgboost/examples/iris-explainer/pyproject.toml new file mode 100644 index 00000000..56e527a6 --- /dev/null +++ b/recipes/xgboost/examples/iris-explainer/pyproject.toml @@ -0,0 +1,24 @@ +[project] +name = "xgboost-iris-explainer" +version = "1.0.0" +description = "Predicts iris species with a bundled XGBoost .joblib model and explains each prediction." +requires-python = ">=3.12" + +dependencies = [ + "flet==1.0.3", + "xgboost==3.4.1", + "joblib==1.6.0", +] + +[dependency-groups] +dev = ["flet-cli", "flet-desktop", "flet-web"] +train = ["scikit-learn==1.9.0"] + +[tool.flet.app] +path = "src" + +[tool.flet.android] +target_arch = ["arm64-v8a", "x86_64"] + +[tool.flet.cleanup] +package_files = ["scipy/**/tests", "numpy/tests", "numpy/*/tests"] diff --git a/recipes/xgboost/examples/iris-explainer/src/assets/iris_xgb.joblib b/recipes/xgboost/examples/iris-explainer/src/assets/iris_xgb.joblib new file mode 100644 index 00000000..2cca3557 Binary files /dev/null and b/recipes/xgboost/examples/iris-explainer/src/assets/iris_xgb.joblib differ diff --git a/recipes/xgboost/examples/iris-explainer/src/classifier.py b/recipes/xgboost/examples/iris-explainer/src/classifier.py new file mode 100644 index 00000000..df83151d --- /dev/null +++ b/recipes/xgboost/examples/iris-explainer/src/classifier.py @@ -0,0 +1,70 @@ +import os +import time +from dataclasses import dataclass + +import joblib +import numpy as np +import xgboost as xgb + +MODEL_FILE = "iris_xgb.joblib" + +# XGBoost encodes classes as 0..n-1, so the names live with the app. +SPECIES = ("setosa", "versicolor", "virginica") + +# Label, slider minimum, maximum and starting value, in centimetres. +FEATURES = ( + ("Sepal length", 4.3, 7.9, 5.8), + ("Sepal width", 2.0, 4.4, 3.0), + ("Petal length", 1.0, 6.9, 4.4), + ("Petal width", 0.1, 2.5, 1.3), +) + + +@dataclass +class Prediction: + """One classification as plain values, ready for the UI.""" + + species: str + probabilities: list[float] + contributions: list[float] + elapsed_ms: float + + +def load(): + """Load the classifier train.py wrote into the app's assets. + + A .joblib file is a pickle, and unpickling runs code, so load only a file your + own build shipped. Built apps and `flet run` put the assets folder in + FLET_ASSETS_DIR; a plain `python main.py` does not, hence the fallback. + """ + assets = os.getenv("FLET_ASSETS_DIR") or os.path.join( + os.path.dirname(__file__), "assets" + ) + return joblib.load(os.path.join(assets, MODEL_FILE)) + + +def describe(model): + """Name the model and the build running it, so a device run shows which it got.""" + rounds = model.get_booster().num_boosted_rounds() + openmp = "on" if xgb.build_info()["USE_OPENMP"] else "off" + return f"xgboost {xgb.__version__} · {rounds} rounds · OpenMP {openmp}" + + +def predict(model, measurements): + """Classify one flower and say how much each measurement pushed the winner. + + `pred_contribs=True` is XGBoost's built-in TreeSHAP: per class, one score per + feature plus a bias column, which together sum to that class's raw margin. + """ + row = np.asarray([measurements], dtype=np.float32) + start = time.perf_counter() + probabilities = model.predict_proba(row)[0] + winner = int(probabilities.argmax()) + shap = model.get_booster().predict(xgb.DMatrix(row), pred_contribs=True) + elapsed_ms = (time.perf_counter() - start) * 1000 + return Prediction( + species=SPECIES[winner], + probabilities=probabilities.tolist(), + contributions=shap[0, winner, :-1].tolist(), + elapsed_ms=elapsed_ms, + ) diff --git a/recipes/xgboost/examples/iris-explainer/src/main.py b/recipes/xgboost/examples/iris-explainer/src/main.py new file mode 100644 index 00000000..bb43b7ae --- /dev/null +++ b/recipes/xgboost/examples/iris-explainer/src/main.py @@ -0,0 +1,90 @@ +import flet as ft +from classifier import FEATURES, SPECIES, describe, load, predict + + +def main(page: ft.Page): + model = None + + def start(): + """Load the bundled model on a worker thread, then show the first prediction. + + run_thread only logs a worker's exception, never showing it in the UI, and + does not carry an automatic update, so this catches its own failures and + ends with page.update(). + """ + nonlocal model + try: + model = load() + header.value = describe(model) + show() + except Exception as exc: + header.value = ( + f"Could not load or run the model: {type(exc).__name__}: {exc}" + ) + page.update() + + def show(): + """Predict for the slider values and redraw the verdict, odds and reasons.""" + if model is None: + return + result = predict(model, [slider.value for slider in sliders]) + verdict.value = f"{result.species} · {max(result.probabilities):.0%}" + for bar, label, p in zip(bars, odds, result.probabilities): + bar.value = p + label.value = f"{p:.0%}" + for bar, label, c in zip(pushes, scores, result.contributions): + bar.width = min(abs(c), 4) * 30 + bar.bgcolor = ft.Colors.GREEN if c >= 0 else ft.Colors.RED + label.value = f"{c:+.2f}" + why.value = f"Why {result.species}: how each measurement moved its score" + timing.value = f"predicted and explained in {result.elapsed_ms:.1f} ms" + page.update() + + sliders = [ + ft.Slider( + min=low, + max=high, + value=start, + divisions=round((high - low) * 10), + round=1, + label="{value} cm", + on_change=show, + ) + for _, low, high, start in FEATURES + ] + bars = [ft.ProgressBar(value=0, expand=True) for _ in SPECIES] + odds = [ft.Text("", width=44) for _ in SPECIES] + pushes = [ft.Container(height=12, width=0) for _ in FEATURES] + scores = [ft.Text("", width=52) for _ in FEATURES] + + page.appbar = ft.AppBar(title="Iris explainer", center_title=True) + page.add( + ft.SafeArea( + expand=True, + content=ft.Column( + scroll=ft.ScrollMode.AUTO, + controls=[ + header := ft.Text("Loading iris_xgb.joblib…", size=12), + *( + ft.Column(spacing=0, controls=[ft.Text(f"{name} (cm)"), slider]) + for (name, *_), slider in zip(FEATURES, sliders) + ), + verdict := ft.Text("", size=24, weight=ft.FontWeight.BOLD), + *( + ft.Row(controls=[ft.Text(name, width=90), bar, label]) + for name, bar, label in zip(SPECIES, bars, odds) + ), + why := ft.Text("", size=12), + *( + ft.Row(controls=[ft.Text(name, width=90), label, bar]) + for (name, *_), bar, label in zip(FEATURES, pushes, scores) + ), + timing := ft.Text("", size=12), + ], + ), + ) + ) + page.run_thread(start) + + +ft.run(main) diff --git a/recipes/xgboost/examples/iris-explainer/train.py b/recipes/xgboost/examples/iris-explainer/train.py new file mode 100644 index 00000000..b4dfc34e --- /dev/null +++ b/recipes/xgboost/examples/iris-explainer/train.py @@ -0,0 +1,14 @@ +"""Train the model this app ships, on a desktop. + +Run it with `uv run --group train python train.py`. It writes +src/assets/iris_xgb.joblib, the file the app loads on the device. +""" + +import joblib +from sklearn.datasets import load_iris +from xgboost import XGBClassifier + +features, species = load_iris(return_X_y=True) +model = XGBClassifier(n_estimators=30, max_depth=3, random_state=0) +model.fit(features, species) +joblib.dump(model, "src/assets/iris_xgb.joblib", compress=3) diff --git a/recipes/xgboost/meta.yaml b/recipes/xgboost/meta.yaml new file mode 100644 index 00000000..127f4c86 --- /dev/null +++ b/recipes/xgboost/meta.yaml @@ -0,0 +1,60 @@ +package: + name: xgboost + version: "3.4.1" + # Upstream assumes a 64-bit size_t (common/quantile.cc narrows an unsigned + # long long into one, a hard -Wc++11-narrowing error) and ships no 32-bit + # builds anywhere. + excluded_arches: + - armeabi-v7a + +build: + number: 1 + script_env: +# {% if sdk == 'android' %} + # libxgboost is a plain C API library loaded through ctypes, not an + # extension module, so no Python_* hints are needed. OpenMP stays on: the + # NDK ships a real runtime. + CMAKE_ARGS: >- + -DCMAKE_TOOLCHAIN_FILE={NDK_ROOT}/build/cmake/android.toolchain.cmake + -DANDROID_ABI={ANDROID_ABI} + -DANDROID_NATIVE_API_LEVEL={ANDROID_API_LEVEL} + -DANDROID_STL=c++_shared + -DCMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=16384 +# {% else %} + # The `ninja` pip package crashes importing on the iOS crossenv python + # (sysconfig.get_preferred_scheme('user') -> 'posix_user' invalid on iOS), + # so use the Makefiles generator on iOS; scikit-build-core then never imports + # ninja. (Android keeps Ninja, which works there.) + CMAKE_GENERATOR: Unix Makefiles + # iOS has no OpenMP runtime, and upstream's Apple OpenMP lookup would ask + # Homebrew for the build host's macOS libomp. Single-threaded is a build + # configuration upstream supports. + CMAKE_ARGS: >- + -DCMAKE_SYSTEM_NAME=iOS + -DCMAKE_OSX_SYSROOT={{ sdk }} + -DCMAKE_OSX_DEPLOYMENT_TARGET={{ sdk_version }} + -DCMAKE_OSX_ARCHITECTURES={{ arch }} + -DUSE_OPENMP=OFF +# {% endif %} + +requirements: + build: + # scikit-build-core comes from the sdist's own build-system.requires; with + # forge's `--no-isolation` the CMake toolchain must be in the build venv. + - cmake + - ninja +# {% if sdk == 'android' %} + host: + # libxgboost links libc++_shared.so and the NDK's libomp.so dynamically; + # neither is on the device unless the app bundles it. + - flet-libcpp-shared >=27.2.12479018 + - flet-libomp 27.3.13750724 +# {% endif %} + +patches: + - mobile.patch + +test: + # Loading a desktop-trained .joblib model is the main way apps get one. + requires: + - joblib diff --git a/recipes/xgboost/patches/mobile.patch b/recipes/xgboost/patches/mobile.patch new file mode 100644 index 00000000..95f8bf1c --- /dev/null +++ b/recipes/xgboost/patches/mobile.patch @@ -0,0 +1,97 @@ +xgboost finds libxgboost and its own VERSION file through paths built from +__file__, and names the library after the platform it was built on. Four +mobile adjustments, all inert on desktop builds: + +1. CMakeLists.txt: name the library libxgboost.so on iOS, not .dylib, so + it has one name on both mobile platforms. serious_python moves it into + an embedded framework and leaves libxgboost.fwork in its place. +2. cpp_src/CMakeLists.txt: no VERSION/SOVERSION on Android and iOS, so the + soname matches the unversioned file the wheel ships. +3. xgboost/libpath.py: teach it the "android" sys.platform (3.13+; Android + 3.12 reports "linux") and "ios" with its .fwork pointer, and give + Android a fallback to the bare soname -- Flet relocates the .so into the + APK's native library dir, where no __file__-relative path can reach it. +4. xgboost/_c_api.py: read VERSION through importlib.resources. On Android + Flet ships site-packages as a zip, so the __file__-relative open() runs + into NotADirectoryError on every `import xgboost`. + +diff -ruN a/CMakeLists.txt b/CMakeLists.txt +--- a/CMakeLists.txt 2026-10-03 20:43:24 ++++ b/CMakeLists.txt 2026-10-04 03:18:37 +@@ -39,6 +39,9 @@ + # Platform-specific library file name (matches xgboost/libpath.py). + if(WIN32) + set(_XGB_LIBNAME "xgboost.dll") ++elseif(CMAKE_SYSTEM_NAME STREQUAL "iOS") ++ # mobile-forge: one library name on both mobile platforms. ++ set(_XGB_LIBNAME "libxgboost.so") + elseif(APPLE) + set(_XGB_LIBNAME "libxgboost.dylib") + else() +diff -ruN a/cpp_src/CMakeLists.txt b/cpp_src/CMakeLists.txt +--- a/cpp_src/CMakeLists.txt 2026-10-03 20:43:24 ++++ b/cpp_src/CMakeLists.txt 2026-10-03 20:43:50 +@@ -405,9 +405,11 @@ + else() + # Provide versioned shared object + add_library(xgboost SHARED) +- set_target_properties(xgboost PROPERTIES +- VERSION ${PROJECT_VERSION} +- SOVERSION ${PROJECT_VERSION_MAJOR}) ++ if(NOT CMAKE_SYSTEM_NAME MATCHES "^(Android|iOS)$") ++ set_target_properties(xgboost PROPERTIES ++ VERSION ${PROJECT_VERSION} ++ SOVERSION ${PROJECT_VERSION_MAJOR}) ++ endif() + endif() + target_link_libraries(xgboost PRIVATE objxgboost) + target_include_directories(xgboost +diff -ruN a/xgboost/_c_api.py b/xgboost/_c_api.py +--- a/xgboost/_c_api.py 2026-10-03 21:12:29 ++++ b/xgboost/_c_api.py 2026-10-03 21:12:29 +@@ -82,9 +82,12 @@ + + def _py_version() -> str: + """Get the XGBoost version from Python version file.""" +- version_file = os.path.join(os.path.dirname(__file__), "VERSION") +- with open(version_file, encoding="ascii") as f: +- return f.read().strip() ++ # mobile-forge: importlib.resources also reads it from a zipped ++ # site-packages (Flet on Android), where a __file__ path cannot. ++ from importlib.resources import files ++ ++ version_file = files(__package__).joinpath("VERSION") ++ return version_file.read_text(encoding="ascii").strip() + + + def _register_log_callback(lib: ctypes.CDLL) -> None: +diff -ruN a/xgboost/libpath.py b/xgboost/libpath.py +--- a/xgboost/libpath.py 2026-10-03 20:43:24 ++++ b/xgboost/libpath.py 2026-10-04 03:18:37 +@@ -49,16 +49,25 @@ + ] + ) + dll_path = [os.path.join(p, "xgboost.dll") for p in dll_path] +- elif sys.platform.startswith(("linux", "freebsd", "emscripten")): ++ elif sys.platform.startswith(("linux", "freebsd", "emscripten", "android")): + dll_path = [os.path.join(p, "libxgboost.so") for p in dll_path] + elif sys.platform == "darwin": + dll_path = [os.path.join(p, "libxgboost.dylib") for p in dll_path] ++ elif sys.platform == "ios": ++ # mobile-forge: the app packager moves the library into an embedded ++ # framework and leaves a .fwork pointer, which iOS ctypes follows. ++ names = ("libxgboost.fwork", "libxgboost.so") ++ dll_path = [os.path.join(p, n) for p in dll_path for n in names] + elif sys.platform == "cygwin": + dll_path = [os.path.join(p, "cygxgboost.dll") for p in dll_path] + if platform.system() == "OS400": + dll_path = [os.path.join(p, "libxgboost.so") for p in dll_path] + + lib_path = [p for p in dll_path if os.path.exists(p) and os.path.isfile(p)] ++ if not lib_path and hasattr(sys, "getandroidapilevel"): ++ # mobile-forge: Flet moves the .so out of site-packages into the APK's ++ # native library directory, where the linker resolves it by bare name. ++ lib_path = ["libxgboost.so"] + + if not lib_path and not is_sphinx_build(): + link = "https://xgboost.readthedocs.io/en/stable/install.html" diff --git a/recipes/xgboost/tests/iris_classifier.joblib b/recipes/xgboost/tests/iris_classifier.joblib new file mode 100644 index 00000000..7d3fade6 Binary files /dev/null and b/recipes/xgboost/tests/iris_classifier.joblib differ diff --git a/recipes/xgboost/tests/test_xgboost.py b/recipes/xgboost/tests/test_xgboost.py new file mode 100644 index 00000000..6936bf17 --- /dev/null +++ b/recipes/xgboost/tests/test_xgboost.py @@ -0,0 +1,100 @@ +import os +import platform +import sys + +import numpy as np +import pytest + +HERE = os.path.dirname(__file__) + +# iris_classifier.joblib was written on a desktop with xgboost 3.4.1 and +# scikit-learn 1.9.0: +# model = XGBClassifier(n_estimators=8, max_depth=2, random_state=0) +# model.fit(*load_iris(return_X_y=True)) +# joblib.dump(model, "iris_classifier.joblib", compress=3) +# IRIS_PROBA is that model's predict_proba(IRIS_ROWS), computed there. +IRIS_ROWS = [[5.1, 3.5, 1.4, 0.2], [6.6, 3.0, 4.4, 1.4], [6.7, 3.1, 5.6, 2.4]] +IRIS_PROBA = [ + [0.922946, 0.039821, 0.037233], + [0.04813, 0.903655, 0.048215], + [0.037542, 0.04443, 0.918028], +] + + +def _data(seed=0, n=400): + rng = np.random.default_rng(seed) + X = rng.normal(size=(n, 5)).astype(np.float32) + y = ((X[:, 0] > 0) & (X[:, 1] < 0.5) | (X[:, 2] > 1.0)).astype(np.int32) + return X, y + + +def test_train_and_predict(): + """Full train->predict loop through the ctypes C API (libxgboost).""" + import xgboost as xgb + + X, y = _data() + booster = xgb.train( + {"objective": "binary:logistic", "nthread": 2, "seed": 0}, + xgb.DMatrix(X, label=y), + num_boost_round=30, + ) + acc = ((booster.predict(xgb.DMatrix(X)) > 0.5) == y).mean() + assert acc > 0.95 + + +def test_model_save_load_roundtrip(tmp_path): + """Persistence in the portable UBJSON format.""" + import xgboost as xgb + + X, y = _data(seed=1) + dtrain = xgb.DMatrix(X, label=y) + booster = xgb.train({"objective": "binary:logistic", "seed": 0}, dtrain, 10) + + path = str(tmp_path / "model.ubj") + booster.save_model(path) + loaded = xgb.Booster(model_file=path) + assert np.allclose(loaded.predict(xgb.DMatrix(X)), booster.predict(xgb.DMatrix(X))) + + +def test_pred_contribs_treeshap(): + """predict(pred_contribs=True) is built-in TreeSHAP — feature attribution + without the (numba-blocked) shap package.""" + import xgboost as xgb + + X, y = _data(seed=2) + dtrain = xgb.DMatrix(X, label=y) + booster = xgb.train({"objective": "binary:logistic", "seed": 0}, dtrain, 10) + + dtest = xgb.DMatrix(X[:8]) + contrib = booster.predict(dtest, pred_contribs=True) + assert contrib.shape == (8, X.shape[1] + 1) # + bias column + raw = booster.predict(dtest, output_margin=True) + assert np.allclose(contrib.sum(axis=1), raw, atol=1e-5) + + +def test_desktop_joblib_classifier(): + """A desktop-trained XGBClassifier loaded from .joblib predicts exactly as it + did there, with no scikit-learn installed, and takes a thread cap through + its booster.""" + import joblib + + model = joblib.load(os.path.join(HERE, "iris_classifier.joblib")) + rows = np.asarray(IRIS_ROWS) + assert np.allclose(model.predict_proba(rows), IRIS_PROBA, atol=1e-5) + assert model.predict(rows).tolist() == [0, 1, 2] + + model.get_booster().set_param({"nthread": 1}) + assert np.allclose(model.predict_proba(rows), IRIS_PROBA, atol=1e-5) + + +def test_openmp_matches_platform(): + """OpenMP is compiled in on Android, where flet-libomp ships the runtime, + and out on iOS, which has none.""" + import xgboost as xgb + + if hasattr(sys, "getandroidapilevel"): + assert xgb.build_info()["USE_OPENMP"] is True + elif sys.platform == "ios" or platform.system() == "iOS": + assert xgb.build_info()["USE_OPENMP"] is False + else: + pytest.skip("not a mobile build")