SPICE-0015: C library for Pkl #16

KushalP · 2025-03-18T15:30:05Z

This adds a new SPICE to provide native bindings for Pkl.

Relevant discussions/issues

spices/SPICE-0015-c-library-for-pkl.adoc

This introduces native C bindings for Pkl. Dynamic Library --------------- Using `org.graalvm.nativeimage` and the `native-image` binary we produce a dynamic library (for each OS/Arch variant) that provides a way to initialise itself with a `graal_isolatethread_t`. Methods are annotated with `@CEntryPoint` and exported. This change results in an architecture and OS-specific directory being created which now produces the headers for our shared library functions: ``` ❯ ll libpkl/build/libs/macos-aarch64/ graal_isolate_dynamic.h graal_isolate.h libpkl-internal_dynamic.h libpkl-internal-macos-aarch64_dynamic.h libpkl-internal-macos-aarch64.dylib libpkl-internal-macos-aarch64.h libpkl-internal.dylib libpkl-internal.h ``` `libpkl` -------- The produced `libpkl` dynamic library wraps the GraalVM C interface into something that is future-friendly for the needs of a Pkl integrator. It exports an interface which aligns with SPICE-0015[1]. ``` ❯ ll libpkl/build/libs/macos-aarch64/ graal_isolate_dynamic.h graal_isolate.h libpkl-internal_dynamic.h libpkl-internal-macos-aarch64_dynamic.h libpkl-internal-macos-aarch64.dylib libpkl-internal-macos-aarch64.h libpkl-internal.dylib libpkl-internal.h libpkl.dylib <--- this is new libpkl.h <--- this is new ``` JNA --- Testing of the produced `libpkl` dynamic library is done using Java Native Access[2] for ease. We provide an `interface` in Kotlin which JNA transposes against the `libpkl` dynamic library discoverable at the path that is discoverable with `jna.library.path`. Load in `projects.pklCommonsCli` to deal with `UnsupportedFeatureException` --------------------------------------------------------------------------- This is to deal with the following error: ``` Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a started Thread in the image heap. Thread name: main. Threads running in the image generator are no longer running at image runtime. If these objects should not be stored in the image heap, you can use '--trace-object-instantiation=java.lang.Thread' ``` [1] apple/pkl-evolution#16

This introduces native C bindings for Pkl. Dynamic Library --------------- Using `org.graalvm.nativeimage` and the `native-image` binary we produce a dynamic library (for each OS/Arch variant) that provides a way to initialise itself with a `graal_isolatethread_t`. Methods are annotated with `@CEntryPoint` and exported. This change results in an architecture and OS-specific directory being created which now produces the headers for our shared library functions: ``` ❯ ll libpkl/build/libs/macos-aarch64/ graal_isolate_dynamic.h graal_isolate.h libpkl-internal_dynamic.h libpkl-internal-macos-aarch64_dynamic.h libpkl-internal-macos-aarch64.dylib libpkl-internal-macos-aarch64.h libpkl-internal.dylib libpkl-internal.h ``` `libpkl` -------- The produced `libpkl` dynamic library wraps the GraalVM C interface into something that is future-friendly for the needs of a Pkl integrator. It exports an interface which aligns with SPICE-0015[1]. ``` ❯ ll libpkl/build/libs/macos-aarch64/ graal_isolate_dynamic.h graal_isolate.h libpkl-internal_dynamic.h libpkl-internal-macos-aarch64_dynamic.h libpkl-internal-macos-aarch64.dylib libpkl-internal-macos-aarch64.h libpkl-internal.dylib libpkl-internal.h libpkl.dylib <--- this is new libpkl.h <--- this is new ``` JNA --- Testing of the produced `libpkl` dynamic library is done using Java Native Access[2] for ease. We provide an `interface` in Kotlin which JNA transposes against the `libpkl` dynamic library discoverable at the path that is discoverable with `jna.library.path`. Load in `projects.pklCommonsCli` to deal with `UnsupportedFeatureException` --------------------------------------------------------------------------- This is to deal with the following error: ``` Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a started Thread in the image heap. Thread name: main. Threads running in the image generator are no longer running at image runtime. If these objects should not be stored in the image heap, you can use '--trace-object-instantiation=java.lang.Thread' ``` [1] apple/pkl-evolution#16 Rename to `LibPklLibrary` Remove `lib` prefix from `.h` and `.c` files Add doxygen-style comments to `pkl.h` file Move pointer next to variable name Add TODO comment to clean this up once on a feature-branch Add `@SuppressWarnings("unused")` annotation to `LibPkl` class Move `LibPkl` constructor between fields and methods Add `package-info.java` for `org.pkl.libpkl` package Propagate param `handlerContext` so users can trace messages This allows a user to provide a pointer, and have it be passed through and back to their `PklMessageResponseHandler` handler for them track as part of their own bookkeeping. Drop the OS and Arch name from the `native-image` shared library TODO: Provide a meaningful error to user if `cb == null` Log exceptions in `NativeTransport` Address PR comments Gradle adjustments for libpkl (#1081) * Move native tests into `nativeTest` source set; they are now run with `./gradlew testNative` and skipped in `./gradlew test`. * Fix native tests to be platform independent * In `NativeImageBuild`: Add `abstract val sharedLibrary: Property<Boolean>` property to `NativeImageBuild` * In `NativeImageBuild`: Make `mainClass` optional * Introduce enum class `Target`, which enumerates over all the target machines that we support building * Simplify build logic (move common logic into `configure()` extension methods) * Remove `LibPkl.main`, which is no longer needed * Run `libpkl` tests in CircleCI (let's run these for now on this branch, but let's remove it prior to merging). Make libpkl tests use AbstractServerTest (#1084) * Move AbstractServerTest and its dependencies into pkl-commons-test * Make LibPklTest implement AbstractServerTest * Rename "NativeTest" -> "LibPklTest" * Rename "LibPklLibrary" -> "LibPklJNA" Fix `pkl.h` header include path for `Exec.configureCompile` (#1123) Otherwise the main interface `pkl.h` won't be present in the generated library directory. Expose version of Pkl within native library using `pkl_version` (#1124) * Be consistent: use `userData` as parameter name, as per SPICE * Expose version of Pkl within native library using `pkl_version` This allows consumers of the native library to know which version of Pkl they're using, e.g. to build version-specific gates around newer functionality. Run `graal_attach_thread` ahead of all calls (#1131) * Rename `isolatethread` to `graal_isolatethread` * Run `graal_attach_thread` ahead of all calls This is to deal with the case when using a language like Golang, where a Goroutine is managed by the runtime, and can be moved to an arbitrary system thread behind the scenes.

This introduces native C bindings for Pkl. Dynamic Library --------------- Using `org.graalvm.nativeimage` and the `native-image` binary we produce a dynamic library (for each OS/Arch variant) that provides a way to initialise itself with a `graal_isolatethread_t`. Methods are annotated with `@CEntryPoint` and exported. This change results in an architecture and OS-specific directory being created which now produces the headers for our shared library functions: ``` ❯ ll libpkl/build/libs/macos-aarch64/ graal_isolate_dynamic.h graal_isolate.h libpkl-internal_dynamic.h libpkl-internal-macos-aarch64_dynamic.h libpkl-internal-macos-aarch64.dylib libpkl-internal-macos-aarch64.h libpkl-internal.dylib libpkl-internal.h ``` `libpkl` -------- The produced `libpkl` dynamic library wraps the GraalVM C interface into something that is future-friendly for the needs of a Pkl integrator. It exports an interface which aligns with SPICE-0015[1]. ``` ❯ ll libpkl/build/libs/macos-aarch64/ graal_isolate_dynamic.h graal_isolate.h libpkl-internal_dynamic.h libpkl-internal-macos-aarch64_dynamic.h libpkl-internal-macos-aarch64.dylib libpkl-internal-macos-aarch64.h libpkl-internal.dylib libpkl-internal.h libpkl.dylib <--- this is new libpkl.h <--- this is new ``` JNA --- Testing of the produced `libpkl` dynamic library is done using Java Native Access[2] for ease. We provide an `interface` in Kotlin which JNA transposes against the `libpkl` dynamic library discoverable at the path that is discoverable with `jna.library.path`. Load in `projects.pklCommonsCli` to deal with `UnsupportedFeatureException` --------------------------------------------------------------------------- This is to deal with the following error: ``` Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a started Thread in the image heap. Thread name: main. Threads running in the image generator are no longer running at image runtime. If these objects should not be stored in the image heap, you can use '--trace-object-instantiation=java.lang.Thread' ``` [1] apple/pkl-evolution#16 Rename to `LibPklLibrary` Remove `lib` prefix from `.h` and `.c` files Add doxygen-style comments to `pkl.h` file Move pointer next to variable name Add TODO comment to clean this up once on a feature-branch Add `@SuppressWarnings("unused")` annotation to `LibPkl` class Move `LibPkl` constructor between fields and methods Add `package-info.java` for `org.pkl.libpkl` package Propagate param `handlerContext` so users can trace messages This allows a user to provide a pointer, and have it be passed through and back to their `PklMessageResponseHandler` handler for them track as part of their own bookkeeping. Drop the OS and Arch name from the `native-image` shared library TODO: Provide a meaningful error to user if `cb == null` Log exceptions in `NativeTransport` Address PR comments Gradle adjustments for libpkl (apple#1081) * Move native tests into `nativeTest` source set; they are now run with `./gradlew testNative` and skipped in `./gradlew test`. * Fix native tests to be platform independent * In `NativeImageBuild`: Add `abstract val sharedLibrary: Property<Boolean>` property to `NativeImageBuild` * In `NativeImageBuild`: Make `mainClass` optional * Introduce enum class `Target`, which enumerates over all the target machines that we support building * Simplify build logic (move common logic into `configure()` extension methods) * Remove `LibPkl.main`, which is no longer needed * Run `libpkl` tests in CircleCI (let's run these for now on this branch, but let's remove it prior to merging). Make libpkl tests use AbstractServerTest (apple#1084) * Move AbstractServerTest and its dependencies into pkl-commons-test * Make LibPklTest implement AbstractServerTest * Rename "NativeTest" -> "LibPklTest" * Rename "LibPklLibrary" -> "LibPklJNA" Fix `pkl.h` header include path for `Exec.configureCompile` (apple#1123) Otherwise the main interface `pkl.h` won't be present in the generated library directory. Expose version of Pkl within native library using `pkl_version` (apple#1124) * Be consistent: use `userData` as parameter name, as per SPICE * Expose version of Pkl within native library using `pkl_version` This allows consumers of the native library to know which version of Pkl they're using, e.g. to build version-specific gates around newer functionality. Run `graal_attach_thread` ahead of all calls (apple#1131) * Rename `isolatethread` to `graal_isolatethread` * Run `graal_attach_thread` ahead of all calls This is to deal with the case when using a language like Golang, where a Goroutine is managed by the runtime, and can be moved to an arbitrary system thread behind the scenes.

This introduces native C bindings for Pkl. Dynamic Library --------------- Using `org.graalvm.nativeimage` and the `native-image` binary we produce a dynamic library (for each OS/Arch variant) that provides a way to initialise itself with a `graal_isolatethread_t`. Methods are annotated with `@CEntryPoint` and exported. This change results in an architecture and OS-specific directory being created which now produces the headers for our shared library functions: ``` ❯ ll libpkl/build/libs/macos-aarch64/ graal_isolate_dynamic.h graal_isolate.h libpkl-internal_dynamic.h libpkl-internal-macos-aarch64_dynamic.h libpkl-internal-macos-aarch64.dylib libpkl-internal-macos-aarch64.h libpkl-internal.dylib libpkl-internal.h ``` `libpkl` -------- The produced `libpkl` dynamic library wraps the GraalVM C interface into something that is future-friendly for the needs of a Pkl integrator. It exports an interface which aligns with SPICE-0015[1]. ``` ❯ ll libpkl/build/libs/macos-aarch64/ graal_isolate_dynamic.h graal_isolate.h libpkl-internal_dynamic.h libpkl-internal-macos-aarch64_dynamic.h libpkl-internal-macos-aarch64.dylib libpkl-internal-macos-aarch64.h libpkl-internal.dylib libpkl-internal.h libpkl.dylib <--- this is new libpkl.h <--- this is new ``` JNA --- Testing of the produced `libpkl` dynamic library is done using Java Native Access[2] for ease. We provide an `interface` in Kotlin which JNA transposes against the `libpkl` dynamic library discoverable at the path that is discoverable with `jna.library.path`. Load in `projects.pklCommonsCli` to deal with `UnsupportedFeatureException` --------------------------------------------------------------------------- This is to deal with the following error: ``` Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a started Thread in the image heap. Thread name: main. Threads running in the image generator are no longer running at image runtime. If these objects should not be stored in the image heap, you can use '--trace-object-instantiation=java.lang.Thread' ``` [1] apple/pkl-evolution#16

KushalP marked this pull request as ready for review March 18, 2025 15:36

bioball reviewed Mar 18, 2025

View reviewed changes

spices/SPICE-0015-c-library-for-pkl.adoc Outdated Show resolved Hide resolved

bioball reviewed Mar 18, 2025

View reviewed changes

spices/SPICE-0015-c-library-for-pkl.adoc Outdated Show resolved Hide resolved

KushalP mentioned this pull request Mar 18, 2025

C library for Pkl apple/pkl#1026

Merged

KushalP added 7 commits June 3, 2025 14:13

Add SPICE: "C library for Pkl"

1dd67a9

Remove options on pkl_init

5035a52

pkl_init: returns 0 on success, or a non-zero value on failure

0df102b

Switch to doxygen-style comments

656b266

pkl_init registers the callback handler

785c002

Be explicit about return codes

2c819cc

Update doc comments to match what's in-progress on apple/pkl

beeaa06

KushalP force-pushed the c-library-for-pkl branch from 72860f1 to beeaa06 Compare June 3, 2025 13:15

KushalP added 3 commits July 14, 2025 17:01

Fix broken opening comment

df1c64e

Be consistent, use userData everywhere

d09845e

pkl_version provides the Pkl version within the native library

78ac110

KushalP mentioned this pull request Jul 22, 2025

C library for Pkl apple/pkl#1134

Closed

6 tasks

This was referenced Jul 22, 2025

C library for Pkl apple/pkl#1135

Closed

C library for Pkl apple/pkl#1136

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

SPICE-0015: C library for Pkl #16

SPICE-0015: C library for Pkl #16

Uh oh!

KushalP commented Mar 18, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SPICE-0015: C library for Pkl #16

Are you sure you want to change the base?

SPICE-0015: C library for Pkl #16

Uh oh!

Conversation

KushalP commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant discussions/issues

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

KushalP commented Mar 18, 2025 •

edited

Loading