-
Notifications
You must be signed in to change notification settings - Fork 358
Cosmos: record Go v2 FFI decision #4802
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ananth7592
wants to merge
8
commits into
main
Choose a base branch
from
users/amudumba/go-v2-ffi-decision
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+232
−0
Open
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
72e7098
docs(cosmos): add Go v2 FFI decision
ananth7592 57988cd
docs(cosmos): shape Go v2 decision as ADR
ananth7592 4bb21b3
docs(cosmos): move Go FFI ADR under native docs
ananth7592 5e12106
docs(cosmos): clarify Go FFI packaging caveats
ananth7592 a8dcdf2
docs(cosmos): adjust Go FFI platform notes
ananth7592 0cc2ded
Potential fix for pull request finding
ananth7592 6abaaed
docs(cosmos): scope Go FFI platform matrix
ananth7592 3d68382
docs(cosmos): fix Go FFI Kafka reference
ananth7592 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
20 changes: 20 additions & 0 deletions
20
...driver/docs/native-driver-distribution/go-native-distribution/adr/0000-index.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| # Architecture Decision Records — Go v2 FFI direction | ||
|
|
||
| ADRs (Architecture Decision Records) capture **what we decided** and a brief | ||
| **why**, in a minimal, easy-to-reference form. They are **numbered and | ||
| immutable**: once accepted, an ADR is not edited; a later ADR may supersede it. | ||
| Detailed discussion and exploration live in | ||
| [`go-v2-ffi-exploration.md`](https://github.com/Azure/azure-sdk-for-rust/blob/main/sdk/cosmos/azure_data_cosmos_driver/docs/native-driver-distribution/go-native-distribution/go-v2-ffi-exploration.md), | ||
| not here. | ||
|
|
||
| **Format (template):** Status (immediately under the title) · Context (2-4 | ||
| sentences) · Decision (1-3 bullets) · Consequences (2-4 bullets) · Alternatives | ||
| considered (1 line each). | ||
|
|
||
| | # | Title | Status | | ||
| |---|-------|--------| | ||
| | [0001](https://github.com/Azure/azure-sdk-for-rust/blob/main/sdk/cosmos/azure_data_cosmos_driver/docs/native-driver-distribution/go-native-distribution/adr/0001-go-v2-uses-ffi.md) | Go v2 uses the Rust driver through FFI | Proposed | | ||
|
|
||
| > This ADR is proposed for review. Packaging mechanics are intentionally left to | ||
| > the native distribution ADRs; this record captures the Go v2 implementation | ||
| > direction. |
50 changes: 50 additions & 0 deletions
50
...cs/native-driver-distribution/go-native-distribution/adr/0001-go-v2-uses-ffi.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| # ADR 0001 — Go v2 uses the Rust driver through FFI | ||
|
|
||
| **Status:** Proposed | ||
|
|
||
| ## Context | ||
|
|
||
| Go v2 needs a path that can deliver in the current timeframe while staying close | ||
| to the Rust driver's behavior. A time-boxed pure-Go spike proved that a | ||
| gateway-mode core path can be ported and validated, but it also clarified the | ||
| central trade-off: avoiding cgo and native packaging means re-owning driver | ||
| logic in Go and carrying long-term Rust-to-Go drift risk. | ||
|
|
||
| ## Decision | ||
|
|
||
| - Go v2 proceeds with the **Rust-driver FFI path** for the current delivery | ||
| window. | ||
| - The Go binding consumes the **prebuilt Rust native driver** through cgo and the | ||
| C ABI surface. | ||
| - Packaging mechanics, platform matrix, signing, ABI versioning, and fan-out are | ||
| handled by the native distribution design and ADRs. | ||
|
|
||
| ## Consequences | ||
|
|
||
| - Go v2 gets the same driver implementation as Rust, reducing short-term feature | ||
| drift and implementation risk. | ||
| - Go v2 takes a dependency on `CGO_ENABLED=1` for the native driver path. | ||
| - The Go packaging model becomes part of the product decision: customers should | ||
| not manually install a Rust toolchain or copy native libraries for the common | ||
| path, but cgo still requires an available C build toolchain. | ||
| - Differential validation remains important so Go-visible behavior can be tested | ||
| against the Rust driver. | ||
|
|
||
| ## Alternatives considered | ||
|
|
||
| - **Pure-Go driver port** — technically feasible for the core path tested, but | ||
| not selected for Go v2 because it shifts cost from packaging to long-term Go | ||
| ownership and Rust-to-Go drift. | ||
| - **Manual native-library installation** — rejected for the common path because | ||
| it breaks the normal `go get` / `go build` expectation for a first-party Go | ||
| SDK. | ||
| - **Pure-Go downloader shim** — rejected by the native distribution ADRs; the | ||
| native driver must be packaged and linked through the FFI path, not fetched by | ||
| a downloader stub. | ||
|
|
||
| ## Discussion | ||
|
|
||
| See | ||
| [`go-v2-ffi-exploration.md`](https://github.com/Azure/azure-sdk-for-rust/blob/main/sdk/cosmos/azure_data_cosmos_driver/docs/native-driver-distribution/go-native-distribution/go-v2-ffi-exploration.md) | ||
| for the meeting context, pure-Go spike evidence, packaging concerns, tentative | ||
| platform matrix, and market reference points. | ||
162 changes: 162 additions & 0 deletions
162
...docs/native-driver-distribution/go-native-distribution/go-v2-ffi-exploration.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,162 @@ | ||
| <!-- cspell:ignore amd64 azcosmos GSSAPI librdkafka --> | ||
| # Go v2 FFI exploration notes | ||
|
Copilot marked this conversation as resolved.
|
||
|
|
||
| > **Status:** Discussion notes for ADR review. The decision is recorded in | ||
| > [`ADR 0001`](https://github.com/Azure/azure-sdk-for-rust/blob/main/sdk/cosmos/azure_data_cosmos_driver/docs/native-driver-distribution/go-native-distribution/adr/0001-go-v2-uses-ffi.md). | ||
|
|
||
| ## 1. Purpose | ||
|
|
||
| This document captures the exploration behind the Go v2 decision to proceed with | ||
| the Rust-driver FFI path in the current delivery window. It is intentionally more | ||
| descriptive than the ADR: it records the trade-offs, pure-Go spike evidence, | ||
| packaging implications, and market reference points that informed the decision. | ||
|
|
||
| ## 2. Implementation paths evaluated | ||
|
|
||
| The team evaluated two paths: | ||
|
|
||
| | Path | Shape | | ||
| |---|---| | ||
| | **FFI path** | Go calls the prebuilt Rust driver through cgo and a C ABI. | | ||
| | **Pure-Go port** | Go reimplements the driver behavior directly in Go. | | ||
|
|
||
| Both were technically feasible. The decision is not "can we port it?" The | ||
| decision is: | ||
|
|
||
| > Is avoiding cgo and native packaging worth re-owning driver logic that Rust | ||
| > already implements? | ||
|
|
||
| For the Go v2 timeframe, the team aligned on **FFI** because it gives faster | ||
| delivery, stronger parity with the Rust driver, and lower short-term | ||
| implementation risk. | ||
|
|
||
| ## 3. What the pure-Go spike proved | ||
|
|
||
| The pure-Go spike showed that a gateway-mode core path can be ported in | ||
| idiomatic Go: | ||
|
|
||
| - `CGO_ENABLED=0` | ||
| - zero third-party dependencies | ||
| - point create/read | ||
| - retry and routing behavior | ||
| - basic failover behavior | ||
| - cross-partition unordered query fan-out | ||
| - unsupported-query rejection parity | ||
|
|
||
| The spike was validated with a scenario-based differential harness. The harness | ||
| put an HTTP endpoint in front of the Rust driver's in-memory emulator so the Go | ||
| port and the Rust driver could run against the same emulator-backed state and | ||
| compare caller-visible outcomes. | ||
|
|
||
| This is valuable evidence: the Go port is credible. It is also a useful | ||
| validation reference for Go-visible behavior. It is not the selected Go v2 | ||
| delivery path because every Rust-driver change would need to be tracked, ported, | ||
| and revalidated in Go. | ||
|
|
||
| ## 4. Why FFI for Go v2 | ||
|
|
||
| FFI has the better short-term delivery shape: | ||
|
|
||
| - **Parity:** Go consumes the same driver implementation as Rust. | ||
| - **Velocity:** new Rust-driver behavior does not need to be re-ported before Go | ||
| can benefit. | ||
| - **Risk containment:** the hard driver logic stays in one implementation. | ||
| - **Testing leverage:** the Go binding can validate language-level behavior | ||
| against the Rust driver's known behavior instead of re-proving the full driver. | ||
|
|
||
| The main cost is not binary size. The main cost is **Go packaging and customer | ||
| deployment experience**. | ||
|
|
||
| ## 5. Packaging implications | ||
|
|
||
| Cosmos Go v1 is consumed like a normal Go SDK: | ||
|
|
||
| ```bash | ||
| go get github.com/Azure/azure-sdk-for-go/sdk/data/azcosmos | ||
| go build ./... | ||
| ``` | ||
|
|
||
| There is no Rust compiler, native shared library, or post-install step in the | ||
| customer path. If Go v2 uses FFI, prebuilt Rust artifacts can preserve that part | ||
| of the experience, but cgo still requires a C build toolchain. The packaging bar | ||
|
ananth7592 marked this conversation as resolved.
|
||
| is therefore: | ||
|
|
||
| > Can the Go SDK still feel like one normal Go module across the supported | ||
| > Windows, macOS, and Linux targets? | ||
|
|
||
| That means native Rust driver artifacts must be produced, versioned, selected, | ||
| and linked for the supported Go OS/architecture matrix. | ||
|
|
||
| This discussion scopes the initial packaging estimate to the mainstream | ||
| OS/architecture targets: | ||
|
|
||
| | OS family | Target combinations discussed | | ||
| |---|---| | ||
| | Windows | x64/amd64, ARM64, possibly x86 as a consideration | | ||
| | macOS | ARM64/Apple Silicon, x64/Intel | | ||
| | Linux glibc | x64/amd64, ARM64 | | ||
|
ananth7592 marked this conversation as resolved.
|
||
|
|
||
| That is **six firm targets**, or **seven** if Windows x86 is included. Linux | ||
| musl/Alpine is intentionally not counted here until the Go packaging plan | ||
| decides whether it is in scope for the initial release. | ||
|
|
||
| With an estimated **~5 MB optimized native binary per platform**, this scoped | ||
| matrix is roughly **~30-35 MB before compression**. Any additional targets, such | ||
| as musl/Alpine, add to that footprint. This is manageable for an initial release, | ||
| but it is a real packaging design point and should not be treated as an | ||
| implementation detail. | ||
|
|
||
| ## 6. Packaging models to evaluate | ||
|
|
||
| Two practical models were discussed: | ||
|
|
||
| | Model | Customer experience | Trade-off | | ||
| |---|---|---| | ||
| | **Bundle the full platform matrix in the Go package** | Closest to `go get` / `go build` just working everywhere | Larger module/package footprint; all platform binaries travel together | | ||
| | **Per-platform native dependency** | Customer obtains only the platform-specific binary they need | Smaller per-customer footprint, but more setup burden and more room for install mistakes | | ||
|
|
||
| A bundle-first hybrid has industry precedent. For example, | ||
| [`confluent-kafka-go`](https://github.com/confluentinc/confluent-kafka-go/blob/master/README.md#librdkafka) | ||
| includes prebuilt `librdkafka` binaries for common platforms, requires | ||
| `CGO_ENABLED` to remain enabled, and falls back to a manual dynamic install only | ||
| when customers need unsupported platforms or special features such as | ||
| GSSAPI/Kerberos. | ||
|
|
||
| That reference shows a viable market pattern for a Go library backed by native | ||
| code: **bundle the common path, document unsupported/special paths clearly, and | ||
| make native requirements explicit**. It aligns most closely with the first model, | ||
| with a manual native dependency as an escape hatch. It also confirms that this | ||
| is a materially different customer experience from a pure-Go Azure SDK. | ||
|
|
||
| ## 7. Pure-Go port trade-off | ||
|
|
||
| The pure-Go path has a strong developer-experience story: | ||
|
|
||
| - `CGO_ENABLED=0` | ||
| - normal Go cross-compilation | ||
| - no native runtime dependency | ||
| - Go standard library transport/TLS/runtime | ||
|
|
||
| But it moves the cost into ownership: | ||
|
|
||
| - driver behavior must be reimplemented in Go | ||
| - every Rust-driver change has to be tracked, ported, and revalidated | ||
| - long-term drift becomes the main risk | ||
| - the hardest remaining areas still need evidence: richer query execution, | ||
| continuation/split-resume behavior, broader multi-region behavior, change | ||
| feed, and bulk/batch | ||
|
|
||
| The pure-Go spike remains valuable because it proved the fallback is not | ||
| fictional. It also produced a useful differential-validation approach. But for | ||
| Go v2, it is better treated as evidence and risk retirement, not the shipping | ||
| plan. | ||
|
|
||
| ## 8. Non-goals | ||
|
|
||
| - This discussion does not define the final Go packaging shape. | ||
| - This discussion does not require customers to manually install native | ||
| libraries. | ||
| - This discussion does not introduce direct mode; both the Rust and Go v2 paths | ||
| are gateway-mode SDKs. | ||
| - This discussion does not remove the pure-Go spike as a reference or validation | ||
| asset. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.