Do not expose hugr-rs's Hugr in hugr-py

[aborgna-q]

This text is not a change proposal, but rather a warning against future implementations.

ABI incompatibilities

@ss2165 raised the issue of incompatibilities between in-memory rust representations of a Hugr if we were to expose functions that read/write hugr::Hugrs from independently compiled hugr-py and tket-py wheels.

Rust does not provide any ABI stability between repr(Rust) data structures on independent compilation runs. In practice, we may assume that two runs of the same rustc with the same flags and the same environment will produce deterministic layouts when compiling the exact same crate (so we could do wheel compilations in shared container definitions) but this may break if there's a dependency patch-version bump, or if the phase of the moon is not-quite-right. (There's even a -Zrandomize_layout flag to test you're not depending on that assertion).

This is not quite compatible with having separate packages for tket-py and hugr-py that try to access and modify shared in-memory rust representations of a hugr. Even if we were to build the wheels in the same workspace, we'd have to lock together wheel versions or not allow any rust-side dependency updates between breaking versions.

Exposing bindings from hugr-py

Currently we have some rust bindings in hugr-py, but they are only used for model AST manipulations.
As long as they don't interact with rust code from other wheels, they should be fine.

However, we should not export bindings that expect or return a hugr::Hugrs or related structures.
I'd like to propose that we rather keep the rust-bindings for Hugr definitions relegated to the tket-py crate (as is the case currently). We should relegate all "efficient" optimization code that interacts with the rust repr to be only defined from those wheels.

Alternatives

We could instead compile the Hugr structure using a stable ABI.
Given that crabi is far from being stable, we would instead use abi_stable or stabby.

These representations however are not free. They must do sacrifices to ensure API stability that fight against the ability of rustc to optimize the code. I haven't benchmarked the performance loss yet (we should probably try stabby), but any slowdown is an unnecessary cost to the hot path of the compilation stack and I seriously doubt we should pay it.

We do have a versioned stable representation of a Hugr already, it's called hugr-model. We should depend on that instead.

Implications for user-written rust code

If a user writes their own rust-backed optimisation and they want to provide python bindings in their own extension, any hugr communication should go via envelope encoding and local decoding in the appropriate in-memory representation.

See also

• "Sharing pyclasses between multiple Rust packages" PyO3/pyo3#1444

• How can 3rd party pyo3 projects do Circuit manipulation? tket2#846

• https://github.com/CQCL/tket2/issues/1167