remove private types from pyo3-ffi

[davidhewitt]

Related to #4379, also closes #5062

This PR removes all types and structures starting with _Py from pyo3-ffi. These are private and may be freely altered in patch releases (as seen in #5062), so we should not expose these to users, and should only bother defining them in pyo3-ffi when they're necessary to satisfy public CPython ABI.

As part of this PR I changed the opaque_struct! macro to require explicitly opting-in to pub (or pub(crate)), rather than always making the defined item public.

[mejrs]

Thanks! Were you planning to just limit this to underscore apis? There are things like PyCodeObject and PyHeapTypeObject that aren't exactly public api either.

[davidhewitt]

There are things like PyCodeObject and PyHeapTypeObject that aren't exactly public api either.

As per #5064 (comment) have made PyCodeObject opaque.

There are definitely other structures which should not be public, but given that each time we change this we might break downstream code I would prefer to just remove these when there's a good reason, e.g. the CPython headers hide the symbol or explicitly document as private API. Also if we removed PyHeapTypeObject, probably warrants its own PR at the very least.

[mejrs]

Thanks, that's a nice line diff.

[clin1234]

Wholeheartedly agree, especially when CPython's C interfaces changes in prerelease builds!

[mejrs]

@davidhewitt the auto merge didn't quite work:

 error: type `_PyCompilerSrcLocation` is more private than the item `PyFutureFeatures::ff_location`
    --> pyo3-ffi/src/cpython/compile.rs:53:5
     |
  53 |     pub ff_location: _PyCompilerSrcLocation,
     |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ field `PyFutureFeatures::ff_location` is reachable at visibility `pub`
     |
  note: but type `_PyCompilerSrcLocation` is only usable at visibility `pub(self)`
    --> pyo3-ffi/src/cpython/compile.rs:36:1
     |
  36 | struct _PyCompilerSrcLocation {
     | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     = note: `-D private-interfaces` implied by `-D warnings`
     = help: to override `-D warnings` add `#[allow(private_interfaces)]`
  
  error: could not compile `pyo3-ffi` (lib) due to 1 previous error

[davidhewitt]

Thanks, I've decided to fix by restricting visibility of PyFutureFeatures struct to < 3.10 (it was unused in the public C API after the PyFuture_From* functions were made internal in 3.10).

[ngoldbaum]

Let's try to merge once again.

[ngoldbaum]

Windows build failed with:

  error[E0609]: no field `co_freevars` on type `code::PyCodeObject`
    --> pyo3-ffi\src\cpython\code.rs:89:35
     |
  89 |     crate::PyTuple_GET_SIZE((*op).co_freevars)
     |                                   ^^^^^^^^^^^ unknown field
     |
     = note: available field is: `0`
  
  For more information about this error, try `rustc --explain E0609`

[davidhewitt]

I'm just going to remove that function; if we can't depend on the layout of PyCodeObject the current implementation makes no sense. If a user reports this as an issue, we can think about solutions then.