Ctypes and editable mode

[peppedilillo]

Dears. Using pip install -e . --no-build-isolation, as suggested by the docs, I can correctly install the package and compile my C code, which is called through ctypes. However once I try to use the package it fails to find the shared libraries:

  File "/home/deppep/Dropbox/Progetti/hbstools/hbstools/hbstools/triggers/bft_cwrap.py", line 16, in <module>
    clib_bft = ctypes.CDLL(_LIBCFOCUS)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /home/deppep/Dropbox/Progetti/hbstools/hbstools/hbstools/triggers/.sharedlibs/lib-pfocus.so: cannot open shared object file: No such file or directory

The _LIBCFOCUS variable above is implemented here:

def _library_path(libname: str) -> str:
    """Gets the path to the C shared library, with appropriate extension.
    Assumes the library and this file to be installed in the same directory"""
    from pathlib import Path
    import sys

    if sys.platform.startswith("win32"):
        suffix = ".dll"
    elif sys.platform.startswith("linux"):
        suffix = ".so"
    elif sys.platform.startswith("darwin"):
        suffix = ".dylib"
    else:
        raise OSError("System not supported")

    return str(Path(__file__).parent / f"{libname}{suffix}")


_LIBCFOCUS = _library_path(".sharedlibs/lib-pfocus")

If you want to take a look, here is the software I'm working on.

The problem is that, once in editable mode, the code looks for the shared libraries in the wrong place. While not in editable mode, the libraries are installed in my local environment directory (site-packages/hbstools/triggers/.sharedlibs/lib-pfocus.so). However, once in editable-mode, I do not know where they are actually installed, but they are expected to be in my source directory.

I have asked this question under an issue (sorry @rgommers for reviving a dead issue) and got a kind reply from @eli-schwartz, which suggested to install the shared libraries in multiple places, so that I can try to import from these. However it is not fully clear to me how this should be carried out, and if there is any better solution not requiring multiple copies of the same file.

Is it possible to install my project in editable mode?

Thank you very much.

[keturn]

I think the trick here is to use importlib.resources instead of __file__:

importlib.resources.files(__package__).joinpath(".sharedlibs/libp-focus.so")

meson-python's editable-mode hooks will resolve that to the build directory.

[eli-schwartz]

importlib-resources is totally unrelated to the challenges of finding files in an editable install, using a different source/build layout than a "real" install would have.

[rgommers]

I don't think it is, this answer looks correct to me. The editable install works through importlib hooks, and importlib.resources is the best way to load files. This is what I needed to do in SciPy as well to load generated files: scipy/scipy@2ddb28b.

[eli-schwartz]

I have asked this question under an issue (sorry @rgommers for reviving a dead issue) and got a kind reply from @eli-schwartz, which suggested to install the shared libraries in multiple places, so that I can try to import from these. However it is not fully clear to me how this should be carried out, and if there is any better solution not requiring multiple copies of the same file.

I think I had suggested at the time, not that you install the libraries in multiple places, but rather that your _LIBCFOCUS variable should look in multiple places.

The library is created in hbstools/triggers/meson.build

You look for the library in hbstools/triggers/__init__.py

That means that the library itself may be in one of two places:

• installed, non-editable, next to that __init__.py, in ./.sharedlibs/{libname}{suffix}
• inside the meson build directory, at {build_directory}/hsbtools/triggers/{libname}{suffix}

The trick is going to be locating {build_directory} so you can look at that instead. The "easiest" way is if __file__ for your editable install is pointing to {build_directory}/hsbtools/triggers/__init__.py.