headers installed to a virtualenv are omitted from builds

[dlech]

setuptools version

68.0.0

Python version

CPython 3.11

OS

macOS

Additional environment information

using virtual environment created with python -m venv .venv.

Description

I am trying to build a binary extension (example2) that uses headers from another binary extension (example1) that has been installed in the virtual environment. pip installs the headers to .venv/include/site/python3.11/example2. Here is the code from pip where this path comes from.

However, setuptools (invoked with python -m build example2 --no-isolation) is only passing -I/Users/david/work/junk/setuptools-test/.venv/include -I/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 so the headers from example1 are not found and the build fails.

Expected behavior

Setuptools should pass -I/Users/david/work/junk/setuptools-test/.venv/include/site/python3.11 to the compiler instead of (or in addition to?) -I/Users/david/work/junk/setuptools-test/.venv/include.

How to Reproduce

1. git clone https://github.com/dlech/setuptools-venv-headers-dir-bug
2. cd setuptools-venv-headers-dir-bug
3. python3.11 -m venv .venv
4. . .venv bin activate
5. pip install setuptools setuptools-scm wheel build
6. python -m build example1
7. pip install example1/dist/example1-0.0.0-cp311-cp311-macosx_10_9_universal2.whl
8. python -m build example2 --no-isolation

Output

% python -m build example2 --no-isolation
* Getting build dependencies for sdist...
/Users/david/work/junk/setuptools-test/.venv/lib/python3.11/site-packages/setuptools/config/expand.py:144: UserWarning: File '/Users/david/work/junk/setuptools-test/example2/README.rst' cannot be found
  warnings.warn(f"File {path!r} cannot be found")
running egg_info
writing example2.egg-info/PKG-INFO
writing dependency_links to example2.egg-info/dependency_links.txt
writing top-level names to example2.egg-info/top_level.txt
listing git files failed - pretending there aren't any
reading manifest file 'example2.egg-info/SOURCES.txt'
writing manifest file 'example2.egg-info/SOURCES.txt'
* Building sdist...
/Users/david/work/junk/setuptools-test/.venv/lib/python3.11/site-packages/setuptools/config/expand.py:144: UserWarning: File '/Users/david/work/junk/setuptools-test/example2/README.rst' cannot be found
  warnings.warn(f"File {path!r} cannot be found")
running sdist
running egg_info
writing example2.egg-info/PKG-INFO
writing dependency_links to example2.egg-info/dependency_links.txt
writing top-level names to example2.egg-info/top_level.txt
listing git files failed - pretending there aren't any
reading manifest file 'example2.egg-info/SOURCES.txt'
writing manifest file 'example2.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md

running check
creating example2-0.0.0
creating example2-0.0.0/example2
creating example2-0.0.0/example2.egg-info
copying files to example2-0.0.0...
copying ext.c -> example2-0.0.0
copying pyproject.toml -> example2-0.0.0
copying setup.py -> example2-0.0.0
copying example2/__init__.py -> example2-0.0.0/example2
copying example2.egg-info/PKG-INFO -> example2-0.0.0/example2.egg-info
copying example2.egg-info/SOURCES.txt -> example2-0.0.0/example2.egg-info
copying example2.egg-info/dependency_links.txt -> example2-0.0.0/example2.egg-info
copying example2.egg-info/top_level.txt -> example2-0.0.0/example2.egg-info
Writing example2-0.0.0/setup.cfg
Creating tar archive
removing 'example2-0.0.0' (and everything under it)
* Building wheel from sdist
* Getting build dependencies for wheel...
/Users/david/work/junk/setuptools-test/.venv/lib/python3.11/site-packages/setuptools/config/expand.py:144: UserWarning: File '/private/var/folders/40/x9m96rt12mg6r515s8sqg4y00000gn/T/build-via-sdist-xj0yxc88/example2-0.0.0/README.rst' cannot be found
  warnings.warn(f"File {path!r} cannot be found")
running egg_info
writing example2.egg-info/PKG-INFO
writing dependency_links to example2.egg-info/dependency_links.txt
writing top-level names to example2.egg-info/top_level.txt
listing git files failed - pretending there aren't any
reading manifest file 'example2.egg-info/SOURCES.txt'
writing manifest file 'example2.egg-info/SOURCES.txt'
* Building wheel...
/Users/david/work/junk/setuptools-test/.venv/lib/python3.11/site-packages/setuptools/config/expand.py:144: UserWarning: File '/private/var/folders/40/x9m96rt12mg6r515s8sqg4y00000gn/T/build-via-sdist-xj0yxc88/example2-0.0.0/README.rst' cannot be found
  warnings.warn(f"File {path!r} cannot be found")
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.9-universal2-cpython-311
creating build/lib.macosx-10.9-universal2-cpython-311/example2
copying example2/__init__.py -> build/lib.macosx-10.9-universal2-cpython-311/example2
running egg_info
writing example2.egg-info/PKG-INFO
writing dependency_links to example2.egg-info/dependency_links.txt
writing top-level names to example2.egg-info/top_level.txt
listing git files failed - pretending there aren't any
reading manifest file 'example2.egg-info/SOURCES.txt'
writing manifest file 'example2.egg-info/SOURCES.txt'
running build_ext
building 'example2.ext' extension
creating build/temp.macosx-10.9-universal2-cpython-311
clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -I/Users/david/work/junk/setuptools-test/.venv/include -I/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c ext.c -o build/temp.macosx-10.9-universal2-cpython-311/ext.o
ext.c:3:10: fatal error: 'example1/ext.h' file not found
#include <example1/ext.h>
         ^~~~~~~~~~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit code 1

ERROR Backend subprocess exited when trying to invoke build_wheel

[dlech]

I came up with the following workaround:

import os
import sysconfig

from setuptools import Extension, setup

INCLUDE_DIRS = (
    [
        os.path.join(
            sysconfig.get_path("data"),
            "include",
            "site",
            f"python{sysconfig.get_python_version()}",
        )
    ]
    if "VIRTUAL_ENV" in os.environ
    else None
)

setup(
    ext_modules=[
        Extension(name="example2.ext", sources=["ext.c"], include_dirs=INCLUDE_DIRS),
    ]
)

[abravalheri]

Hi @dlech, I might be wrong, but I have the impression that this issue has to be first handled in pypa/disutils and then merged into pypa/setuptools. @jaraco might have more information on this.

[jaraco]

I think Anderson means pypa/distutils.

It's likely the issue is root caused in logic from distutils. I'll go ahead and move it there.

@dlech Are you interested and willing to dig into the code and explore a possible remedy?