turbodbc does not play well with poetry

[bgemmill]

When configuring a python project with poetry, issuing the command poetry add turbodbc fails with this error:

Using version ^4.7.0 for turbodbc

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing turbodbc (4.7.0): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  Traceback (most recent call last):
...much backtrace spam goes here...
  ModuleNotFoundError: No module named 'pybind11'
  
Note: This error originates from the build backend, and is likely not a problem with poetry but with turbodbc (4.7.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "turbodbc (==4.7.0)"'.

Poetry seems right that this has something to do with turbodbc, in that running pip wheel --no-cache-dir --use-pep517 "turbodbc (==4.7.0)" produces the same error.

A straight pip install turbodbc does appear to work, where a wheel is compiled locally and installed.

Is this a pep517 thing, a lack of prebuilt wheel for ubuntu 22.04, or something else?

I'm happy to provide any sort of help to make this go; it would be great to get this into poetry.

[rxm7706]

cc @xhochy @cshaley

Hi @bgemmill
Thats an issue I was running into too - I believe the PR that just got merged #401 will address this - Version 4.8.0 when released should be installable via pip, poetry and eventually conda too.

[rxm7706]

@bgemmill will you be able to test this -
I am getting still getting an error with Poetry & TurbODBC 4.9.0
If you are getting an error too - We should open a new issues

 creating build/temp.linux-x86_64-cpython-310/src/turbodbc/result_sets
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/column_description.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/column_description.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/connection.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/connection.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/cpp_odbc.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/cpp_odbc.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/environment.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/environment.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/error.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/error.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level1/api.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level1/api.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level1/level1.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level1/level1.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level1/unixodbc_backend.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level1/unixodbc_backend.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level1/unixodbc_backend_debug.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level1/unixodbc_backend_debug.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level2/api.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level2/api.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level2/handles.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level2/handles.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level2/input_string_buffer.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level2/input_string_buffer.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level2/input_u16string_buffer.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level2/input_u16string_buffer.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level2/level1_connector.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level2/level1_connector.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level2/level2.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level2/level2.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level2/string_buffer.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level2/string_buffer.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level2/u16string_buffer.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level2/u16string_buffer.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level3/level3.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level3/level3.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level3/raii_connection.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level3/raii_connection.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level3/raii_environment.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level3/raii_environment.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/level3/raii_statement.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/level3/raii_statement.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/make_environment.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/make_environment.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/multi_value_buffer.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/multi_value_buffer.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/cpp_odbc/statement.cpp -o build/temp.linux-x86_64-cpython-310/src/cpp_odbc/statement.o --std=c++17
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iinclude/ -I/tmp/tmpok_bmpw7/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpok_bmpw7/.venv/include -I/opt/hostedtoolcache/Python/3.10.13/x64/include/python3.10 -c src/turbodbc/buffer_size.cpp -o build/temp.linux-x86_64-cpython-310/src/turbodbc/buffer_size.o --std=c++17
  In file included from include/turbodbc/description.h:3,
                   from include/turbodbc/buffer_size.h:3,
                   from src/turbodbc/buffer_size.cpp:1:
  include/turbodbc/field.h:3:10: fatal error: boost/variant/variant.hpp: No such file or directory
      3 | #include <boost/variant/variant.hpp>
        |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/gcc' failed with exit code 1
  

  at /opt/pipx/venvs/poetry/lib/python3.10/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with turbodbc (4.9.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "turbodbc (==4.9.0)"'

[xhochy]

No, this error only signals that you have not installed the native dependencies required for building turbodbc on your system.

[rxm7706]

No, this error only signals that you have not installed the native dependencies required for building turbodbc on your system.

Ah makes sense -- do we have a list of these dependencies.. Can you point me that list.

[rxm7706]

I think I found what I was looking for - mostly..
#351 (comment)
https://turbodbc.readthedocs.io/en/latest/pages/contributing.html
https://turbodbc.readthedocs.io/en/latest/pages/contributing.html

# Install poetry package requirements
COPY poetry.lock pyproject.toml ./
RUN poetry install

# Hacky install turbodbc
RUN poetry run python -c "import pyarrow; pyarrow.create_library_symlinks()"
RUN CFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0" poetry run pip install --no-cache-dir turbodbc

[bgemmill]

Dependencies are here

But if you have a local install of boost, it gets tricky with poetry and you may run into this python-poetry/poetry#8727