Skip to content

Enable using clickhouse HTTP interface #319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pofl wants to merge 19 commits into
base: main
Choose a base branch
from
Open

Enable using clickhouse HTTP interface #319

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
4d2b49e
Add clickhouse-connect dependency
pofl Mar 12, 2023
c9a2dd0
Enable using HTTP port
pofl Mar 12, 2023
9a3c217
Adjust doctest
pofl Mar 12, 2023
de8502a
Repair import
pofl Mar 12, 2023
564ce66
Handle all connect errors
pofl Mar 12, 2023
cc4e563
Reformat doctest
pofl Mar 12, 2023
8682063
Revert some accidental changes
pofl Mar 12, 2023
04e999c
Add type
pofl Mar 12, 2023
4554473
Satisfy linter
pofl Mar 24, 2023
9008da5
Improve docstring
pofl Mar 24, 2023
b50984d
Update clickhouse/testcontainers/clickhouse/__init__.py
pofl Mar 24, 2023
21a7f00
Repair doctest
pofl Mar 24, 2023
c1761e2
Trim whitespace
pofl Mar 24, 2023
ac47799
Merge remote-tracking branch 'upstream/master' into patch-1
pofl Apr 12, 2023
66ccd27
Fix merge
pofl Apr 12, 2023
4932238
Add test for HTTP
pofl Apr 12, 2023
b851c6e
Merge branch 'master' into patch-1
pofl Apr 13, 2023
8e097c6
Merge remote-tracking branch 'upstream/main' into patch-1
pofl May 8, 2023
e06bbdd
Update requirements
pofl May 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions clickhouse/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
url="https://github.com/testcontainers/testcontainers-python",
install_requires=[
"testcontainers-core",
"clickhouse-connect",
Copy link
Contributor

@tillahoffmann tillahoffmann Apr 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the updated requirements, let's update the lock files. Could you run make requirements from the root directory and push the changes?

Copy link
Author

@pofl pofl May 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see there have been some changes to requirements management. Is there anything I should do?

Copy link
Contributor

@tillahoffmann tillahoffmann May 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can run python get_requirements.py --pr=[your PR number] to update the requirements. Full details here. You may have to rebase on main and wait for CI to run before the script will work.

Copy link
Author

@pofl pofl May 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am getting an error

$ python get_requirements.py --pr=319
we need a GitHub access token to fetch the requirements; please visit https://github.com/settings/tokens/new, create a token with `public_repo` scope, and paste it here: ----------------------------
do you want to cache the token in a `.github-token` file [Ny]? N
fetching most recent commit for PR #319
Traceback (most recent call last):
  File "/home/pofl/testcontainers-python/get_requirements.py", line 94, in <module>
    __main__()
  File "/home/pofl/testcontainers-python/get_requirements.py", line 65, in __main__
    raise RuntimeError(f"could not identify unique workflow run: {runs}")
RuntimeError: could not identify unique workflow run: []

Copy link
Contributor

@tillahoffmann tillahoffmann May 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the CI needs to have completed first, and GitHub doesn't allow us to start the run automatically for first-time contributors. I've started it now. We should probably also improve that error message.

"clickhouse-driver",
],
python_requires=">=3.7",
Expand Down
37 changes: 28 additions & 9 deletions clickhouse/testcontainers/clickhouse/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,31 +13,46 @@
import os
from typing import Optional

import clickhouse_connect
from clickhouse_connect.driver.exceptions import Error as ClickhouseConnectError
import clickhouse_driver
from clickhouse_driver.errors import Error
from clickhouse_driver.errors import Error as ClickhouseDriverError

from testcontainers.core.generic import DbContainer
from testcontainers.core.waiting_utils import wait_container_is_ready


class ClickHouseContainer(DbContainer):
"""
ClickHouse database container.
ClickHouse database container. This testcontainer defaults to exposing the TCP port of
ClickHouse. If you want to use the HTTP interface, specify port 8123 to be exposed.

Example:

The example spins up a ClickHouse database and connects to it using the
:code:`clickhouse-driver`.
This example shows how to spin up ClickHouse.
It demonstrates how to connect to the *TCP* interface using :code:`clickhouse-driver`
and how to connect to the *HTTP* interface using :code:`clickhouse-connect`, the
official client library.

.. doctest::

>>> import clickhouse_driver
>>> from testcontainers.clickhouse import ClickHouseContainer

>>> with ClickHouseContainer("clickhouse/clickhouse-server:21.8") as clickhouse:
>>> # clickhouse_driver is a client lib that uses the TCP interface
>>> import clickhouse_driver
>>> # ClickHouseContainer exports the TCP port by default
>>> with ClickHouseContainer(image="clickhouse/clickhouse-server:21.8") as clickhouse:
... client = clickhouse_driver.Client.from_url(clickhouse.get_connection_url())
... client.execute("select 'working'")
[('working',)]

>>> # clickhouse_connect is the official client lib, based on the HTTP interface
>>> import clickhouse_connect
>>> # If you want to use the HTTP interface, port 8123 needs to be exposed
>>> with ClickHouseContainer(port=8123) as clickhouse:
... client = clickhouse_connect.get_client(dsn=clickhouse.get_connection_url())
... client.query("select 'working'").result_rows
[('working',)]
"""

CLICKHOUSE_USER = os.environ.get("CLICKHOUSE_USER", "test")
Expand All @@ -60,10 +75,14 @@ def __init__(
self.port_to_expose = port
self.with_exposed_ports(self.port_to_expose)

@wait_container_is_ready(Error, EOFError)
@wait_container_is_ready(ClickhouseDriverError, ClickhouseConnectError, EOFError)
def _connect(self) -> None:
with clickhouse_driver.Client.from_url(self.get_connection_url()) as client:
client.execute("SELECT version()")
if self.port_to_expose == 8123:
with clickhouse_connect.get_client(dsn=self.get_connection_url()) as client:
client.command("SELECT version()")
else:
with clickhouse_driver.Client.from_url(self.get_connection_url()) as client:
client.execute("SELECT version()")

def _configure(self) -> None:
self.with_env("CLICKHOUSE_USER", self.CLICKHOUSE_USER)
Expand Down