Skip to content

Commit 9d0b63e

Browse files
fnikolaifnikolai
authored
improve database deployment and testing (#2055)
Signed-off-by: Fotis Nikolaidis <[email protected]>
1 parent 840ce54 commit 9d0b63e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+637
-129
lines changed

‎.gitattributes

-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,2 @@
1-
notebooks/* linguist-vendored
21
requirements/* linguist-vendored
32
Makefile linguist-vendored

‎.gitmodules

-3
This file was deleted.

‎.pre-commit-config.yaml

-7
This file was deleted.

‎CHANGELOG.md

+1
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
4848
- Optimized LLM finetuning usage experience
4949
- Auto-infer Schema from data
5050
- Lazy-creation of output tables for ibis to enable auto-inference of output schema
51+
- Add database packages that improve deployment and connection testing.
5152

5253
#### Bug Fixes
5354
- Fixed cross platfrom issue in cli command

‎Makefile

+30-21
Original file line numberDiff line numberDiff line change
@@ -56,10 +56,6 @@ new_release: ## Release a new version of SuperDuperDB
5656
git push --set-upstream origin release-$(RELEASE_VERSION) --tags
5757

5858
install-devkit: ## Add essential development tools
59-
# Add pre-commit hooks to ensure that no strange stuff are being committed.
60-
# https://stackoverflow.com/questions/3462955/putting-git-hooks-into-a-repository
61-
python -m pip install pre-commit
62-
6359
@echo "Download Docs dependencies"
6460
python -m pip install --user sphinx furo myst_parser
6561

@@ -178,6 +174,17 @@ testenv_init: ## Initialize a local Testing environment
178174
@echo "===> Waiting for TestEnv to become ready"
179175
@cd deploy/testenv/; ./wait_ready.sh
180176

177+
testenv_shutdown: ## Terminate the local Testing environment
178+
@echo "===> Shutting down the local Testing environment"
179+
docker compose -f deploy/testenv/docker-compose.yaml down
180+
181+
testenv_restart: testenv_shutdown testenv_init ## Restart the local Testing environment
182+
183+
184+
##@ Database Testing
185+
186+
# When the default paths are fixed, this function can be replaced with:
187+
# make testdb_init DB=mongodb
181188
testenv_init_mongodb: ## Initialize a local Testing environment
182189
@echo "===> Discover Hostnames"
183190
@deploy/testenv/validate_hostnames.sh
@@ -192,18 +199,27 @@ testenv_init_mongodb: ## Initialize a local Testing environment
192199
@echo "===> Run TestEnv MongoDB"
193200
docker compose -f deploy/testenv/docker-compose.yaml up --detach mongodb --remove-orphans &
194201

195-
testenv_shutdown: ## Terminate the local Testing environment
196-
@echo "===> Shutting down the local Testing environment"
197-
docker compose -f deploy/testenv/docker-compose.yaml down
198202

199-
testenv_restart: testenv_shutdown testenv_init ## Restart the local Testing environment
203+
## Helper function for starting database containers
204+
VALID_DATABASES := mongodb postgres mysql mssql azuresql clickhouse
205+
check_db_variable:
206+
@if [ -z "$(DB)" ]; then \
207+
echo "Error: 'DB' is not set."; \
208+
exit 1; \
209+
fi; \
210+
if ! echo "$(VALID_DATABASES)" | grep -qw "$(DB)"; then \
211+
echo "Error: '$(DB)' is not a valid database name. Valid options are: $(VALID_DATABASES)"; \
212+
exit 1; \
213+
fi
200214

201-
testdb_init: ## Initialize databases in Docker
202-
@mkdir -p deploy/testenv/cache && chmod -R 777 deploy/testenv/cache
203-
cd deploy/databases/; docker compose up --remove-orphans &
215+
testdb_init: check_db_variable ## Init Database Container (DB=<mongodb|postgres|mysql|mssql|azuresql|clickhouse>)
216+
@database_path="deploy/databases/$(DB)" && cd "$$database_path" && make init_db
204217

205-
testdb_shutdown: ## Terminate Databases Containers
206-
cd deploy/databases/; docker compose down
218+
testdb_connect: check_db_variable ## Init Database Container (DB=<mongodb|postgres|mysql|mssql|azuresql|clickhouse>)
219+
@database_path="deploy/databases/$(DB)" && cd "$$database_path" && make requirements && make run-example
220+
221+
testdb_shutdown: check_db_variable ## Shutdown Databases Containers (DB=<mongodb|postgres|mysql|mssql|azuresql|clickhouse>)
222+
@database_path="deploy/databases/$(DB)" && cd "$$database_path" && make shutdown_db
207223

208224
##@ CI Testing Functions
209225

@@ -224,11 +240,4 @@ ext-testing: ## Execute integration testing
224240
pytest $(PYTEST_ARGUMENTS) ./test/integration/ext
225241

226242
smoke-testing: ## Execute smoke testing
227-
SUPERDUPERDB_CONFIG=deploy/testenv/env/smoke/config.yaml pytest $(PYTEST_ARGUMENTS) ./test/smoke
228-
229-
test_notebooks: ## Test notebooks (argument: NOTEBOOKS=<test|dir>)
230-
@echo "Notebook Path: $(NOTEBOOKS)"
231-
232-
@if [ -n "$(NOTEBOOKS)" ]; then \
233-
pytest --nbval-lax $(NOTEBOOKS); \
234-
fi
243+
SUPERDUPERDB_CONFIG=deploy/testenv/env/smoke/config.yaml pytest $(PYTEST_ARGUMENTS) ./test/smoke

‎deploy/databases/.env

-4
This file was deleted.

‎deploy/databases/.gitignore

-1
This file was deleted.

‎deploy/databases/azuresql/Makefile

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
.DEFAULT_GOAL := help
2+
3+
help: ## Display this help
4+
@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m<target>\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf " \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)
5+
6+
7+
init_db: ## Init database on Docker
8+
docker compose up --remove-orphans &
9+
10+
shutdown_db: ## Init database on Docker
11+
docker compose down
12+
13+
14+
requirements: ## Install client requirements
15+
pip install -r requirements.txt
16+
17+
run-example: ## Run the example
18+
python example.py

‎deploy/databases/azuresql/README.md

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Azure SQL Server
2+
3+
4+
## Server Installation Instructions
5+
6+
To begin using the database server, execute the following command:
7+
```
8+
make init-db
9+
```
10+
11+
To halt the database server, use the following command:
12+
```bash
13+
make shutdown-db
14+
```
15+
16+
## Client Execution Guidelines
17+
18+
The `example.py` serves as a basic client script for establishing a connection with the database.
19+
20+
As certain databases may necessitate additional libraries, it is imperative to install these prerequisites before utilizing the client.
21+
22+
Execute the subsequent command to install the required dependencies:
23+
24+
```bash
25+
make requirements
26+
```
27+
28+
Subsequently, run the client by executing:
29+
30+
31+
```bash
32+
make run-example
33+
```
34+
35+
36+
37+
# Known Issues
38+
39+
**Question:** I'm getting the following issue: `Error initializing to DataBackend Client: libodbc.
40+
so.2: cannot open shared object file: No such file or directory`
41+
42+
**Answer:** You need to install `unixODBC` package on your system.

‎deploy/databases/azuresql/docker-compose.yaml

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
version: "3.9"
2+
services:
3+
4+
# Azure SQL Server
5+
# ------------------------------
6+
azuresql:
7+
image: mcr.microsoft.com/azure-sql-edge
8+
ports:
9+
- "1433:1433"
10+
environment:
11+
ACCEPT_EULA: "Y"
12+
SA_PASSWORD: "Superduper#1"

‎deploy/databases/azuresql/example.py

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
from superduperdb import superduper
2+
3+
user = 'sa'
4+
password = 'Superduper#1'
5+
port = 1433
6+
host = 'localhost'
7+
8+
db = superduper(f"mssql://{user}:{password}@{host}:{port}")

‎deploy/databases/azuresql/requirements.txt

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
pyodbc

‎deploy/databases/clickhouse/Makefile

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
.DEFAULT_GOAL := help
2+
3+
help: ## Display this help
4+
@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m<target>\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf " \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)
5+
6+
7+
init_db: ## Init database on Docker
8+
docker compose up --remove-orphans &
9+
10+
shutdown_db: ## Init database on Docker
11+
docker compose down
12+
13+
14+
requirements: ## Install client requirements
15+
pip install -r requirements.txt
16+
17+
run-example: ## Run the example
18+
python example.py

‎deploy/databases/clickhouse/README.md

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Clickhouse SQL Server
2+
3+
4+
## Server Installation Instructions
5+
6+
To begin using the database server, execute the following command:
7+
```
8+
make init-db
9+
```
10+
11+
To halt the database server, use the following command:
12+
```bash
13+
make shutdown-db
14+
```
15+
16+
## Client Execution Guidelines
17+
18+
The `example.py` serves as a basic client script for establishing a connection with the database.
19+
20+
As certain databases may necessitate additional libraries, it is imperative to install these prerequisites before utilizing the client.
21+
22+
Execute the subsequent command to install the required dependencies:
23+
24+
```bash
25+
make requirements
26+
```
27+
28+
Subsequently, run the client by executing:
29+
30+
31+
```bash
32+
make run-example
33+
```
34+
35+
36+
37+
# Known Issues
38+
39+
**Question:** I'm getting the following issue: `Error initializing to DataBackend Client: libodbc.
40+
so.2: cannot open shared object file: No such file or directory`
41+
42+
**Answer:** You need to install `unixODBC` package on your system.

‎deploy/databases/clickhouse/docker-compose.yaml

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
version: "3.9"
2+
services:
3+
4+
5+
# Clickhouse
6+
# ------------------------------
7+
clickhouse:
8+
image: clickhouse/clickhouse-server:latest
9+
ports:
10+
- "8123:8123"

‎deploy/databases/clickhouse/example.py

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
from superduperdb import superduper
2+
3+
user = 'default'
4+
password = ''
5+
port = 8123
6+
host = 'localhost'
7+
8+
db = superduper(f"clickhouse://{user}:{password}@{host}:{port}", metadata_store=f'mongomock://meta')

‎deploy/databases/clickhouse/requirements.txt

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
pyodbc

‎deploy/databases/docker-compose-macos.yaml

-43
This file was deleted.

‎deploy/databases/docker-compose.yaml

-43
This file was deleted.

0 commit comments

Comments
 (0)