Skip to content

Commit 67e9370

Browse files
michael-groundlightmichael-groundlight
authored
Basic python SDK (#2)
* A small wrapper layer around the auto-generated sdk * Update api * Fix url * Change to pip/poetry install * Add github action for publishing on releases * Use v0.1.2-rc1 to test publishing * 0.1.2 * Add instructions on releases * Improve readme * simple_example * readme * readme * more readme * Apply patch; add tests * Use alternative python * files * Better client with datamodel and working submit_image_query * IMprove interface with types * Examples * Better test-local and test-integ * Add github action to run integration tests * Add pillow * Bump to version 0.1.3 * Some cleanup * Address feedback
1 parent d7d005c commit 67e9370

File tree

13 files changed

+526
-1
lines changed

13 files changed

+526
-1
lines changed

‎.github/workflows/publish.yaml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# When a release is created on github, publish the groundlight package to our internal pypi repository
2+
# (Similar to https://github.com/positronix-ai/predictors/blob/main/.github/workflows/publish.yaml)
3+
name: publish package
4+
on:
5+
release:
6+
types: [created]
7+
jobs:
8+
publish-python-package:
9+
runs-on: ubuntu-latest
10+
env:
11+
INTERNAL_REPO_URL: https://positronix-723181461334.d.codeartifact.us-west-2.amazonaws.com/pypi/internal/
12+
steps:
13+
- name: install python
14+
uses: actions/setup-python@v2
15+
with:
16+
python-version: 3.9
17+
- name: install poetry
18+
run: |
19+
pip install -U pip
20+
pip install poetry
21+
- name: configure AWS credentials
22+
uses: aws-actions/configure-aws-credentials@v1
23+
with:
24+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
25+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
26+
aws-region: us-west-2
27+
- name: get code
28+
uses: actions/checkout@v2
29+
- name: build package
30+
run: poetry build
31+
- name: configure poetry and publish
32+
run: |
33+
export POETRY_HTTP_BASIC_INTERNAL_USERNAME=aws
34+
export POETRY_HTTP_BASIC_INTERNAL_PASSWORD=$(aws codeartifact get-authorization-token --domain positronix --query authorizationToken --output text)
35+
poetry config repositories.internal $INTERNAL_REPO_URL
36+
poetry publish -r internal

‎.github/workflows/test-integ.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Run integration tests against the integ API endpoint
2+
name: test integ
3+
on: [push]
4+
jobs:
5+
run-tests:
6+
runs-on: ubuntu-latest
7+
env:
8+
# This is associated with the "sdk-integ-test" user, credentials on 1password
9+
GROUNDLIGHT_API_TOKEN: ${{ secrets.GROUNDLIGHT_API_TOKEN }}
10+
steps:
11+
- name: get code
12+
uses: actions/checkout@v2
13+
- name: install python
14+
uses: actions/setup-python@v2
15+
with:
16+
python-version: 3.9
17+
- name: install poetry
18+
run: |
19+
pip install -U pip
20+
pip install poetry
21+
poetry install
22+
- name: run tests
23+
run: make test-integ

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,3 +158,6 @@ cython_debug/
158158
# and can be added to the global gitignore or merged into this file. For a more nuclear
159159
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
160160
#.idea/
161+
162+
# This is a library, so we want clients to have more flexible dependencies. So we don't include a lockfile.
163+
poetry.lock

‎Makefile‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,20 @@
1+
.PHONY: all test clean
2+
3+
install: ## Install the package from source
4+
poetry install
5+
16
# Java weirdness - see https://github.com/OpenAPITools/openapi-generator/issues/11763#issuecomment-1098337960
2-
generate: ## Generate the SDK from our public openapi spec
7+
generate: install ## Generate the SDK from our public openapi spec
38
_JAVA_OPTIONS="--add-opens=java.base/java.lang=ALL-UNNAMED \
49
--add-opens=java.base/java.util=ALL-UNNAMED" \
510
openapi-generator generate -i spec/public-api.yaml \
611
-g python \
712
-o ./generated
813
poetry run datamodel-codegen --input spec/public-api.yaml --output generated/model.py
14+
15+
test-local: install ## Run integration tests against an API server running at http://localhost:8000/device-api (needs GROUNDLIGHT_API_TOKEN)
16+
GROUNDLIGHT_TEST_API_ENDPOINT="http://localhost:8000/device-api" poetry run pytest --cov=src test --log-cli-level INFO
17+
18+
19+
test-integ: install ## Run integration tests against the integ API server (needs GROUNDLIGHT_API_TOKEN)
20+
GROUNDLIGHT_TEST_API_ENDPOINT="https://device.integ.positronix.ai/device-api" poetry run pytest --cov=src test --log-cli-level INFO

‎README.md‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,117 @@
11
# Groundlight Python SDK
2+
3+
This package holds an SDK for accessing the Groundlight public API.
4+
5+
### Installation
6+
7+
[Make sure you have internal pypi credentials set up](https://github.com/positronix-ai/tools/blob/main/internal-pip-login.sh), and then install with `pip` or `poetry`.
8+
9+
```Bash
10+
# pip
11+
$ pip install groundlight
12+
13+
# poetry
14+
$ poetry add groundlight
15+
```
16+
17+
### Basic Usage
18+
19+
To access the API, you need an API token. You can create one at [app.groundlight.ai](https://app.positronix.ai/reef/my-account/api-tokens). Then, add it as an environment variable called `GROUNDLIGHT_API_TOKEN`:
20+
21+
```Bash
22+
$ export GROUNDLIGHT_API_TOKEN=tok_abc123
23+
```
24+
25+
Now you can use the python SDK!
26+
27+
```Python
28+
from groundlight import Groundlight
29+
30+
# Load the API client. This defaults to the prod endpoint,
31+
# but you can specify a different endpoint like so:
32+
# gl = Groundlight(endpoint="https://device.integ.positronix.ai/device-api")
33+
gl = Groundlight()
34+
35+
# Call an API method (e.g., retrieve a list of detectors)
36+
detectors = gl.list_detectors()
37+
```
38+
39+
### What API methods are available?
40+
41+
Check out the [User Guide](UserGuide.md)!
42+
43+
For more details, see the [Groundlight](src/groundlight/client.py) class. This SDK closely follows the methods in our [API Docs](https://app.positronix.ai/reef/admin/api-docs).
44+
45+
46+
## Development
47+
48+
The auto-generated SDK code is in the `generated/` directory. To re-generate the client code, you'll need to install [openapi-generator](https://openapi-generator.tech/docs/installation#homebrew) (I recommend homebrew if you're on a mac). Then you can run it with:
49+
50+
```Bash
51+
$ make generate
52+
```
53+
54+
## Testing
55+
56+
Most tests need an API endpoint to run.
57+
58+
### Local API endpoint
59+
60+
1. Set up a local [janzu API endpoint](https://github.com/positronix-ai/zuuul/blob/main/deploy/README.md#development-using-local-microk8s) running (e.g., on an AWS GPU instance).
61+
62+
1. Set up an ssh tunnel to your laptop. That way, you can access the endpoint at http://localhost:8000/device-api (and the web UI at http://localhost:8000/reef):
63+
64+
```Bash
65+
$ ssh instance-name -L 8000:localhost:80
66+
```
67+
68+
1. Run the tests (with an API token)
69+
70+
```Bash
71+
$ export GROUNDLIGHT_API_TOKEN=tok_abc123
72+
$ make test-local
73+
```
74+
75+
(Note: in theory, it's possible to run the janzu API server on your laptop without microk8s - but some API methods don't work because of the dependence on GPUs)
76+
77+
### Integ API endpoint
78+
79+
1. Run the tests (with an API token)
80+
81+
```Bash
82+
$ export GROUNDLIGHT_API_TOKEN=tok_abc123
83+
$ make test-integ
84+
```
85+
86+
## Releases
87+
88+
To publish a new package version to our [internal pypi repository](https://github.com/positronix-ai/packaging/tree/main/aws), you create a release on github.
89+
90+
```Bash
91+
# Create a git tag locally. Use semver "vX.Y.Z" format.
92+
$ git tag -a v0.1.2 -m "Short description"
93+
94+
# Push the tag to the github repo
95+
$ git push origin --tags
96+
```
97+
98+
Then, go to the [github repo](https://github.com/positronix-ai/groundlight-python-sdk/tags) -> choose your tag -> create a release from this tag -> type in some description -> release. A [github action](https://github.com/positronix-ai/groundlight-python-sdk/actions/workflows/publish.yaml) will trigger a release, and then `groundlight-X.Y.Z` will be available for consumers.
99+
100+
## TODOs
101+
102+
- Figure out how we want to handle tests (since almost everything is an integration test). And, running the stateful (creation) tests can lead to a bunch of objects in the DB.
103+
- Improve wrappers around API functions (e.g., simplify the responses even further, add auto-pagination managers, etc.)
104+
- The SDK should allow you to work with the most natural interface, rather than trying to exactly mirror the REST API.
105+
- E.g.
106+
- Add an image query long polling helper method (calls POST, then several GETs)
107+
- It would be nice to have a `get_or_create_detector()` function (even better if it's supported in the API directly). That way, "submit image query" code examples will be simpler.
108+
- Better auto-generated code docs (e.g. [sphinx](https://www.sphinx-doc.org/en/master/))
109+
- Model types (e.g., [autodoc_pydantic](https://github.com/mansenfranzen/autodoc_pydantic))
110+
- Cleaner auto-generated model names (e.g., `PaginatedDetectorList` is a little ugly)
111+
- Better versioning strategy. On the one hand, this package will closely follow the versioning in the HTTP API. On the other hand, we may add features in the client (like image utils, shortcuts, etc.) that are not in the REST API.
112+
- Better way of managing dependency on `public-api.yaml` OpenAPI spec (right now, we just copy the file over manually)
113+
- Update the web links (links to website, link to API endpoint, etc.)
114+
- `with` context manager (auto cleanup the client object)
115+
- It would be great to add notebooks with interactive examples that can actually run out of the box
116+
- Have a cleaner distinction between dev docs and user guide docs
117+
- ...

‎UserGuide.md‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# User Guide
2+
3+
## Pre-reqs
4+
5+
For all the examples, there are 3 pre-reqs:
6+
7+
1. [Make sure you have internal pypi credentials set up](https://github.com/positronix-ai/tools/blob/main/internal-pip-login.sh), and then install with `pip` or `poetry`.
8+
9+
```Bash
10+
# pip
11+
$ pip install groundlight
12+
13+
# poetry
14+
$ poetry add groundlight
15+
```
16+
17+
1. To access the API, you need an API token. You can create one at [app.groundlight.ai](https://app.positronix.ai/reef/my-account/api-tokens). Then, add it as an environment variable called `GROUNDLIGHT_API_TOKEN`:
18+
19+
```Bash
20+
$ export GROUNDLIGHT_API_TOKEN=tok_abc123
21+
```
22+
23+
1. Create the `Groundlight` API client. We usually use `gl` as a shorthand name, but you are free to name it what you like!
24+
25+
```Python
26+
from groundlight import Groundlight
27+
gl = Groundlight()
28+
```
29+
## Basics
30+
31+
#### Create a new detector
32+
33+
```Python
34+
detector = gl.create_detector(name="Dog", query="Is it a dog?")
35+
```
36+
37+
#### Retrieve a detector
38+
39+
```Python
40+
detector = gl.get_detector(id="YOUR_DETECTOR_ID")
41+
```
42+
43+
#### List your detectors
44+
45+
```Python
46+
# Defaults to 10 results per page
47+
detectors = gl.list_detectors()
48+
49+
# Pagination: 3rd page of 25 results per page
50+
detectors = gl.list_detectors(page=3, page_size=25)
51+
```
52+
53+
#### Submit an image query
54+
55+
```Python
56+
image_query = gl.submit_image_query(detector_id="YOUR_DETECTOR_ID", image="path/to/filename.jpeg")
57+
```
58+
59+
#### Retrieve an image query
60+
61+
In practice, you may want to check for a new result on your query. For example, after a cloud reviewer labels your query. For example, you can use the `image_query.id` after the above `submit_image_query()` call.
62+
63+
```Python
64+
image_query = gl.get_image_query(id="YOUR_IMAGE_QUERY_ID")
65+
```
66+
67+
#### List your previous image queries
68+
69+
```Python
70+
# Defaults to 10 results per page
71+
image_queries = gl.list_image_queries()
72+
73+
# Pagination: 3rd page of 25 results per page
74+
image_queries = gl.list_image_queries(page=3, page_size=25)
75+
```
76+
77+
## Advanced
78+
79+
#### Use a different API endpoint
80+
81+
```Python
82+
from groundlight import Groundlight
83+
84+
# Integ
85+
integ_gl = Groundlight(endpoint="https://device.integ.positronix.ai/device-api")
86+
87+
# Local
88+
local_gl = Groundlight(endpoint="http://localhost:8000/device-api")
89+
```
90+
91+
#### Do more with the object models
92+
93+
You can see the different model types [here](generated/model.py). (TODO: Use something like [autodoc_pydantic](https://github.com/mansenfranzen/autodoc_pydantic) to create docs).
94+
95+
All of the `Groundlight` methods return [pydantic](https://pydantic-docs.helpmanual.io/) models - `Detector`, `ImageQuery`, `PaginatedDetectorList`, etc. This provides several benefits: you can access model fields with dot notation, get auto-complete in your IDE, have `model.dict()`, `model.json()`, `model.pickle()` serializers, etc. See more on the [pydantic docs](https://pydantic-docs.helpmanual.io/usage/models/).
96+
97+
### Handling HTTP errors
98+
99+
If there is an HTTP error during an API call, it will raise an `ApiException`. You can access different metadata from that exception:
100+
101+
```Python
102+
from groundlight import ApiException, Groundlight
103+
104+
gl = Groundlight()
105+
try:
106+
detectors = gl.list_detectors()
107+
except ApiException as e:
108+
print(e)
109+
print(e.args)
110+
print(e.body)
111+
print(e.headers)
112+
print(e.reason)
113+
print(e.status)
114+
```

‎bin/simple_example.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
from groundlight import Groundlight
2+
3+
gl = Groundlight()
4+
detectors = gl.list_detectors()
5+
print(f"Found {detectors.count} detectors")

‎pyproject.toml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
[tool.poetry]
2+
name = "groundlight"
3+
version = "0.1.4"
4+
description = "Call the Groundlight API from python"
5+
authors = ["Groundlight AI <[email protected]>"]
6+
packages = [
7+
{ include = "**/*.py", from = "src" },
8+
{ include = "**/*.py", from = "generated" },
9+
]
10+
11+
[tool.poetry.dependencies]
12+
python = "^3.9"
13+
python-dateutil = "^2.8.2"
14+
urllib3 = "^1.26.9"
15+
frozendict = "^2.3.2"
16+
certifi = "^2021.10.8"
17+
pydantic = "^1.9.0"
18+
19+
[tool.poetry.dev-dependencies]
20+
pytest-cov = "^3.0.0"
21+
pylint = "^2.13.8"
22+
black = "^22.3.0"
23+
flake8 = "^4.0.1"
24+
mypy = "^0.950"
25+
pytest = "^7.1.2"
26+
isort = "^5.10.1"
27+
autoflake = "^1.4"
28+
datamodel-code-generator = "^0.12.0"
29+
30+
[build-system]
31+
requires = ["poetry-core>=1.0.0"]
32+
build-backend = "poetry.core.masonry.api"

‎src/groundlight/__init__.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# flake8: noqa
2+
# Add useful imports from the generated code here at the top level, as a convenience.
3+
from openapi_client import ApiException
4+
5+
# Imports from our code
6+
from .client import Groundlight

0 commit comments

Comments
 (0)