Skip to content

Commit 2a9c1fa

Browse files
b-longgithub-actions[bot]gemini-code-assist[bot]
authored
feat!: rewrite in pure Python (#62)
* Begin rewrite in pure Python * Organize: git mv src/otdf_python/test_*.py tests/ * Format according to 'ruff' * Fix static analysis * Cleanup and organize tests/test_validate_otdf_python.py * Remove 'TDFConfig' type from 'otdf_python.tdf' * Fix description & formatting * Add 'pydantic-settings' to dev & update dependencies * Correct version number * Cleanup and fix OIDC tests * Comment old style integration test * Execute majority of tests * Allow import from 'tests' * Fix string encryption test * Remove dead code * Adjust integration test * Remove old build scripts * Update README * Update GHA triggers * Fix endpoint URL and TLS verification * ✅ Significant update 143 out of 150 tests passing - When run with the proper .env file: 7 failed, 142 passed, 2 skipped, 1 warning - Critical naming fix - Update .proto files - Add script to update .proto files - Ditch HTTP impl - Improve manifest and encrypt test - Python CLI decrypt now works correctly with TDF files created by otdfctl * Run all tests, except integration * Update GHA configuration * Mark integration tests * Fix mocked tests/test_kas_client.py * Mark integration tests * Only build for 3.13 (temporary) * Update license * Enable and fix integration tests in CI Cleanup tests * Improve support for plaintext * Make log collection optional * Fix tests for plaintext * Fix docstrings * Fix docstrings * Extract Connect RPC class * Fix additional roundtrip testing * Fix tests after kas_client updates * Expand KAS client integration tests * Fix mimeType * Expand testing, fix compression bug * Auto-use check_for_otdfctl fixture * Expand static analysis, fix FURB188 * Use 'NULL_POLICY_UUID' for now * Update kas_client.py & tdf.py, expand tests * Expand & organize integration tests * Expand static analysis, fix PT018 * Use configurable attrs in testing * Use configurable attrs in testing * Examine entitlements in CI * Extract 'temp_credentials_file' fixture * Rename file * Modernize release workflows * Modernize release workflows * Update release workflow * Manage 'otdf-python-proto' as a sub-package * Update README * Manage 'otdf-python-proto' as a sub-package * Support Python 3.10+ * Fix version number * Fix Python version requirement * Bump version 0.3.0a4 -> 0.3.0a5 * Fix version extract command * Undo file name change * More support for PE flows, cleanup & improved typing (#70) * Cleanup & improved typing * Disable odd policy enforcement * Add ".env-docker" file for local testing * Add PE test support (GHA and docker) (#71) * Add docker start script * Gemini fixes * Update GHA configuration * Gemini fixes * Enable PE e2e test * Run 'pre-commit autoupdate' & fix lint issues * Extract '_get_sdk_builder' function * Cleanup & remove redundant function * Improve typing * Use patch() context manager, reduce imports * Remove unnecessary import * Combine 'yq' expressions * Point to commit SHA * Remove hallucination * Match version number * Bump 0.3.0a5 to 0.3.0a6 * Chore/update docs and release process (#72) * Cleanup docs * Refine workflows for release management and testing - Implement `release-please` workflow for automated releases. - Create `publish-test` and `publish` workflows to handle package builds and releases. - Introduce `test-suite` workflow to run tests before publishing. - Update configuration files for release management. * Add 'ruff' as dev dependency * Configure ruff to ignore generated files * Fail fast if linting fails * Document release process * Bump version to 0.3.0a7 * Publish new alpha * Allow replacing artifacts with the same name * Remove the duplicate integration-test job * Attempt alpha release * chore: improve pre-commit configuration * chore: revert 'rm CONNECT_RPC_MIGRATION.md' * chore: disable TestPyPIBuild unless workflow_dispatch * chore: bump version 0.3.0a7 -> 0.3.0a8 * chore: bump version 0.3.0a8 -> 0.3.0a9 * chore: target this branch * chore: target develop branch * chore: fix release-please config * chore: fix version number * chore: use standard 'workflow_call' * chore: clean up publishing * fix: fix publishing * chore: release 0.3.0a10 Release-As: 0.3.0a10 * fix: fix publishing * chore: release 0.3.0a11 Release-As: 0.3.0a11 * chore: release develop (#81) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * chore: align version numbers * chore: add 'otdf-python-proto/uv.lock' file * chore: add 'otdf-python-proto/uv.lock' file * fix: omit README from Github releases * chore: document legacy version * fix: address pre-commit (lint) issues * chore: verbose output for pypi uploads * fix: use correct 'extra-files' for uv.lock See also: googleapis/release-please#2561 * chore: release 0.3.1 Release-As: 0.3.1 * chore: release develop (#82) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * chore: organize docs * fix: remove unnecessary 'ncipollo/release-action' * chore: add developer doc * chore: CI improvements (#88) * chore: prevent TestPyPI publishing <= 0.3.2 * chore: update .pre-commit-config.yaml * chore: align versions * chore: ensure future version alignment * chore: comment unused GHA step * chore: simplify version parsing * chore: add tomli for Python < 3.11 * fix: get version dynamically in 'test_cli.py' * fix: guarantee target-version decrypt support (#84) * fix: add test data * fix: improve target-version support * fix: add get_cli_flags function * fix: fix tests * fix: bug handling bytes | BinaryIO & tests * fix: update .gitignore * fix: remove invalid default KAS * fix: disable attrs for now * fix: DRY test fixtures * chore: cleanup * fix:target mode encryption (#86) * chore: update pre-commit * fix: type annotations in tdf.py * chore: expand inspect tests * chore: cleanup tests * chore: organize imports * chore: require sorted imports * chore: add test_cli_decrypt.py * chore: organize integration tests * chore: organize integration tests * Tweak attributes * chore: cleanup tests * chore: cleanup tests * chore: dry tests (#87) * chore: dry tests * chore: relocate run_cli_inspect * chore: fix type annotation * chore: note token isn't important * chore: cleanup args & typing * chore: extract 'get_platform_url' function * chore: extract 'support_otdfctl_args' module * chore: use '*get_cli_flags()' pattern * chore: DRY code * chore: DRY code * chore: extract 'get_testing_environ' function * chore: DRY code * chore: DRY code * chore: DRY code * chore: improve pre-commit config * fix: mirrored workflows for target-mode (#91) * chore: cleanup for mirrored workflows * chore: cleanup for mirrored workflows * chore: cleanup for mirrored workflows * chore: cleanup for mirrored workflows * chore: cleanup for mirrored workflows * chore: remove otdf-python-proto from manifest * chore: cleanup and release (#93) * fix: don't inspect without auth * fix: process otdf-python-proto/pyproject.toml correctly * chore: remove NanoTDF from README * chore: mention legacy version in main README * chore: set version to 0.3.1 * chore: fix release-please * fix: release-please configuration (#95) * fix: "jsonpath" in release-please-config.json * chore: remove invalid changelog entries * chore: cleanup branches used in release-please * chore: remove invalid changelog file * chore: reset version to 0.3.0 * chore: cleanup whitespace * chore: improve release process * chore: document release process * chore: delete invalid information * fix: update prerelease config for develop branch * chore(develop): release otdf-python 0.3.1 (#96) * chore(develop): release otdf-python 0.3.1 * Update CHANGELOG.md --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: b-long <[email protected]> * fix: fix .release-please-config.json file (#97) * fix: fix .release-please-config.json file * chore: align for version 0.3.1 * chore: use importlib for version * chore: manage .py files without relese-please * fix: allow for development version in CLI version test * Update src/otdf_python/cli.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * chore(develop): release otdf-python 0.3.2 (#98) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix: release configuration (#99) * chore: fix release-please config * chore: remove invalid changelog entries * chore: roll back to 0.3.0 * fix: add develop-specific release-please files and update workflow - Add .release-please-config-develop.json with prerelease: true - Add .release-please-manifest-develop.json with current version - Remove dynamic file creation from workflow - Files are now committed to repo instead of generated at runtime * chore(develop): release otdf-python 0.3.1 (#100) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
1 parent 439becd commit 2a9c1fa

File tree

267 files changed

+36937
-3334
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

267 files changed

+36937
-3334
lines changed

.env-docker

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
OPENTDF_PLATFORM_HOST="localhost"
2+
OPENTDF_PLATFORM_PORT=8080
3+
OPENTDF_PLATFORM_URL="http://localhost:8080"
4+
5+
KEYCLOAK_URL="http://localhost:8888/auth"
6+
OIDC_OP_TOKEN_ENDPOINT="http://localhost:8888/auth/realms/opentdf/protocol/openid-connect/token"

.github/check_entitlements.sh

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
#!/bin/bash
2+
3+
# Derive additional environment variables
4+
TOKEN_URL="${OIDC_OP_TOKEN_ENDPOINT}"
5+
OTDF_HOST_AND_PORT="${OPENTDF_PLATFORM_HOST}"
6+
OTDF_CLIENT="${OPENTDF_CLIENT_ID}"
7+
OTDF_CLIENT_SECRET="${OPENTDF_CLIENT_SECRET}"
8+
9+
echo "🔧 Environment Configuration:"
10+
echo " TOKEN_URL: ${TOKEN_URL}"
11+
echo " OTDF_HOST_AND_PORT: ${OTDF_HOST_AND_PORT}"
12+
echo " OTDF_CLIENT: ${OTDF_CLIENT}"
13+
echo " OTDF_CLIENT_SECRET: ${OTDF_CLIENT_SECRET}"
14+
echo ""
15+
16+
get_token() {
17+
curl -k --location "$TOKEN_URL" \
18+
--header "X-VirtruPubKey;" \
19+
--header "Content-Type: application/x-www-form-urlencoded" \
20+
--data-urlencode "grant_type=client_credentials" \
21+
--data-urlencode "client_id=$OTDF_CLIENT" \
22+
--data-urlencode "client_secret=$OTDF_CLIENT_SECRET"
23+
}
24+
25+
echo "🔐 Getting access token..."
26+
BEARER=$( get_token | jq -r '.access_token' )
27+
# NOTE: It's always okay to print this token, because it will
28+
# only be valid / available in dummy / dev scenarios
29+
[[ "${DEBUG:-}" == "1" ]] && echo "Got Access Token: ${BEARER}"
30+
echo ""
31+
32+
# Array of usernames to check
33+
USERNAMES=("opentdf" "sample-user" "sample-user-1" "cli-client" "opentdf-sdk")
34+
35+
for USERNAME in "${USERNAMES[@]}"; do
36+
echo "👤 Fetching entitlements for username: ${USERNAME}"
37+
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
38+
39+
grpcurl -plaintext \
40+
-H "authorization: Bearer $BEARER" \
41+
-d "{
42+
\"entities\": [
43+
{
44+
\"userName\": \"$USERNAME\"
45+
}
46+
]
47+
}" \
48+
"$OTDF_HOST_AND_PORT" \
49+
authorization.AuthorizationService/GetEntitlements
50+
51+
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
52+
echo "✅ Entitlements retrieval complete for ${USERNAME}!"
53+
echo ""
54+
done
55+
56+
echo "🎉 All entitlement checks completed!"

.github/start_opentdf_docker.sh

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
#!/usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
if ! [ -d platform ]; then
6+
git clone https://github.com/opentdf/platform.git
7+
fi
8+
cd platform
9+
git checkout 3360befcb3e6e9791d7bfd2e89128aee0e7d2818 # Branch 'DSPX-1539-keytoolnomore'
10+
11+
yq -i '.realms[0].clients[0].client.directAccessGrantsEnabled = true | .realms[0].clients[0].client.serviceAccountsEnabled = true' service/cmd/keycloak_data.yaml
12+
13+
yq -i '.realms[0].clients[1].client.directAccessGrantsEnabled = true | .realms[0].clients[1].client.serviceAccountsEnabled = true' service/cmd/keycloak_data.yaml
14+
15+
yq -i '.realms[0].clients[4].client.directAccessGrantsEnabled = true | .realms[0].clients[4].client.serviceAccountsEnabled = true' service/cmd/keycloak_data.yaml
16+
17+
18+
if ! [ -d ./keys ]; then
19+
go mod download
20+
21+
go mod verify
22+
23+
.github/scripts/init-temp-keys.sh
24+
cp opentdf-example.yaml opentdf.yaml
25+
26+
# Edit 'opentdf.yaml' for our use case
27+
yq -i 'del(.db) | .services.entityresolution.url = "http://localhost:8888/auth" | .server.auth.issuer = "http://localhost:8888/auth/realms/opentdf"' opentdf.yaml
28+
# The above expression can also be written as 3 separate commands:
29+
# yq -i 'del(.db)' opentdf.yaml
30+
# yq -i '.services.entityresolution.url = "http://localhost:8888/auth"' opentdf.yaml
31+
# yq -i '.server.auth.issuer = "http://localhost:8888/auth/realms/opentdf"' opentdf.yaml
32+
33+
yq -i '
34+
.server.cryptoProvider = {
35+
"type": "standard",
36+
"standard": {
37+
"keys": [
38+
{
39+
"kid": "r1",
40+
"alg": "rsa:2048",
41+
"private": "kas-private.pem",
42+
"cert": "kas-cert.pem"
43+
},
44+
{
45+
"kid": "e1",
46+
"alg": "ec:secp256r1",
47+
"private": "kas-ec-private.pem",
48+
"cert": "kas-ec-cert.pem"
49+
}
50+
]
51+
}
52+
}
53+
' opentdf.yaml
54+
chmod -R 700 ./keys
55+
fi
56+
57+
docker compose up -d --wait --wait-timeout 360
58+
59+
go run ./service provision keycloak
60+
61+
go run ./service provision fixtures

.github/workflows/build-golang-macos.yaml

Lines changed: 0 additions & 50 deletions
This file was deleted.

.github/workflows/build-golang-ubuntu.yaml

Lines changed: 0 additions & 65 deletions
This file was deleted.
Lines changed: 35 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1,44 +1,42 @@
1-
---
2-
name: Build Python package(s)
3-
1+
# Build otdf-python wheel using uv and output the wheel path for downstream workflows
2+
name: "Build Python Wheel"
43
on:
5-
push:
6-
branches:
7-
- disabled
4+
push:
5+
branches:
6+
- chore/rewrite
7+
pull_request:
88

99
jobs:
10-
build:
11-
12-
runs-on: ubuntu-22.04
13-
strategy:
14-
matrix:
15-
go-version: [1.24.x]
10+
build:
11+
runs-on: ubuntu-22.04
12+
outputs:
13+
wheel: ${{ steps.find_wheel.outputs.wheel_path }}
14+
steps:
15+
- name: Checkout this repo
16+
uses: actions/checkout@v4
1617

17-
steps:
18-
- uses: actions/checkout@v4
19-
# - name: Setup Go
20-
# uses: actions/setup-go@v4
21-
# with:
22-
# go-version: ${{ matrix.go-version }}
23-
# cache-dependency-path: go.sum
24-
# - name: Install dependencies
25-
# run: go get .
26-
# - name: Test with Go
27-
# run: go test -timeout 40s -run ^TestHello$ gotdf_python -count=1 # go test
18+
- name: Set up uv
19+
uses: astral-sh/setup-uv@v6
20+
with:
21+
enable-cache: true
22+
cache-dependency-glob: "uv.lock"
2823

29-
- name: Set up Python
30-
uses: actions/setup-python@v4
31-
with:
32-
python-version: '3.12'
33-
- name: Install dependencies
34-
run: |
35-
pip install poetry
36-
- name: Invoke pylint with all dependencies
37-
run: |
38-
# Since we don't have our wheel build / install configured yet we use '--no-root'
39-
poetry install --no-root
24+
- name: Build otdf-python wheel using uv
25+
run: |
26+
uv sync --frozen
27+
uv build
28+
shell: bash
4029

41-
# poetry install
30+
- name: Find built wheel
31+
id: find_wheel
32+
run: |
33+
wheel_path=$(ls dist/*.whl | head -n1)
34+
echo "wheel_path=$wheel_path" >> $GITHUB_OUTPUT
35+
shell: bash
4236

43-
# Bring this back later
44-
# poetry run pytest tests/
37+
# - name: Upload wheel as artifact
38+
# uses: actions/upload-artifact@v4
39+
# with:
40+
# name: python-wheel
41+
# path: dist/*.whl
42+
# overwrite: true

.github/workflows/lint-on-macos.yaml

Lines changed: 0 additions & 34 deletions
This file was deleted.

.github/workflows/lint-on-ubuntu.yaml

Lines changed: 0 additions & 34 deletions
This file was deleted.

0 commit comments

Comments
 (0)