Skip to content

chore: use sdm cli as dockerfile entrypoint #470

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

chore: use sdm cli as dockerfile entrypoint #470

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a83bfea
update dockerfile to use sdm cli
ChristoGrab Apr 10, 2025
4536c33
Merge branch 'main' into christo/sdm-cli-dockerfile
ChristoGrab Apr 15, 2025
c897e13
Merge branch 'main' of https://github.com/airbytehq/airbyte-python-cd…
ChristoGrab Apr 16, 2025
824c63f
add readme first draft
ChristoGrab Apr 17, 2025
9e1f742
chore: update readme
ChristoGrab Apr 17, 2025
7b75141
Update airbyte_cdk/cli/README.md
ChristoGrab Apr 17, 2025
99ed234
Merge branch 'main' into christo/sdm-cli-dockerfile
ChristoGrab Apr 17, 2025
79f28c0
chore: add AIRBYTE_ENTRYPOINT
ChristoGrab Apr 17, 2025
3c59dbf
Merge branch 'christo/sdm-cli-dockerfile' of https://github.com/airby…
ChristoGrab Apr 17, 2025
878c749
Merge branch 'main' into christo/sdm-cli-dockerfile
ChristoGrab Apr 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ COPY dist/*.whl ./dist/
RUN poetry config virtualenvs.create false \
&& poetry install --only main --no-interaction --no-ansi || true

# Build and install the package
# Install source-declarative-manifest CLI and its dependencies
RUN pip install dist/*.whl

# Recreate the original structure
Expand All @@ -33,7 +33,7 @@ RUN rm -rf dist/ pyproject.toml poetry.lock README.md
# Set ownership of /airbyte to the non-root airbyte user and group (1000:1000)
RUN chown -R 1000:1000 /airbyte

# Set the entrypoint
ENV AIRBYTE_ENTRYPOINT="python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
# Set the entrypoint to use the source-declarative-manifest CLI
ENV AIRBYTE_ENTRYPOINT="source-declarative-manifest"
ENTRYPOINT ["source-declarative-manifest"]
USER airbyte
63 changes: 63 additions & 0 deletions airbyte_cdk/cli/README.md
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ChristoGrab - If we move this markdown content into the cli/__init__.py docstring, it will automatically render in pdoc when we generate the API docs via GitHub pages. Wdyt?

Copy link
Contributor

@aaronsteers aaronsteers Apr 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's also a syntax to import markdown file into the pdoc generation if you want to keep it a separate file.

Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Source Declarative Manifest CLI Usage Guide

This guide explains how to install and use the Source Declarative Manifest (SDM) CLI tool for Airbyte connector development.

## Installation

### Standard Installation

```bash
pipx install airbyte-cdk
```

If you encounter an error related to a missing `distutils` module, verify that you are running Python version `<=3.11` and try running:

```bash
python -m pipx install airbyte-cdk
```

## Using the CLI

The SDM CLI follows standard Airbyte connector command patterns:

```bash
source-declarative-manifest [command] --config /path/to/config.json
```

Where [command] can be:

spec - Show connector specification
check - Verify connection to the source
discover - List available streams
read - Read data from streams

:::caution
When developing locally (outside a Docker container), the CLI operates in "remote manifest mode" and expects your manifest to be included in your configuration file.
:::

### Steps for Local Testing

1. Convert your manifest from YAML to JSON

Your manifest is defined in YAML, but must be converted to JSON for the config file. You can use an [online tool](https://onlineyamltools.com/convert-yaml-to-json) to do so.

Create a config file that includes both your config parameters AND the manifest. Add your entire manifest as a JSON object under the `__injected_declarative_manifest` key

Example:

```json
{
"api_key": "very_secret_key",
"start_time": "04:20",
"__injected_declarative_manifest": {
// Insert the JSON version of your manifest here
}
}
```
Comment on lines +46 to +56
Copy link
Contributor

@coderabbitai coderabbitai bot Apr 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

JSON snippet contains invalid comments

JSON doesn’t support // comments, which will break parsing. Should we replace the inline comment with a placeholder or remove it entirely? For example:

-  "__injected_declarative_manifest": {
-    // Insert the JSON version of your manifest here
-  }
+  "__injected_declarative_manifest": { /* insert your manifest JSON here */ }

wdyt?

Committable suggestion skipped: line range outside the PR's diff.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai Oh nice! I wasn't aware of that placeholder pattern, please go ahead and make this change.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ChristoGrab Great! I'll plan that change for you now. The JSON comment will be updated to use the proper format:

%%{init: {"securityLevel":"loose"}}%%
flowchart TB
    A["Fix JSON comment in README"] --> B["Replace // style comment with /* */ style"]
Loading

Another agent will execute this change for you shortly. This will ensure that the example JSON in the documentation is properly formatted and won't cause parsing errors when copied by users.

⏳ Preparing changes...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A summary of the edits CodeRabbit can apply:

We will replace the invalid // comment with a proper /* */ comment in the JSON example in airbyte_cdk/cli/README.md.

A plan of each step that CodeRabbit will take in order to execute the edits for each file (with inline rough draft snippets):

✏️ airbyte_cdk/cli/README.md

Delete lines 51–53 (the opening brace, the // comment line, and the closing brace) and insert a single line:

  "__injected_declarative_manifest": { /* insert your manifest JSON here */ }

Ensure there are two spaces of indentation before "__injected_declarative_manifest" to match the surrounding JSON block, and verify no trailing commas or extra braces are introduced.

Save the file and preview the Markdown rendering to confirm that the JSON block is syntactically valid and that the comment displays correctly in all rendered views.

A summary of the context that CodeRabbit is considering across the codebase:

Locate the JSON snippet containing the invalid // comment in airbyte_cdk/cli/README.md to confirm the exact line numbers to modify.

  • ✅ Create PR with these edits

  • 📋 Get copyable edits

  • 📌 Create commit in current branch

  • 🔄 Try a different approach

  • ⏪ Go back to ordinary chat


2. Run the command against your config file

```bash
source-declarative-manifest check --config /relative/path/to/config.json
source-declarative-manifest read --config /relative/path/to/config.json --catalog /relative/path/to/catalog.json
```
Loading