PoC for file upload #433

maxi297 · 2025-03-19T16:18:50Z

PoC for having file upload in API sources

maxi297 · 2025-03-19T16:18:58Z

/autofix

Auto-Fix Job Info

This job attempts to auto-fix any linting or formating issues. If any fixes are made,
those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

PR auto-fix job started... Check job output.

✅ Changes applied successfully.

coderabbitai · 2025-03-19T16:24:37Z

Caution

Review failed

The head commit changed during the review from d1d5107 to de09ae8.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

‼️ IMPORTANT
Auto-reply has been disabled for this repository in the CodeRabbit settings. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged.

Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

maxi297 · 2025-03-19T16:31:06Z

airbyte_cdk/sources/declarative/declarative_component_schema.yaml

+            anyOf:
+              - "$ref": "#/definitions/CustomRecordExtractor"
+              - "$ref": "#/definitions/DpathExtractor"
+          file_extractor:


I figured some API might return the file as a field in a JSON response for example but for now this would not be needed and we should remove it

maxi297 · 2025-03-19T16:34:02Z

airbyte_cdk/sources/declarative/retrievers/file_uploader.py

+        self._content_extractor = content_extractor
+
+    def upload(self, record: Record) -> None:
+        # TODO validate record shape - is the transformation applied at this point?


There were a couple of discussions that makes me think this is needed:

Adding a AirbyteFileRecordMessage to the protocol instead of relying on a implicit class that we only have in the CDK here

Supporting canonical schemas

In order words, this TODO logic here would validate that we can create the AirbyteMessage that describes a file

This has been outscoped for now and it seems like the record can have any shape now. I would probably add the attachment ID so that users can rely on a foreign key and the would be it. We might want to add a field on this component with jinja interpolation to let the user create the path

maxi297 · 2025-03-19T16:38:05Z

airbyte_cdk/sources/declarative/retrievers/file_uploader.py

+        if self._content_extractor:
+            raise NotImplementedError("TODO")
+        else:
+            with open(str(Path(__file__).parent / record.data["file_name"]), "ab") as f:


The path here would need to rely on the environment variable as such. We can extract this logic somewhere re-usable. There is still some questions to be answered in terms of path as it is not like a file based source i.e. that there is not necessarily a path defined by the API

This is for the base path but we also need to consider that there is no file system in API sources and there might be clashes between streams or even records so we need to take a decision here

This is for the base path but we also need to consider that there is no file system in API sources and there might be clashes between streams or even records so we need to take a decision here

Someething to consider for this one, I will update the description of this issue.

maxi297 · 2025-03-19T16:40:27Z

unit_tests/sources/declarative/file/file_stream_manifest.yaml

+        extractor:
+          type: DpathExtractor
+          field_path: ["article_attachments"]
+    file_uploader:


I added this to the stream level so that it can apply to any type of retrievers. That being said, I'm not sure asyncretrievers, statedelegatingretrievers or propertychunkingretrievers would make a lot of sense here...

I think I've identified the lower boundaries here: it needs to be at least at the end of the RecordSelector meaning. The reason is that if we have client side filtering, we only want to download the files if we know the record will get synced.

This could be an interesting middle to avoid having this only in the Concurrent CDK

We will track it here.

maxi297 · 2025-03-19T16:40:41Z

unit_tests/sources/declarative/file/test_file_stream.py

+            "credentials": {
+                "credentials": "api_token",
+                "email": "[email protected]",
+                "api_token": <redacted>


Set this to run the tests

maxi297 · 2025-03-19T19:38:20Z

airbyte_cdk/sources/declarative/concurrent_declarative_source.py

@@ -206,6 +207,20 @@ def _group_streams(
            # these legacy Python streams the way we do low-code streams to determine if they are concurrent compatible,
            # so we need to treat them as synchronous

+            file_uploader = None


One of the downside of this solution is that we need to add things on the ConcurrentDeclarativeSource level because we don't yet build a DefaultStream. This is added debt that we will clean up when we remove the declarative cursors and move the stream creation back on the model_to_component_factory but the first part is only a stretch project in Q2a so I expect that this will live there for quite some time still

maxi297 · 2025-03-20T19:25:22Z

airbyte_cdk/sources/declarative/stream_slicers/declarative_partition_generator.py

                )
+                if self._file_uploader:
+                    self._file_uploader.upload(record)
+                yield record


The file_uploader should probably set the universal file transfer information so that the destination can read it. This means that this will probably return enough information to create the AirbyteFileTransferRecordMessage

This should make the trick

airbyte_cdk/sources/declarative/stream_slicers/declarative_partition_generator.py

Co-authored-by: Maxime Carbonneau-Leclerc <[email protected]> Co-authored-by: octavia-squidington-iii <[email protected]>

PoC for file upload

d1d5107

maxi297 marked this pull request as draft March 19, 2025 16:19

Auto-fix lint and format issues

de09ae8

maxi297 commented Mar 19, 2025

View reviewed changes

maxi297 commented Mar 20, 2025

View reviewed changes

aldogonzalez8 reviewed Mar 27, 2025

View reviewed changes

airbyte_cdk/sources/declarative/stream_slicers/declarative_partition_generator.py Show resolved Hide resolved

coderabbitai bot approved these changes Mar 28, 2025

View reviewed changes

maxi297 and others added 2 commits April 2, 2025 10:19

fix mypy

b410187

PoC for emit file reference record (#443)

b122c0a

Co-authored-by: Maxime Carbonneau-Leclerc <[email protected]> Co-authored-by: octavia-squidington-iii <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PoC for file upload #433

PoC for file upload #433

maxi297 commented Mar 19, 2025

maxi297 commented Mar 19, 2025 •

edited by github-actions bot

Loading

coderabbitai bot commented Mar 19, 2025

Review failed

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

maxi297 Mar 19, 2025

maxi297 Mar 19, 2025 •

edited

Loading

maxi297 Mar 20, 2025 •

edited

Loading

maxi297 Mar 19, 2025 •

edited

Loading

maxi297 Mar 20, 2025 •

edited

Loading

aldogonzalez8 Mar 27, 2025

maxi297 Mar 19, 2025

maxi297 Mar 21, 2025 •

edited

Loading

aldogonzalez8 Mar 27, 2025

maxi297 Mar 19, 2025

maxi297 Mar 19, 2025

maxi297 Mar 20, 2025

aldogonzalez8 Mar 27, 2025

PoC for file upload #433

Are you sure you want to change the base?

PoC for file upload #433

Conversation

maxi297 commented Mar 19, 2025

maxi297 commented Mar 19, 2025 • edited by github-actions bot Loading

coderabbitai bot commented Mar 19, 2025

Review failed

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Choose a reason for hiding this comment

maxi297 Mar 19, 2025 • edited Loading

Choose a reason for hiding this comment

maxi297 Mar 20, 2025 • edited Loading

Choose a reason for hiding this comment

maxi297 Mar 19, 2025 • edited Loading

Choose a reason for hiding this comment

maxi297 Mar 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxi297 Mar 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxi297 commented Mar 19, 2025 •

edited by github-actions bot

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

maxi297 Mar 19, 2025 •

edited

Loading

maxi297 Mar 20, 2025 •

edited

Loading

maxi297 Mar 19, 2025 •

edited

Loading

maxi297 Mar 20, 2025 •

edited

Loading

maxi297 Mar 21, 2025 •

edited

Loading