Add Apache Arrow provider#52330
Conversation
|
Just for anyone looking here - this is a draft for discussion between me, @dabla and @zeroshade - we will still need to start a DISCUSSION thread for the new provider - and we think Arrow and ADBC is a good addition. But we have to first discuss the approach :) |
|
As @potiuk mentioned, I believe this needs a devlist conversation first. |
…ion regarding the configuration parameters
…ge python files changed
Yep. This one is mostly to gather learnigs, get feedback from @zeroshade and see how we can turn it into a "convincing" devlist proposal - by showing some use cases and small POC of implementation and what it allows :). We'll experiment a bit with it and gather our thoughts and see what can come out of it. |
… depend on sqlalchemy anymore
…isting dialects as ADBC dialect doesn't make sense
…mary_keys as it won't work with ADBC
# Conflicts: # uv.lock
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
|
Quickest fix: git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-leaseAutomated nudge — ignore if you're not ready to rebase. This comment is updated in place on future |
Added Apache Arrow provider in Airflow and implemented basic AdbcHook.
The AdbcHook implements the DBApiHook, so it can be reused across all SQL related operators.
I could also be used to test integration with Apache DataFusion.
@zeroshade
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.