-
Notifications
You must be signed in to change notification settings - Fork 17.1k
Add Apache Arrow provider #52330
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
dabla
wants to merge
203
commits into
apache:main
Choose a base branch
from
dabla:feature/apache-arrow-provider
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Add Apache Arrow provider #52330
Changes from 2 commits
Commits
Show all changes
203 commits
Select commit
Hold shift + click to select a range
d26dca8
refactor: Added Apache Arrow provider and implemented basic AdbcHook
davidblain-infrabel a42d821
Merge branch 'main' into feature/apache-arrow-provider
dabla d44a7f2
Merge branch 'main' into feature/apache-arrow-provider
dabla b7ec972
refactor: Fixed adbc connection docs, still needs additional document…
davidblain-infrabel 61f480e
refactor: Updated link to ADBC site
davidblain-infrabel 6171a75
refactor: Fixed version in pyproject.toml
davidblain-infrabel edc7408
refactor: Fixed breeze test_docs_filter for Common SQL provider packa…
davidblain-infrabel 7e0b3ad
Merge branch 'main' into feature/apache-arrow-provider
dabla f0aba05
Merge branch 'main' into feature/apache-arrow-provider
dabla a95ba2a
Merge branch 'main' into feature/apache-arrow-provider
dabla a2695ef
Merge branch 'main' into feature/apache-arrow-provider
dabla 6c307d9
Merge branch 'main' into feature/apache-arrow-provider
dabla 565bbd2
Merge branch 'main' into feature/apache-arrow-provider
dabla 985163d
Merge branch 'main' into feature/apache-arrow-provider
dabla 04163a8
Merge branch 'main' into feature/apache-arrow-provider
dabla 80208ab
refactor: Implemented ADBC dialect
davidblain-infrabel b56cb0c
refactor: Implemented get_column_names in PostgresDialect so we don't…
davidblain-infrabel d4edee5
refactor: Removed AdbcDialect and make sure AdbcHook can work with ex…
davidblain-infrabel 6f85e54
refactor: Don't use prepared statement in PostgresDialect for get_pri…
davidblain-infrabel ff296c6
refactor: Refactored insert_rows of AdbcHook using pyarrow RecordBatch
davidblain-infrabel 1def92d
Merge branch 'main' into feature/apache-arrow-provider
dabla b1bad8f
Merge branch 'main' into feature/apache-arrow-provider
dabla 55ae8ae
Merge branch 'main' into feature/apache-arrow-provider
dabla 0b7993d
refactor: Replace native placeholders with ADBC placeholder to suppor…
davidblain-infrabel f2c19eb
refactor: Use prepared statements in PostgresDialect
davidblain-infrabel a741d46
refactor: Fixed placeholders in prepared statements in PostgresDialect
davidblain-infrabel fcb1d20
refactor: Refactored _run_command in AdbcHook
davidblain-infrabel 6fded87
refactor: Updated adbc-driver-manager and pyarrow dependencies
davidblain-infrabel 11d21d6
refactor: Refactored insert_rows
dabla 8ea29bf
Merge branch 'main' into feature/apache-arrow-provider
dabla fc6068d
Merge branch 'main' into feature/apache-arrow-provider
dabla a44d975
Merge branch 'main' into feature/apache-arrow-provider
dabla 67bc924
Merge branch 'main' into feature/apache-arrow-provider
dabla 1124109
Merge branch 'main' into feature/apache-arrow-provider
dabla 20f3e4e
Merge branch 'main' into feature/apache-arrow-provider
dabla ac59e63
Merge branch 'main' into feature/apache-arrow-provider
dabla fda7c41
Merge branch 'main' into feature/apache-arrow-provider
dabla fc18a8a
Merge branch 'main' into feature/apache-arrow-provider
dabla 59b9454
Merge branch 'main' into feature/apache-arrow-provider
dabla 3e3e1c3
refactor: Removed apache-airflow-providers-apache-arrow from airflow …
dabla c1c30b6
refactor: Removed apache-airflow-providers-apache-arrow from airflow …
dabla 596d8ed
Merge branch 'main' into feature/apache-arrow-provider
dabla ffc3b2c
Revert "refactor: Removed apache-airflow-providers-apache-arrow from …
dabla e36613a
refactor: Removed apache-airflow-providers-apache-arrow from airflow …
dabla a3f166f
refactor: Removed duplicate get_column_names in PostgresDialect
dabla 59f4b39
refactor: Added missing apache arrow entries in pyproject.toml
dabla c8769a4
refactor: Changed signature get_records in AdbcHook
dabla 0bb6a96
refactor: Changed minimum required pyarrow version to 18.0.0 to not h…
dabla 0a3f462
fix: Fixed name for flit module in apache arrow
dabla 507e5e0
fix: Fixed import of AdbcHook in TestAdbcHook
dabla ce8da7d
refactor: Added apache arrow to remove and test sources YAML files
dabla d9eccff
refactor: Updated pyproject.toml
dabla 49d35a5
refactor: Fixed provider package name in docs of arrow
dabla 802108b
refactor: Fixed project urls in pyproject.toml
dabla a688fd1
refactor: Fixed version in apache arrow provider
dabla 3e60219
refactor: Fixed message in RuntimeError for provider info in apache a…
dabla 519d725
refactor: Reformatted get_provider_info
dabla cecdcc8
refactor: Reformatted AdbcHook
dabla 13179ad
refactor: Reorganized imports for TestAdbcHook
dabla 24a3581
Merge branch 'main' into feature/apache-arrow-provider
dabla fe7d65b
Merge branch 'main' into feature/apache-arrow-provider
dabla 600ddf6
refactor: Removed handler parameter from overridden get_records metho…
dabla 5702b6e
refactor: Fixed include for security.rst
dabla 954f7d6
refactor: Fixed include for installing-providers-from-sources.rst
dabla bfb4836
Merge branch 'main' into feature/apache-arrow-provider
dabla 90d85a8
refactor: Fixed static checks and mypy issues on AdbcHook
dabla 6f080bf
Merge branch 'main' into feature/apache-arrow-provider
dabla bde3082
refactor: Fxied fetch_all_handler
dabla f857231
Merge branch 'main' into feature/apache-arrow-provider
dabla 24f04e8
Merge branch 'main' into feature/apache-arrow-provider
dabla d73a2b9
Merge branch 'main' into feature/apache-arrow-provider
dabla 442ccb6
Merge branch 'main' into feature/apache-arrow-provider
dabla aa0301e
Merge branch 'main' into feature/apache-arrow-provider
dabla 899dc5e
Merge branch 'main' into feature/apache-arrow-provider
dabla fd4f318
Merge branch 'main' into feature/apache-arrow-provider
dabla fd77c65
Merge branch 'main' into feature/apache-arrow-provider
dabla 8629eea
Merge branch 'main' into feature/apache-arrow-provider
dabla bae2196
Merge branch 'main' into feature/apache-arrow-provider
dabla a62c473
Merge branch 'main' into feature/apache-arrow-provider
dabla 7751929
Merge branch 'main' into feature/apache-arrow-provider
dabla a49e018
refactor: Changed required python version check
dabla 312410f
refactor: Uncommented apache-arrow in pyproject.toml
dabla 5a14869
refactor: Removed min version from arrow
dabla 3fabaca
Merge branch 'main' into feature/apache-arrow-provider
dabla 996d96e
Merge branch 'main' into feature/apache-arrow-provider
dabla 0129adf
Merge branch 'main' into feature/apache-arrow-provider
dabla 84df206
Merge branch 'main' into feature/apache-arrow-provider
dabla d438807
Merge branch 'main' into feature/apache-arrow-provider
dabla 0f41546
Merge branch 'main' into feature/apache-arrow-provider
dabla 41ae36a
Merge branch 'main' into feature/apache-arrow-provider
dabla b9b940a
Merge branch 'main' into feature/apache-arrow-provider
dabla 8b192a3
Merge branch 'main' into feature/apache-arrow-provider
dabla 8f25ef6
Merge branch 'main' into feature/apache-arrow-provider
dabla 82c4e29
Merge branch 'main' into feature/apache-arrow-provider
dabla e4518ce
Merge branch 'main' into feature/apache-arrow-provider
dabla 9778c9e
Merge branch 'main' into feature/apache-arrow-provider
dabla 47b04bf
Merge branch 'main' into feature/apache-arrow-provider
dabla 75efe12
Merge branch 'main' into feature/apache-arrow-provider
dabla 483d7e0
Merge branch 'main' into feature/apache-arrow-provider
dabla 90c93fc
refactor: Updated pyproject.toml and added LICENSE en NOTICE
dabla 0007ad8
refactor: Updated documentation
dabla 3a369e1
Merge branch 'main' into feature/apache-arrow-provider
dabla bd0ce99
refactor: Added redirect for adbc connections in docs
dabla dc07166
refactor: Fixed title in index.rst for arrow provider
dabla 399bbf5
refactor: Added Apache Arrow provider docs
dabla 15f49bd
docs: Added Apache Arrow connection reference in Guides
dabla 223dca6
Merge branch 'main' into feature/apache-arrow-provider
dabla 614b4de
Merge branch 'main' into feature/apache-arrow-provider
dabla 303301c
refactor: Added importlib resources dependency in apache arrow
dabla 9121d96
refactor: Added apache-arrow provider in providers bug report YAML file
dabla 17d3cf0
Merge branch 'main' into feature/apache-arrow-provider
dabla ead913d
Merge branch 'main' into feature/apache-arrow-provider
dabla 5a67ee3
Merge branch 'main' into feature/apache-arrow-provider
dabla 3a6a8ad
Merge branch 'main' into feature/apache-arrow-provider
dabla 0b2d6e6
Merge branch 'main' into feature/apache-arrow-provider
dabla 0b05d77
Merge branch 'main' into feature/apache-arrow-provider
dabla eef462c
Merge branch 'main' into feature/apache-arrow-provider
dabla dbf7f13
Merge branch 'main' into feature/apache-arrow-provider
dabla d60c08d
Merge branch 'main' into feature/apache-arrow-provider
dabla 9e68499
Merge branch 'main' into feature/apache-arrow-provider
dabla 60012df
refactor: Updated min required version of common sql provider
dabla d7301c1
refactor: Added version compat provider dependency, downgraded min re…
dabla 666f7ff
Merge branch 'main' into feature/apache-arrow-provider
dabla 16e6e7b
Merge branch 'main' into feature/apache-arrow-provider
dabla 57ed88a
refactor: We don't need common compat here
dabla 333f38f
refactor: Changed min required airflow version to 2.11.0
dabla 5a0ee0e
refactor: Removed common-compat dev dependency
dabla cea45ff
refactor: Changed min required Airflow version assertion in arrow pro…
dabla 0d5afbb
refactor: Updated docs
dabla 993d73f
Merge branch 'main' into feature/apache-arrow-provider
dabla 2eef324
refactor: Reformatted index in docs
dabla c618585
Merge branch 'main' into feature/apache-arrow-provider
dabla 09f825b
Merge branch 'main' into feature/apache-arrow-provider
dabla 57e53a8
Merge branch 'main' into feature/apache-arrow-provider
dabla 8f4a8a7
refactor: Return default if dialect name was not specified for AdbcHook
dabla 2a59d40
refactor: Added sqlite as dialect in connection fixture
dabla ad527f5
refactor: Make sure driver property from AdbcHook first checks if dri…
dabla ad1cb76
Revert "refactor: Added sqlite as dialect in connection fixture"
dabla 536cf01
Merge branch 'main' into feature/apache-arrow-provider
dabla 174d0e0
Merge branch 'main' into feature/apache-arrow-provider
dabla 6323764
Merge branch 'main' into feature/apache-arrow-provider
dabla 3a7ea6a
refactor: Added adbc-driver-sqlite as dev dependency to allow running…
dabla fb8a8fa
Merge branch 'main' into feature/apache-arrow-provider
dabla 4cfb29e
refactor: Re-use logic from connection.get_uri but change prefix
dabla b80df9f
Merge branch 'main' into feature/apache-arrow-provider
dabla 3d57bab
Merge branch 'main' into feature/apache-arrow-provider
dabla 9453fc1
Merge branch 'main' into feature/apache-arrow-provider
dabla feb03ef
Merge branch 'main' into feature/apache-arrow-provider
dabla 4fe184f
Merge branch 'main' into feature/apache-arrow-provider
dabla 234c69e
Merge branch 'main' into feature/apache-arrow-provider
dabla 56e737b
Merge branch 'main' into feature/apache-arrow-provider
dabla 4df55b3
Merge branch 'main' into feature/apache-arrow-provider
dabla 525b259
Merge branch 'main' into feature/apache-arrow-provider
dabla 7ee4fdb
Merge branch 'main' into feature/apache-arrow-provider
dabla 87fa511
Merge branch 'main' into feature/apache-arrow-provider
dabla 7bcdb8b
Merge branch 'main' into feature/apache-arrow-provider
dabla 347fca6
Merge branch 'main' into feature/apache-arrow-provider
dabla 0b62f8a
Merge branch 'main' into feature/apache-arrow-provider
dabla facd3bc
Merge branch 'main' into feature/apache-arrow-provider
dabla 592db75
Merge branch 'main' into feature/apache-arrow-provider
dabla 0cf5318
Merge branch 'main' into feature/apache-arrow-provider
dabla 5a4ebdf
Merge branch 'main' into feature/apache-arrow-provider
dabla d7a07b1
refactor: Changed version of ADBC provider to 0.1.0
dabla 3cc7c4c
Merge branch 'main' into feature/apache-arrow-provider
dabla cb646af
Merge branch 'main' into feature/apache-arrow-provider
dabla 829d704
Merge branch 'main' into feature/apache-arrow-provider
dabla 830dedb
refactor: Added one more logo
dabla b796b2d
Merge branch 'main' into feature/apache-arrow-provider
dabla eeb49dc
Merge branch 'main' into feature/apache-arrow-provider
dabla 5bfb9d4
refactor: Removed wrong references to apache-beam
dabla 0591e0a
Merge branch 'main' into feature/apache-arrow-provider
dabla f334fa5
Merge branch 'main' into feature/apache-arrow-provider
dabla 4720bf8
Merge branch 'main' into feature/apache-arrow-provider
dabla dc3780a
Merge branch 'main' into feature/apache-arrow-provider
dabla 1bf4a44
Merge branch 'main' into feature/apache-arrow-provider
dabla 3487f33
Merge branch 'main' into feature/apache-arrow-provider
dabla 61cae75
refactor: Updated uv.lock file
dabla 9021aad
Merge branch 'main' into feature/apache-arrow-provider
dabla c0cba88
Merge branch 'main' into feature/apache-arrow-provider
dabla 7e591c0
refactor: Skip tests if adbc_driver_sqlite isn't available
dabla 230c2b9
refactor: Optimize insert_rows to check use_native_bind outside for loop
dabla 0bea882
refactor: Added unit tests on AdbcHook
dabla 5d6706a
Merge branch 'main' into feature/apache-arrow-provider
dabla 8f6c30f
refactor: Fixed param for test_docs_filter
dabla dd3d692
Merge branch 'main' into feature/apache-arrow-provider
dabla 702ab85
refactor: Must check if host in connection is not None first
dabla 216eddf
refactor: Updated pyproject
dabla 0ed9c0a
refactor: Reformatted files
dabla f2c03fa
refactor: Updated release references
dabla 70b5e2a
Merge branch 'main' into feature/apache-arrow-provider
dabla c9452ae
Merge branch 'main' into feature/apache-arrow-provider
dabla 26b436b
Merge branch 'main' into feature/apache-arrow-provider
dabla 4d7efd7
refactor: Reformatted uri method
dabla cc7c3ac
refactor: Updated pyproject
dabla 88d12d7
refactor: Updated uv.lock file
dabla 9ea765e
Merge branch 'main' into feature/apache-arrow-provider
dabla 754d772
refactor: Updated pyproject
dabla d3ecae3
refactor: Fixed static checks on AdbcHook
dabla bdaea97
refactor: Added adbc drivers in optional dependencies
dabla a871422
Merge branch 'main' into feature/apache-arrow-provider
dabla 61421be
refactor: Added comment for optional dependencies in provider.yaml
dabla 10408eb
refactor: Reformatted replace_placeholders method in adbc module
dabla 5cbac82
refactor: Updated uv.lock
dabla File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
|
|
||
| .. Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
|
|
||
| .. http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| .. Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
|
|
||
| .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN! | ||
|
|
||
| .. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE | ||
| ``PROVIDER_README_TEMPLATE.rst.jinja2`` IN the ``dev/breeze/src/airflow_breeze/templates`` DIRECTORY | ||
|
|
||
| Package ``apache-airflow-providers-apache-arrow`` | ||
|
|
||
| Release: ``1.0.0`` | ||
|
|
||
|
|
||
| `Apache Arrow <https://arrow.apache.org/>`__ | ||
|
dabla marked this conversation as resolved.
|
||
|
|
||
|
|
||
| Provider package | ||
| ---------------- | ||
|
|
||
| This is a provider package for ``apache.arrow`` provider. All classes for this provider package | ||
| are in ``airflow.providers.apache.arrow`` python package. | ||
|
|
||
| You can find package information and changelog for the provider | ||
| in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-apache-arrow/1.0.0/>`_. | ||
|
|
||
| Installation | ||
| ------------ | ||
|
|
||
| You can install this package on top of an existing Airflow 2 installation (see ``Requirements`` below | ||
| for the minimum Airflow version supported) via | ||
| ``pip install apache-airflow-providers-apache-arrow`` | ||
|
|
||
| The package supports the following python versions: 3.9,3.10,3.11,3.12 | ||
|
|
||
| Requirements | ||
| ------------ | ||
|
|
||
| ======================================= ================== | ||
| PIP package Version required | ||
| ======================================= ================== | ||
| ``apache-airflow`` ``>=2.10.0`` | ||
| ``apache-airflow-providers-common-sql`` ``>=1.20.0`` | ||
| ``adbc-driver-manager`` ``>=1.6.0`` | ||
| ======================================= ================== | ||
|
|
||
| Cross provider package dependencies | ||
| ----------------------------------- | ||
|
|
||
| Those are dependencies that might be needed in order to use all the features of the package. | ||
| You need to install the specified providers in order to use them. | ||
|
|
||
| You can install such cross-provider dependencies when installing from PyPI. For example: | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| pip install apache-airflow-providers-arrow[common.sql] | ||
|
|
||
|
|
||
| ============================================================================================================ ============== | ||
| Dependent package Extra | ||
| ============================================================================================================ ============== | ||
| `apache-airflow-providers-common-sql <https://airflow.apache.org/docs/apache-airflow-providers-common-sql>`_ ``common.sql`` | ||
| ============================================================================================================ ============== | ||
|
|
||
| The changelog for the provider package can be found in the | ||
| `changelog <https://airflow.apache.org/docs/apache-airflow-providers-arrow/1.0.0/changelog.html>`_. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 7b2ec33c7ad4998d9c9735b79593fcdcd3b9dd1f |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| .. Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
|
|
||
| .. http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| .. Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
|
|
||
|
|
||
| .. NOTE TO CONTRIBUTORS: | ||
| Please, only add notes to the Changelog just below the "Changelog" header when there are some breaking changes | ||
| and you want to add an explanation to the users on how they are supposed to deal with them. | ||
| The changelog is updated and maintained semi-automatically by release manager. | ||
|
|
||
| ``apache-airflow-providers-arrow`` | ||
|
|
||
| Changelog | ||
| --------- | ||
|
|
||
| 1.0.0 | ||
| ..... | ||
|
|
||
| .. note:: | ||
| This release of provider is only available for Airflow 2.10+ as explained in the | ||
| Apache Airflow providers support policy <https://github.com/apache/airflow/blob/main/PROVIDERS.rst#minimum-supported-version-of-airflow-for-community-managed-providers>_. | ||
|
|
||
| Misc | ||
| ~~~~ | ||
|
|
||
| * ``Bump min Airflow version in providers to 2.10 (#49843)`` | ||
|
|
||
| .. Below changes are excluded from the changelog. Move them to | ||
| appropriate section above if needed. Do not delete the lines(!): | ||
| * ``Update description of provider.yaml dependencies (#50231)`` | ||
| * ``Avoid committing history for providers (#49907)`` | ||
|
|
||
| 1.0.0 | ||
| ..... | ||
|
|
||
| Initial version of the provider. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
|
|
||
| .. Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
|
|
||
| .. http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| .. Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
|
|
||
| .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN! | ||
|
|
||
| .. IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE | ||
| `PROVIDER_COMMITS_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY | ||
|
|
||
| .. THE REMAINDER OF THE FILE IS AUTOMATICALLY GENERATED. IT WILL BE OVERWRITTEN! | ||
|
|
||
| Package apache-airflow-providers-arrow | ||
| ------------------------------------------------------ | ||
|
|
||
| `ADBC: Arrow Database Connectivity <https://github.com/apache/arrow-adbc/>`__ | ||
|
|
||
|
|
||
| This is detailed commit list of changes for versions provider package: ``arrow``. | ||
| For high-level changelog, see :doc:`package information including changelog <index>`. | ||
|
|
||
| .. airflow-providers-commits:: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # Disable Flake8 because of all the sphinx imports | ||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
| """Configuration of Providers docs building.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import os | ||
|
|
||
| os.environ["AIRFLOW_PACKAGE_NAME"] = "apache-airflow-providers-arrow" | ||
|
|
||
| from docs.provider_conf import * # noqa: F403 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| .. Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
|
|
||
| .. http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| .. Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
|
|
||
| .. include:: /../../../devel-common/src/sphinx_exts/includes/providers-configurations-ref.rst | ||
| .. include:: /../../../devel-common/src/sphinx_exts/includes/sections-and-options.rst |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| .. Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
|
|
||
| .. http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| .. Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
|
|
||
| .. _howto/connection:jdbc: | ||
|
|
||
| JDBC connection | ||
|
dabla marked this conversation as resolved.
Outdated
|
||
| =============== | ||
|
|
||
| The JDBC connection type enables connection to a JDBC data source. | ||
|
|
||
| Configuring the Connection | ||
| -------------------------- | ||
|
|
||
| Host (required) | ||
| The host to connect to. | ||
|
|
||
| Schema (required) | ||
| Specify the database name to be used in. | ||
|
|
||
| Login (required) | ||
| Specify the user name to connect to. | ||
|
|
||
| Password (required) | ||
| Specify the password to connect to. | ||
|
|
||
| Port (optional) | ||
| Port of host to connect to. Not user in ``JdbcOperator``. | ||
|
|
||
| Extra (optional) | ||
| Specify the extra parameters (as json dictionary) that can be used in JDBC connection. The following parameters out of the standard python parameters are supported: | ||
|
|
||
| - ``driver_class`` | ||
| * Full qualified Java class name of the JDBC driver. For ``JdbcOperator``. | ||
| Note that this is only considered if ``allow_driver_class_in_extra`` is set to True in airflow config section | ||
| ``providers.jdbc`` (by default it is not considered). Note: if setting this config from env vars, use | ||
| ``AIRFLOW__PROVIDERS_JDBC__ALLOW_DRIVER_CLASS_IN_EXTRA=true``. | ||
|
|
||
| - ``driver_path`` | ||
| * Jar filename or sequence of filenames for the JDBC driver libs. For ``JdbcOperator``. | ||
| Note that this is only considered if ``allow_driver_path_in_extra`` is set to True in airflow config section | ||
| ``providers.jdbc`` (by default it is not considered). Note: if setting this config from env vars, use | ||
| ``AIRFLOW__PROVIDERS_JDBC__ALLOW_DRIVER_PATH_IN_EXTRA=true``. | ||
|
|
||
| .. note:: | ||
| Setting ``allow_driver_path_in_extra`` or ``allow_driver_class_in_extra`` to True allows users to set the driver | ||
| via the Airflow Connection's ``extra`` field. By default this is not allowed. If enabling this functionality, | ||
| you should make sure that you trust the users who can edit connections in the UI to not use it maliciously. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.