Add deferred pagination mode to GenericTransfer#44809

Open

dabla wants to merge 114 commits intoapache:mainfrom dabla:feature/paginated-generic-transfer

+583-153

Commits on Jan 2, 2025

refactor: Refactored the GenericTransfer operator to support paginated reads (in deferred mode) and introduce a SQLExecuteQueryTrigger

davidblain-infrabel
authored and
potiuk
committed
refactor: updated provider dependencies

dabla
authored and
potiuk
committed
refactor: Added TestSQLExecuteQueryTrigger and moved test code which allows you to run deferrable operators in test common test utils

davidblain-infrabel
authored and
potiuk
committed
refactor: Fixed static checks

davidblain-infrabel
authored and
potiuk
committed
refactor: Fixed static checks

davidblain-infrabel
authored and
potiuk
committed
refactor: Fixed static checks

davidblain-infrabel
authored and
potiuk
committed
refactor: Reformatted GenericTransfer

davidblain-infrabel
authored and
potiuk
committed
refactor: Moved source and destination hooks into cached properties

davidblain-infrabel
authored and
potiuk
committed
refactor: Moved imports to type checking block

davidblain-infrabel
authored and
potiuk
committed
refactor: Fixed execute method of GenericTransfer

davidblain-infrabel
authored and
potiuk
committed
refactor: Refactored get_hook method of GenericTransfer which checks if hook is instance of DbApiHook instead of checking presence of get_records and insert_rows method

davidblain-infrabel
authored and
potiuk
committed
refactor: Remove white lines from mock_context

davidblain-infrabel
authored and
potiuk
committed
refactor: Reformatted get_hook in GenericTransfer operator

davidblain-infrabel
authored and
potiuk
committed
refactor: Added sql.pyi for SQLExecuteQueryTrigger

davidblain-infrabel
authored and
potiuk
committed
refactor: Reformatted SQLExecuteQueryTrigger definition

davidblain-infrabel
authored and
potiuk
committed
refactor: Added alias in SQLExecuteQueryTrigger definition

davidblain-infrabel
authored and
potiuk
committed
Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored
Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Jan 7, 2025

Commits on Jan 9, 2025

Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Jan 11, 2025

Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Jan 13, 2025

Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Jan 15, 2025

Commits on Jan 23, 2025

Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Jan 28, 2025

Commits on Jan 29, 2025

Commits on Jan 30, 2025

Updates docs to improve Readability (apache#46205 )

aritra24
authored and
davidblain-infrabel
committed
Add Webserver parameters: max_form_parts, max_form_memory_size (apache#45749 )

moiseenkov
authored and
davidblain-infrabel
committed
Fix api_connexion test failures in main (apache#46245 )

vincbeck
authored and
davidblain-infrabel
committed
Add autorefresh to Task Instance and Dag Run pages (apache#46213 )

authored and
davidblain-infrabel
committed
use populate_by_name (apache#46168 )

rawwar
authored and
davidblain-infrabel
committed
Disable Flask-SQLAlchemy modification tracking in FAB provider (apache#46249 )

jedcunningham
authored and
davidblain-infrabel
committed
Fix AWS auth manager compatibility test (apache#46246 )

vincbeck
authored and
davidblain-infrabel
committed
Prevent from being added to root logger by default (apache#46236 )

amoghrajesh
authored and
davidblain-infrabel
committed
Move more DAG parsing related config to dag_processor section (apache#46034 )

jedcunningham
authored and
davidblain-infrabel
committed
Update test fixture docstring (apache#46252 )

jedcunningham
authored and
davidblain-infrabel
committed
AIP-38 Invalidate DryRun query cache on submit (apache#46238 )

pierrejeambrun
authored and
davidblain-infrabel
committed
Fix pre-commit to re-generate on dependency changes (apache#46259 )

jscheffl
authored and
davidblain-infrabel
committed
update some icons and add dag import error to dagslist (apache#46251 )

bbovenzi
authored and
davidblain-infrabel
committed
Limit google-cloud-aiplatform to fix issues in CI (apache#46242 )

authored and
davidblain-infrabel
committed
Adding from_path.exists() check in a couple of conditions (apache#46255 )

kunaljubce
authored and
davidblain-infrabel
committed
Remove caplog from HDFS tests (apache#46263 )

jscheffl
authored and
davidblain-infrabel
committed
Remove caplog from Livy tests (apache#46272 )

potiuk
authored and
davidblain-infrabel
committed
Replace caplog with patching log property in k8s tests (apache#46273 )

potiuk
authored and
davidblain-infrabel
committed
Remove import from MySQL provider tests in generic transfer test (apache#46274 )

potiuk
authored and
davidblain-infrabel
committed
Fix GitDagBundle to support https (include 46073/46179) (apache#46226 )

jx2lee
authored and
davidblain-infrabel
committed
AIP-72: Port _validate_inlet_outlet_assets_activeness into Task SDK (apache#46020 )

amoghrajesh
authored and
davidblain-infrabel
committed
Try and reduce flakiness of number of test call asserts (apache#46276 )

amoghrajesh
authored and
davidblain-infrabel
committed
Add evaluation extra to google-cloud-aiplatform (apache#46270 )

authored and
davidblain-infrabel
committed
Move Docker Provider to the New Structure (apache#46097 )

bugraoz93
authored and
davidblain-infrabel
committed
Move Atlassian Jira Provider to the new structure (apache#46271 )

bugraoz93
authored and
davidblain-infrabel
committed
moved apache.cassandra provider to new structure and fixed check-for-inclusive-language (apache#46171 )

authored and
davidblain-infrabel
committed
Make azure test less flaky/racy (apache#46281 )

potiuk
authored and
davidblain-infrabel
committed
move Influxdb provider to new provider structure (apache#46277 )

authored and
davidblain-infrabel
committed
Move Apprise provider to new structure (apache#46161 )

authored and
davidblain-infrabel
committed
Make racy test test_start_pod_startup_interval_seconds less racy (apache#46282 )

potiuk
authored and
davidblain-infrabel
committed
Move FTP Provider to the New Structure (apache#46206 )

aritra24
authored and
davidblain-infrabel
committed
move Trino provider to new provider structure (apache#46162 )

authored and
davidblain-infrabel
committed
refactor(providers/slack): move slack provider to new structure (apache#46209 )

josix
authored and
davidblain-infrabel
committed
Move Apache Livy to new provider structure (apache#46131 )

jason810496
authored and
davidblain-infrabel
committed
refactor: Re-added test_generic_transfer under standard provider
davidblain-infrabel
committed
Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored
refactor: Moved test_generic_transfer from standard to common sql provider
davidblain-infrabel
committed
Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Jan 31, 2025

Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored