Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow check_response to be defined in extra_options of HTTP connection #45451

Open
wants to merge 42 commits into
base: main
Choose a base branch
from

Conversation

dabla
Copy link
Contributor

@dabla dabla commented Jan 7, 2025

This PR allows you to define the check_response option in the extra options fields of the HTTP connection so that you can disable the check globally for that connection instead of always having to pass it to each HttpOperator using that same connection id.

I also took the opportunity to refactor the get_conn method of the HttpHook so you can easily override the default host and default headers in the specialized hooks so those subclasses (e.g. LivyHook or DingdingHook) don't need to override the get_conn method anymore, otherwise as the signature has changed (e.g. added extra_options parameter) those has to be changed also and the exisiting solution wasn't DRY also, now it is.

In the past if you want to disable the response_check in the HttpHook through the HtpOperator, you had to define the HttpOperator like this:

validate_custom_fields_task = HttpOperator(
    task_id="get_data",
    method="GET",
    http_conn_id="http_conn_id",
    endpoint="/endpoint",
    headers={},
    response_check=response_check,
    extra_options={"check_response": False},  # allows you to use the custom response_check instead of the HttpHook buildin check
    dag=dag,
)

Now thanks to this PR, you can define the check_response globally in the extra_options of the connection, so it will be applied to each HttpOperator using that connection.

image


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

davidblain-infrabel and others added 23 commits January 7, 2025 13:49
…gHook so we don't need have to override the complex get_conn method
…of HttpHook so we don't need to override the get_conn method in DingdingHook and LivyHook
@dabla
Copy link
Contributor Author

dabla commented Jan 14, 2025

@potiuk or @eladkal Could any of you have a look and check if this MR can be merged please? It's a fix for issue #45237

@potiuk
Copy link
Member

potiuk commented Jan 14, 2025

You will need to bump HTTP provider version to new minor and make the other providers depend on it, otherwise someone can install old version of HTTP provider and new version of (say) livy and it wil not work.

…e them dependent of http provider 5.0.1 or higher
@dabla
Copy link
Contributor Author

dabla commented Jan 15, 2025

You will need to bump HTTP provider version to new minor and make the other providers depend on it, otherwise someone can install old version of HTTP provider and new version of (say) livy and it wil not work.

Ok thx @potiuk bumped the version of the http, livy and dingding provider and made them dependent on version 5.0.1 of http provider.

@dabla
Copy link
Contributor Author

dabla commented Jan 17, 2025

You will need to bump HTTP provider version to new minor and make the other providers depend on it, otherwise someone can install old version of HTTP provider and new version of (say) livy and it wil not work.

I've bumped the http provider (in provider.yaml) but is it normal I have to modify the version manually in module as well?

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "5.0.0"
+__version__ = "5.0.1"

Maybe it's because I can't use the pre-commits on my Windows machine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants