Error Importing Provider with Custom Provider

[JDRanpariya]

Python: 3.10
Apache Airflow: 2.7

I'm creating a custom provider. I have followed the advice of having an entry point and mentioned the required fields along with a connection-types array with connection-type and hook-class-name. I've provided.yml with hooks and connection details as mentioned. But when I install the package locally It shows under airflow providers but when I do airflow providers hooks it says Error When importing hook. And I'm also not able to use `airflow.providers.customprovider.*

Am I missing something? I tried running the Astronomers sample repo directly but there too same issue.

I guess the package is not working with the provider manager but don't know what to do about it.

[potiuk]

What does your "get_provider_info" entrypoint returns ? can you post the dictionary it returns? Does it validate correctly with "provider_info.schema.json" ?

[JDRanpariya]

def get_provider_info():
    return {
            "package-name": "apache-airflow-providers-custom",
            "name": "Custom",
            "description": "https://abc.io/",
            "connection-types": [
                {
                    "connection-type": "custom",
                    "hook-class-name": "custom.hooks.custom.CustomHook"
                    }
                ],
            }

I have this inside init.py of my root dir. One step down where my pyproject.toml there I've provided entry point as custom:init:get_provider_info

[JDRanpariya]

Also, do I need provider.yml in the root directory? Or shell I mention Integrations, Operators, and hooks here itself? I looked into the airflow/providers folder on how they are installed and they seem to have those added and instead of init they have entry point in get_provider_info.py but I don't think that should matter.

[potiuk]

No. Provider.yaml is only for airflow built-in providers https://airflow.apache.org/docs/apache-airflow-providers/howto/create-custom-providers.html

Again. The way to verify if your provider_info is good is to check if it can be validated with the provider info schema. Have you done that? does it validete? Are there any errors?

Fir the info ofd yours is your "custom.hooks.custom.CustomHook" importable?

What is the exact error you are getting - where are you getting the "Error when importing hook!" ? What's the output?

Also Look here for details describing your custom provider preparation: https://airflow.apache.org/docs/apache-airflow-providers/howto/create-custom-providers.html

[JDRanpariya]

Thanks, Apologies but may I know how do I validate my return of get_provider_info with provider info schema?

1. I did check manually and seems fine
2. I tried doing it with validator called pajv. Giving me following output

➜ pajv -s ~/provider_info.schema.json -d mydata.json --verbose --all-errors --error-data-path=property 
schema /home/user/provider_info.schema.json is invalid
error: no schema with key or ref "http://json-schema.org/draft-07/schema#"

Also Importing hook in that file which contains get_provider_info() with from .hook.custom.CustomHook doesn't throw any errors.

Here is the output of error.

I've followed the guide you described. Just that I'm using pyproject.toml but I'm providing entry point as mentioned.

[JDRanpariya]

It's validated successfully. I had to use schema provided in that json. But Still I'm unable to import the provider.

[JDRanpariya]

I hope python 3.10.12 isn't the issue. Edit: Tried with 3.9.0 but still the same issue.

[potiuk]

Your problem is different. Why Are you doing `from airflow.providers import custom`` ?

Your provider is apparently in "custom" package, not in "airflow.providers.custom".

Your import should simply be import custom. The airflow.providers is where the community providers are, but your own provider should be elsewhere. Why custom is not the package I'd use but it's correctly outside of airflow.providers and you should import it directly - there is absolutely no reasons why you should use from airflow.providers from it.

Providers are just regular packages you can just import. The airflow.providers. for community ones is just a convention

[JDRanpariya]

Thanks!! For some reason, I was thinking just in terms of community providers and that had me. It's working all fine now. I would close the discussion.