"Extra" connection field purpose

[asuvork]

Hello!

Could you help me to understand what is the purpose of extra connection field, please?

I used to keep in "extra" any specific for my project properties. For example, "oracle" connection contains information to handle connection to this database from Spark. So, I am able to connect to Oracle database with Spark and directly form Airflow and to keep connection information for these cases in single place.

  "oracle_connection": {
    "conn_type": "Oracle",
    "host": "address",
    "schema": "",
    "login": "login",
    "password": "password",
    "port": 1524,
    "extra": "{\"sid\": "schema", \"spark_parameters\": {\"url\": \"jdbc:oracle:thin:@{host}:{port}:{sid}\", \"fetchsize\": 100, \"driver\": \"oracle.jdbc.driver.OracleDriver\"}}"
  },

It works well, because Oracle provider uses get method to connect parameters from extra:
conn.extra_dejson.get("sid")
https://github.com/apache/airflow/blob/main/airflow/providers/oracle/hooks/oracle.py#LL154C9-L154C43

Kafka provider sends the whole "extra" field content to Consumer constrictor and fails, because it faced unexpected arguments

config = self.get_connection(self.kafka_config_id).extra_dejson  

https://github.com/apache/airflow/blob/main/airflow/providers/apache/kafka/hooks/base.py#L62

return Consumer(config)

https://github.com/apache/airflow/blob/main/airflow/providers/apache/kafka/hooks/consume.py#L40

Is it good practice to have specific to project key-value in extra or should i avoid it? It there any rule for it?

[potiuk]

Axtra is Airflow way to get generic way of defining extra parameters but they are handled by each connection in the way how author of the connection decided it to use them.

Look at the docs/code of every connection (defined by Hooks) to see whch extras are supported. If Kafka has connection does not support extras, it makes no sense to add it there. Extra is "airflow" specific field and if somoene decided in Kafka connection to put extra fields in config then this is how Kafka connection handles it - and you have to adapt to it. Look at the docs and if in doubt - at the code.

Your use of extras to handle "your" data is just wrong and it is not going to work in general case so don't do it.

[asuvork]

So, looks like I must either implement my own providers or move my data from extra field to some another place (e.g. variables). Thank you!

[ertancelik]

I have a question about Ariflow oracle connection. I tried a lot but couldn't find a solution, I need your help. I am using Airflow 2.5.1 version on Ubuntu machine, I have installed oracle provider etc. everything is available in the db network, but when connecting, I get this error:init() missing 1 required positional argument: 'backend'

[Taragolis]

@ertancelik rather than create question in discussion which are not related to your problem better to create separate one.

With more details:

• Logs from webserver/scheduler/worker at moment when error happen
• Which providers versions are you use + packages directly related to Oracle Provider
• Your installation method

And try to avoid attach huge screenshots which looks like