Show code of other files than DAG one

[JavierLopezT]

Description

Showing code for several files besides the DAG code in the Code view of a DAG

Use case/motivation

Sometimes I have python scripts and sql files in the same folder as the DAG file. I would like to be able to see their codes in the code view. It should be configurable with template_searchpath.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

That is unlikely to happen soon IMHO.

It's a dificult one to pull if you consider how Python code is parsed, it would be next-to-impossible to find out which files to include extra and specifying them manually misses the point.

I also think Airflow is NOT good to show code other than DAG code - it's not "code browser" there is little value and high complexity to make a UI that would let you navigate between files etc. I think a much better approach would be to use another browser for your code and write a custom plugin in airflow to get the link to that browser.

For example if you have the code in Git and use GitSync, you could add a plugin to have a view where you redirect tto GitHub or GitLab UI. Even now GitHub (and I believe GitLab too) have automated detection of the linked code even in Python, so if you keep the code in the same repo you will be able to even navigate between DAG and imported code.

Let me convert this one to discussion , as I do not think we will ever want to make it a feature (but If others think otherwise we can always convert it back to a feature).

[viniciusdsmello]

Hi @potiuk! I found this discussion while searching for Issues related. I have the same need sometimes to check other files related to a DAG. It could be a SQL file or other Python modules where I defined classes and functions used as PythonOperators. I understand the way the DAGs are parsed today makes it almost impossible.

Where I work, we defined a standard to define all DAGs; for instance, a dag ABC would be defined with this structure:

dags/ABC/abc.py --> Contains only the definition of a DAG or a function that creates DAGs based on YAML/JSON files defined at conf/
dags/ABC/src/ --> Contains all classes, functions, etc.
dags/ABC/queries/ --> Contains all SQL Queries used by the DAG
dags/ABC/conf/ --> Contains all YAML/JSON config files used to define DAGs/Tasks dynamically

It would be very helpful to be able to check through Airflow UI all the files inside dags/ABC/.

[potiuk]

This is much bigger discussion and likely related to #24464 . If we are to implement it, then it has to be a much more "generic" approach than assuming specific conventions.

[potiuk]

As discussed in #24464 (comment) - this is likely a job for custom plugin to pull necessary resources or part of a "managed" solution for Airflow. It's highly unlikely we will ever implement something like that in Airlfow IMHO