74
74
75
75
< li class ="nav-item " data-bs-level ="1 "> < a href ="#dag-factory-quick-start-guide " class ="nav-link "> DAG Factory Quick Start Guide</ a >
76
76
< ul class ="nav flex-column ">
77
+ < li class ="nav-item " data-bs-level ="2 "> < a href ="#prerequisites " class ="nav-link "> Prerequisites</ a >
78
+ < ul class ="nav flex-column ">
77
79
</ ul >
78
80
</ li >
79
-
80
- < li class ="nav-item " data-bs-level ="1 "> < a href ="#prerequisites " class ="nav-link "> Prerequisites</ a >
81
+ < li class ="nav-item " data-bs-level ="2 "> < a href ="#step-1-create-a-python-virtual-environment " class ="nav-link "> Step 1: Create a Python Virtual Environment</ a >
81
82
< ul class ="nav flex-column ">
82
83
</ ul >
83
84
</ li >
84
-
85
- < li class ="nav-item " data-bs-level ="1 "> < a href ="#step-1-create-a-python-virtual-environment " class ="nav-link "> Step 1: Create a Python Virtual Environment</ a >
85
+ < li class ="nav-item " data-bs-level ="2 "> < a href ="#step-2-install-apache-airflow " class ="nav-link "> Step 2: Install Apache Airflow</ a >
86
86
< ul class ="nav flex-column ">
87
87
</ ul >
88
88
</ li >
89
-
90
- < li class ="nav-item " data-bs-level ="1 "> < a href ="#step-2-install-apache-airflow " class ="nav-link "> Step 2: Install Apache Airflow</ a >
89
+ < li class ="nav-item " data-bs-level ="2 "> < a href ="#step-3-install-dag-factory " class ="nav-link "> Step 3: Install DAG Factory</ a >
91
90
< ul class ="nav flex-column ">
92
91
</ ul >
93
92
</ li >
94
-
95
- < li class ="nav-item " data-bs-level ="1 "> < a href ="#step-3-install-dag-factory " class ="nav-link "> Step 3: Install DAG Factory</ a >
93
+ < li class ="nav-item " data-bs-level ="2 "> < a href ="#step-4-set-up-the-dags-folder " class ="nav-link "> Step 4: Set Up the DAGS Folder</ a >
96
94
< ul class ="nav flex-column ">
97
95
</ ul >
98
96
</ li >
99
-
100
- < li class ="nav-item " data-bs-level ="1 "> < a href ="#step-4-set-up-the-dags-folder " class ="nav-link "> Step 4: Set Up the DAGS Folder</ a >
101
- < ul class ="nav flex-column ">
102
97
</ ul >
103
98
</ li >
104
99
105
100
< li class ="nav-item " data-bs-level ="1 "> < a href ="#step-5-define-a-dag-in-yaml " class ="nav-link "> Step 5: Define a DAG in YAML</ a >
106
101
< ul class ="nav flex-column ">
102
+ < li class ="nav-item " data-bs-level ="2 "> < a href ="#step-6-generate-the-dag-from-yaml " class ="nav-link "> Step 6: Generate the DAG from YAML</ a >
103
+ < ul class ="nav flex-column ">
107
104
</ ul >
108
105
</ li >
109
-
110
- < li class ="nav-item " data-bs-level ="1 "> < a href ="#step-6-generate-the-dag-from-yaml " class ="nav-link "> Step 6: Generate the DAG from YAML</ a >
106
+ < li class ="nav-item " data-bs-level ="2 "> < a href ="#step-7-start-airflow " class ="nav-link "> Step 7: Start Airflow</ a >
111
107
< ul class ="nav flex-column ">
112
108
</ ul >
113
109
</ li >
114
-
115
- < li class ="nav-item " data-bs-level ="1 "> < a href ="#step-7-start-airflow " class ="nav-link "> Step 7: Start Airflow</ a >
110
+ < li class ="nav-item " data-bs-level ="2 "> < a href ="#view-your-generated-dag " class ="nav-link "> View Your Generated DAG</ a >
116
111
< ul class ="nav flex-column ">
117
112
</ ul >
118
113
</ li >
119
-
120
- < li class ="nav-item " data-bs-level ="1 "> < a href ="#view-your-generated-dag " class ="nav-link "> View Your Generated DAG</ a >
121
- < ul class ="nav flex-column ">
122
114
</ ul >
123
115
</ li >
124
116
</ ul >
128
120
129
121
< h1 id ="dag-factory-quick-start-guide "> DAG Factory Quick Start Guide</ h1 >
130
122
< p > < strong > DAG Factory</ strong > is a Python library < a href ="https://airflow.apache.org "> Apache Airflow®</ a > that simplifies DAG creation using declarative YAML configuration files instead of Python.</ p >
131
- < h1 id ="prerequisites "> Prerequisites</ h1 >
123
+ < h2 id ="prerequisites "> Prerequisites</ h2 >
132
124
< p > The minimum requirements for < strong > dag-factory</ strong > are:</ p >
133
125
< ul >
134
126
< li > Python 3.8.0+</ li >
135
127
< li > < a href ="https://airflow.apache.org "> Apache Airflow®</ a > 2.0+</ li >
136
128
</ ul >
137
- < h1 id ="step-1-create-a-python-virtual-environment "> Step 1: Create a Python Virtual Environment</ h1 >
129
+ < h2 id ="step-1-create-a-python-virtual-environment "> Step 1: Create a Python Virtual Environment</ h2 >
138
130
< p > First, create and activate a virtual environment:</ p >
139
131
< div class ="language-text highlight "> < pre > < span > </ span > < code > < span id ="__span-0-1 "> < a id ="__codelineno-0-1 " name ="__codelineno-0-1 " href ="#__codelineno-0-1 "> </ a > python3 -m venv dagfactory_env
140
132
</ span > < span id ="__span-0-2 "> < a id ="__codelineno-0-2 " name ="__codelineno-0-2 " href ="#__codelineno-0-2 "> </ a > source dagfactory_env/bin/activate
141
133
</ span > </ code > </ pre > </ div >
142
- < h1 id ="step-2-install-apache-airflow "> Step 2: Install Apache Airflow</ h1 >
134
+ < h2 id ="step-2-install-apache-airflow "> Step 2: Install Apache Airflow</ h2 >
143
135
< p > Next, you'll need to install < a href ="https://airflow.apache.org "> Apache Airflow®</ a > . Follow these steps:</ p >
144
136
< ol >
145
137
< li >
@@ -159,11 +151,11 @@ <h1 id="step-2-install-apache-airflow">Step 2: Install Apache Airflow</h1>
159
151
</ span > </ code > </ pre > </ div >
160
152
</ li >
161
153
</ ol >
162
- < h1 id ="step-3-install-dag-factory "> Step 3: Install DAG Factory</ h1 >
154
+ < h2 id ="step-3-install-dag-factory "> Step 3: Install DAG Factory</ h2 >
163
155
< p > Now, install the DAG Factory library in your virtual environment:</ p >
164
156
< div class ="language-text highlight "> < pre > < span > </ span > < code > < span id ="__span-4-1 "> < a id ="__codelineno-4-1 " name ="__codelineno-4-1 " href ="#__codelineno-4-1 "> </ a > pip install dag-factory
165
157
</ span > </ code > </ pre > </ div >
166
- < h1 id ="step-4-set-up-the-dags-folder "> Step 4: Set Up the DAGS Folder</ h1 >
158
+ < h2 id ="step-4-set-up-the-dags-folder "> Step 4: Set Up the DAGS Folder</ h2 >
167
159
< p > Create a dags folder inside the $AIRFLOW_HOME directory, which is where your DAGs will be stored:</ p >
168
160
< div class ="language-text highlight "> < pre > < span > </ span > < code > < span id ="__span-5-1 "> < a id ="__codelineno-5-1 " name ="__codelineno-5-1 " href ="#__codelineno-5-1 "> </ a > mkdir dags
169
161
</ span > </ code > </ pre > </ div >
@@ -263,7 +255,7 @@ <h1 id="step-5-define-a-dag-in-yaml">Step 5: Define a DAG in YAML</h1>
263
255
</ span > < span id ="__span-6-92 "> < a id ="__codelineno-6-92 " name ="__codelineno-6-92 " href ="#__codelineno-6-92 "> </ a > bash_command: "echo 1"
264
256
</ span > < span id ="__span-6-93 "> < a id ="__codelineno-6-93 " name ="__codelineno-6-93 " href ="#__codelineno-6-93 "> </ a > dependencies: [task_group_1]
265
257
</ span > </ code > </ pre > </ div >
266
- < h1 id ="step-6-generate-the-dag-from-yaml "> Step 6: Generate the DAG from YAML</ h1 >
258
+ < h2 id ="step-6-generate-the-dag-from-yaml "> Step 6: Generate the DAG from YAML</ h2 >
267
259
< p > Create a Python script named < code > example_dag_factory.py</ code > in the < code > $AIRFLOW_HOME/dags</ code > folder. This script will generate the DAG from the YAML configuration</ p >
268
260
< div class ="language-text highlight "> < span class ="filename "> example_dag_factory.py</ span > < pre > < span > </ span > < code > < span id ="__span-7-1 "> < a id ="__codelineno-7-1 " name ="__codelineno-7-1 " href ="#__codelineno-7-1 "> </ a > import os
269
261
</ span > < span id ="__span-7-2 "> < a id ="__codelineno-7-2 " name ="__codelineno-7-2 " href ="#__codelineno-7-2 "> </ a > from pathlib import Path
@@ -283,12 +275,12 @@ <h1 id="step-6-generate-the-dag-from-yaml">Step 6: Generate the DAG from YAML</h
283
275
</ span > < span id ="__span-7-16 "> < a id ="__codelineno-7-16 " name ="__codelineno-7-16 " href ="#__codelineno-7-16 "> </ a > example_dag_factory.clean_dags(globals())
284
276
</ span > < span id ="__span-7-17 "> < a id ="__codelineno-7-17 " name ="__codelineno-7-17 " href ="#__codelineno-7-17 "> </ a > example_dag_factory.generate_dags(globals())
285
277
</ span > </ code > </ pre > </ div >
286
- < h1 id ="step-7-start-airflow "> Step 7: Start Airflow</ h1 >
278
+ < h2 id ="step-7-start-airflow "> Step 7: Start Airflow</ h2 >
287
279
< p > To start the Airflow environment with your DAG Factory setup, run the following command:</ p >
288
280
< div class ="language-text highlight "> < pre > < span > </ span > < code > < span id ="__span-8-1 "> < a id ="__codelineno-8-1 " name ="__codelineno-8-1 " href ="#__codelineno-8-1 "> </ a > airflow standalone
289
281
</ span > </ code > </ pre > </ div >
290
282
< p > This will take a few minutes to set up. Once completed, you can access the Airflow UI and the generated DAG at http://localhost:8080 🚀.</ p >
291
- < h1 id ="view-your-generated-dag "> View Your Generated DAG</ h1 >
283
+ < h2 id ="view-your-generated-dag "> View Your Generated DAG</ h2 >
292
284
< p > Once Airflow is up and running, you can login with the username < code > admin</ code > and the password in < code > $AIRFLOW_HOME/standalone_admin_password.txt</ code > . You should be able to see your generated DAG in the Airflow UI.</ p >
293
285
< p > < strong > Generated DAG</ strong > </ p >
294
286
< p > < img alt ="Airflow DAG " src ="../static/images/airflow-home.png " /> </ p >
0 commit comments