Airflow Error and Warning Counter

[warrenatkison-Vega]

I recently set up my DAG's to send notifications to Slack when they complete, and I really want to include the number of warnings and errors in that notification. The best I can do right now it just a link to the logs which is fine but not as practical as just the number of errors and warnings. Anyone know how I can get the number of errors and warnings?

[potiuk]

You can add your custom notifier to read the log, parse them and calculate things. Airflow is a platform that can be extended and most of the components can be extended by writing python code to do it. If you want more "robust" solution you can also add a custom handled following https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html - then you could count the errors and warnings (assuming they are printed as "log" entries - and store them next to the log file in a separate file that will save you later the need to parse the logs and count the warnings/errors - you would just neead to read that file with counts.