How to use get logs end point in REST API

[wuminorb]

Hi, we are try to using get logs end point ( https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_log ) in REST API. But we can't find any doc for the log's formate. How should we parse the content, and how to find out if logs are ending or not?

[potiuk]

You get the same logs as you see in the UI. They are content of the log files written by tasks. You can control logs by airflow configuration: https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-architecture.html - you do not know whether the log is complete or not. They are not streaming, you get the full content of logs present for the task at the moment you query. You can also check the status of tasks using other API endpoints and get the log when task is finished - then you know it is "complete".

[potiuk]

I think you get logs line by line so likely not. But best is to try and see.

[anovv]

Came here searching for the answer to the same question and not sure why this reply is marked as an answer. Specifically, the content field of API response is encoded and as the question author asked, it is not clear how to properly decode it.

Also the usage of continuation_token is not clear. Consider the case where we want to continuously monitor logs for a task (should it be done by calling endpoint in a loop?), it is obviously infeasible to fetch whole file on every call, we want to fetch diff only. Do we send continuation_token from the previous response as a part of a new request? Is there a code example?

[anovv]

To answer the first question content is utf-8 encoded string with escape characters. Example getting it with Python, decoding with codec module:

api_instance = TaskInstanceApi(airflow_api_client)
api_response = api_instance.get_log(dag_id, dag_run_id, task_id, task_try_number)
encoded_content = api_response['content']
decoded = codecs.escape_decode(bytes(encoded_content, "utf-8"))[0].decode("utf-8")

[anovv]

For the second question:

By default, get_log api does not fetch all of the task's logs content. The endpoint has full_content flag (False by default, so it fetches only the first block), indicating if full content should be fetched.

Regarding continuation_token, example to stream logs using Python:

  def watch_task_logs(user_id: str, task_id: str, dag_id: str, dag_run_id: str) -> Generator:
      continuation_token = None
      while True:
          kwargs = {
              'full_content': continuation_token is None,
          }
          if continuation_token is not None:
              kwargs['token'] = continuation_token

          logs = self.airflow_task_instance_api.get_log(
              dag_id=dag_id,
              dag_run_id=dag_run_id,
              task_id=task_id,
              task_try_number=1,
              **kwargs
          )
          content, continuation_token = logs['content'], logs['continuation_token']
          decoded = codecs.escape_decode(bytes(content, 'utf-8'))[0].decode('utf-8')

          yield decoded

          time.sleep(1)

g = watch_task_logs()

for log_line in g:
    print(log_line)

[potiuk]

Cool. So I would encourage you to improve the description of the API with your findings @dirtyValera

I think the API description is quite clear, but if you think it might benefit from clarifications/improvements, you are absolutely welcome to make PRs and update the description. This is how Airflow is created by > 2500 contributors. They find out that there is a missing or misleading description, spend their time on figuring out how it works and contribute back improvements (for example better documentation).

This is how airflow has been created and continues to be updated. And this is how it can be used for free, because most of those people who contribute do it in their free time and contribute back in order to help others. I think it would be great if you can contribute back your findings. Our API description is https://github.com/apache/airflow/blob/main/airflow/api_connexion/openapi/v1.yaml and you can easily contribute improvements there.

Also a side comment: rather than using REST api of airflow, I would heartily recommend a dedicated remote logging and configure it (Stackdriver for example) : https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-tasks.html#logging-for-tasks - those logging solutions are usually way better in handling all the different APIs, storage, filtering cases and provide you many more capabilities.