Does someone know how to get the Dataproc resource links for all DataProcPigOperator tasks?

[xuyangwang0825]

There is a dataproc link for each pig task:

The format of that link is like this https://console.cloud.google.com/dataproc/jobs/{resource_value}
e.g. https://console.cloud.google.com/dataproc/jobs/sample_pig_20230814_dadb2bb2

I want to get all those links for all tasks. I have tried the following code to get the resource value:

  ti = kwargs['ti']
  conf = ti.xcom_pull(key='conf')
  resource_value = conf.get('resource')

But the resource parameter in conf will be overridden so that I can only get the last one for the last task.

Does someone know how to get the Dataproc resource links for all DataProcPigOperator tasks?
Any opinions will be valuable! Thanks!

[xuyangwang0825]

It can be access by the following way.

conf = ti.xcom_pull(dag_id='dag_id', task_ids='task_id', key='conf')