Skip to content

Commit 3a7a110

Browse files
Java SDK: Let a task stop through onKill when Airflow asks it to (#75538)
* Reapply "Make Java SDK context signal-aware (#74690)" Signed-off-by: PoAn Yang <payang@apache.org> * Fix Java tasks hanging after their first request to the supervisor Signed-off-by: PoAn Yang <payang@apache.org> * Java SDK: Let a task stop through onKill when Airflow asks it to Signed-off-by: PoAn Yang <payang@apache.org> * Java SDK: Throw when a non-stoppable task checks for a stop request Signed-off-by: PoAn Yang <payang@apache.org> * Java SDK: Make log calls asynchronous and tighten stop checks Signed-off-by: PoAn Yang <payang@apache.org> * Java SDK: Prevent hangs when log lines pile up Signed-off-by: PoAn Yang <payang@apache.org> * Java SDK: Make the log flush timeout a parameter and shorten comments Signed-off-by: PoAn Yang <payang@apache.org> * Java SDK: Limit the final log flush and print late warnings to stderr Signed-off-by: PoAn Yang <payang@apache.org> * Java SDK: Handle both non-retryable exceptions Signed-off-by: PoAn Yang <payang@apache.org> --------- Signed-off-by: PoAn Yang <payang@apache.org>
1 parent 0fb56a6 commit 3a7a110

23 files changed

Lines changed: 3900 additions & 60 deletions

File tree

‎airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -748,6 +748,93 @@ the defaults of the Dag's params, and the conf of a scheduled run is empty. For
748748
accepts and the cases where it throws, see ``DagRun.getConf`` in the `Java SDK API reference
749749
<https://airflow.apache.org/docs/java-sdk/stable/>`__.
750750

751+
.. _java-sdk/cancellation:
752+
753+
Stopping when Airflow asks
754+
~~~~~~~~~~~~~~~~~~~~~~~~~~
755+
756+
Airflow asks a running task to stop when, for example, a user marks its task instance as failed. The
757+
supervisor, the Airflow process that started the task's JVM, then sends the JVM ``SIGTERM``. If the JVM
758+
is still running after a grace period, the supervisor sends ``SIGKILL``. When a user marks the task
759+
instance failed or success, or clears it, the grace period is
760+
:ref:`core.killed_task_cleanup_time <config:core__killed_task_cleanup_time>` seconds.
761+
762+
By default the JVM handles the signal as it does without the SDK: shutdown hooks run and the JVM exits.
763+
A process the task started gets the same signals, because the supervisor signals the JVM's whole process
764+
group. So a task does not need to stop a process that exits on ``SIGTERM`` and stays in that group.
765+
766+
To stop anything else, such as a job in an external system, implement ``StoppableTask``. This interface
767+
adds ``onKill``, the Java counterpart of an operator's ``on_kill``:
768+
769+
.. code-block:: java
770+
771+
public class ReportTask implements StoppableTask {
772+
private final ReportService service = new ReportService();
773+
private volatile String jobId;
774+
775+
@Override
776+
public void execute(Context context, Client client) throws Exception {
777+
jobId = service.submit();
778+
if (context.isCancelled()) {
779+
service.cancel(jobId);
780+
throw new IllegalStateException("Airflow asked the task to stop");
781+
}
782+
service.waitFor(jobId); // Throws if the job fails or is cancelled.
783+
}
784+
785+
@Override
786+
public void onKill(Context context) throws Exception {
787+
if (jobId != null) {
788+
service.cancel(jobId);
789+
}
790+
}
791+
}
792+
793+
With annotations, mark the method that stops the task with ``@Airflow.OnKill`` and name the task. The
794+
method can take a ``Context``, and the annotation processor generates a ``StoppableTask`` that calls it on
795+
the same object as the task method:
796+
797+
.. code-block:: java
798+
799+
private final ReportService service = new ReportService();
800+
private volatile String jobId;
801+
802+
@Airflow.Task(id = "build_report", retries = 2)
803+
public void buildReport(Context context) throws Exception {
804+
jobId = service.submit();
805+
if (context.isCancelled()) {
806+
service.cancel(jobId);
807+
throw new IllegalStateException("Airflow asked the task to stop");
808+
}
809+
service.waitFor(jobId); // Throws if the job fails or is cancelled.
810+
}
811+
812+
@Airflow.OnKill(task = "build_report")
813+
public void stopReport() throws Exception {
814+
if (jobId != null) {
815+
service.cancel(jobId);
816+
}
817+
}
818+
819+
After starting such a task, the SDK catches the first ``SIGTERM`` or ``SIGINT``, and the JVM keeps running.
820+
``Context.isCancelled()`` then returns true and the SDK calls ``onKill``. A task that checks for a stop request in a loop
821+
can leave ``onKill``, or its ``@Airflow.OnKill`` method, empty. In that loop, it can call
822+
``Context.awaitCancellation(timeout)`` instead of ``Thread.sleep``. In a task that is not a ``StoppableTask``,
823+
``isCancelled`` and ``awaitCancellation`` throw. ``StoppableTask`` in the
824+
`Java SDK API reference <https://airflow.apache.org/docs/java-sdk/stable/>`__ explains when the SDK calls ``onKill`` and
825+
what ``onKill`` must handle.
826+
827+
Two things differ from ``on_kill`` in Python. ``on_kill`` runs on the task's own thread. ``onKill`` runs on a
828+
separate thread while the task keeps running. And Python calls ``on_kill`` as soon as ``execution_timeout``
829+
passes, while the Java SDK has no timer for ``execution_timeout``. ``onKill`` runs when the supervisor sends
830+
``SIGTERM``, which is ``killed_task_cleanup_time`` seconds after the timeout.
831+
832+
If the SDK can catch neither ``SIGTERM`` nor ``SIGINT``, for example because the JVM runs with ``-Xrs``, the SDK
833+
fails a ``StoppableTask`` before the task starts. Airflow does not retry the task. If the SDK let the task start
834+
anyway, the task could begin work that ``onKill`` would never get the chance to stop. To run a ``StoppableTask``,
835+
remove ``-Xrs`` from the JVM options. A runtime image built with ``jlink`` also needs the ``jdk.unsupported``
836+
module.
837+
751838
.. _java-sdk/task-state-store:
752839

753840
Task state store

‎airflow-e2e-tests/java-test-bundle/build.gradle‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ repositories {
2727
}
2828

2929
dependencies {
30+
annotationProcessor("org.apache.airflow:airflow-sdk-processor:${projectVersion}")
3031
implementation("org.apache.airflow:airflow-sdk:${projectVersion}")
3132
implementation("org.apache.airflow:airflow-sdk-jpl:${projectVersion}")
3233
}

‎airflow-e2e-tests/java-test-bundle/src/java/org/apache/airflow/e2e/TestBundleBuilder.java‎

Lines changed: 55 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,15 +19,20 @@
1919

2020
package org.apache.airflow.e2e;
2121

22+
import static java.lang.System.Logger.Level.INFO;
23+
24+
import java.time.Duration;
2225
import java.util.List;
2326
import java.util.Map;
27+
import java.util.concurrent.CountDownLatch;
28+
import java.util.concurrent.TimeUnit;
2429
import org.apache.airflow.sdk.*;
2530
import org.jetbrains.annotations.NotNull;
2631

2732
/**
2833
* Bundle for the runner-behaviour E2E tests: deliberately broken task classes that exercise
29-
* instantiation failures, and tasks that round-trip Airflow Variables and task state through the
30-
* supervisor.
34+
* instantiation failures, tasks that round-trip Airflow Variables and task state through the
35+
* supervisor, and tasks that run until Airflow asks them to stop.
3136
*/
3237
public class TestBundleBuilder implements BundleBuilder {
3338
public static class MissingNoArgConstructor implements Task {
@@ -94,6 +99,50 @@ public void execute(@NotNull Context context, Client client) {
9499
}
95100
}
96101

102+
/**
103+
* Runs until Airflow asks it to stop. The task sees the request through its context. The E2E test
104+
* makes the request by marking the task instance failed.
105+
*/
106+
public static class WaitForStopRequest implements StoppableTask {
107+
private static final System.Logger log = System.getLogger(WaitForStopRequest.class.getName());
108+
109+
public void execute(@NotNull Context context, Client client) throws InterruptedException {
110+
log.log(INFO, "Waiting for a stop request");
111+
if (!context.awaitCancellation(Duration.ofMinutes(5))) {
112+
throw new IllegalStateException("No stop request arrived within five minutes");
113+
}
114+
log.log(INFO, "Stopping at Airflow's request");
115+
}
116+
117+
public void onKill(@NotNull Context context) {}
118+
}
119+
120+
/**
121+
* Blocks in a call that never looks at the context, so only the {@code @Airflow.OnKill} method can
122+
* release the task. The E2E test asks this task to stop the same way as it asks {@link
123+
* WaitForStopRequest}.
124+
*/
125+
public static class OnKillHandlers {
126+
private static final System.Logger log = System.getLogger(OnKillHandlers.class.getName());
127+
128+
private final CountDownLatch killed = new CountDownLatch(1);
129+
130+
@Airflow.TaskHandler(dag = "java_stop_request_on_kill", task = "block_until_killed")
131+
public void blockUntilKilled() throws InterruptedException {
132+
log.log(INFO, "Waiting for a stop request");
133+
if (!killed.await(5, TimeUnit.MINUTES)) {
134+
throw new IllegalStateException("onKill did not run within five minutes");
135+
}
136+
log.log(INFO, "Stopping at Airflow's request");
137+
}
138+
139+
@Airflow.OnKill(task = "block_until_killed")
140+
public void stop() {
141+
log.log(INFO, "onKill ran");
142+
killed.countDown();
143+
}
144+
}
145+
97146
@NotNull
98147
@Override
99148
public Iterable<DagDef> getDags() {
@@ -104,11 +153,13 @@ public Iterable<DagDef> getDags() {
104153
variableWrite.addTask("write_and_delete", WriteAndDeleteVariable.class);
105154
var taskState = new DagDef("java_task_state");
106155
taskState.addTask("roundtrip_task_state", RoundtripTaskState.class);
107-
return List.of(uninstantiable, variableWrite, taskState);
156+
var stopRequest = new DagDef("java_stop_request");
157+
stopRequest.addTask("wait_for_stop_request", WaitForStopRequest.class);
158+
return List.of(uninstantiable, variableWrite, taskState, stopRequest);
108159
}
109160

110161
public static void main(String[] args) {
111-
var bundle = new TestBundleBuilder().build();
162+
var bundle = new TestBundleBuilder().build().register(OnKillHandlers.class);
112163
Server.create(args).serve(bundle);
113164
}
114165
}

‎airflow-e2e-tests/java-test-bundle/src/resources/dags/java_test_dags.py‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,3 +60,27 @@ def java_task_state():
6060

6161

6262
java_task_state()
63+
64+
65+
@task.stub(queue="java-test")
66+
def wait_for_stop_request(): ...
67+
68+
69+
@dag(dag_id="java_stop_request")
70+
def java_stop_request():
71+
wait_for_stop_request()
72+
73+
74+
java_stop_request()
75+
76+
77+
@task.stub(queue="java-test")
78+
def block_until_killed(): ...
79+
80+
81+
@dag(dag_id="java_stop_request_on_kill")
82+
def java_stop_request_on_kill():
83+
block_until_killed()
84+
85+
86+
java_stop_request_on_kill()

‎airflow-e2e-tests/tests/airflow_e2e_tests/e2e_test_utils/clients.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -249,6 +249,14 @@ def get_task_instances(self, dag_id: str, run_id: str):
249249
endpoint=f"dags/{dag_id}/dagRuns/{run_id}/taskInstances",
250250
)
251251

252+
def set_task_instance_state(self, dag_id: str, run_id: str, task_id: str, state: str):
253+
"""Set the state of a task instance, as marking it in the UI does."""
254+
return self._make_request(
255+
method="PATCH",
256+
endpoint=f"dags/{dag_id}/dagRuns/{run_id}/taskInstances/{task_id}",
257+
json={"new_state": state},
258+
)
259+
252260
def get_hitl_detail(self, dag_id: str, run_id: str, task_id: str, map_index: int = -1):
253261
"""Get the Human-in-the-loop request of a task instance."""
254262
return self._make_request(

‎airflow-e2e-tests/tests/airflow_e2e_tests/java_sdk_tests/test_java_sdk_dag.py‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,12 @@
9898
_VARIABLE_WRITE_DESCRIPTION = "written by the Java SDK e2e test"
9999
_TASK_STATE_DAG_ID = "java_task_state"
100100
_TASK_STATE_TASK_ID = "roundtrip_task_state"
101+
_STOP_REQUEST_DAG_ID = "java_stop_request"
102+
_STOP_REQUEST_TASK_ID = "wait_for_stop_request"
103+
_ON_KILL_DAG_ID = "java_stop_request_on_kill"
104+
_ON_KILL_TASK_ID = "block_until_killed"
105+
_ON_KILL_EVENT = "onKill ran"
106+
_STOPPING_EVENT = "Stopping at Airflow's request"
101107

102108

103109
@dataclass
@@ -427,6 +433,65 @@ def test_scratch_variable_deleted_by_java_task_is_gone(self, variable_write_run:
427433
)
428434

429435

436+
class TestJavaSDKStopRequest:
437+
"""Verify that a running Java task can stop when Airflow asks it to.
438+
439+
Marking a running task instance failed makes the supervisor send the task's JVM SIGTERM. For a
440+
``StoppableTask``, the SDK keeps the JVM running and tells the task. ``WaitForStopRequest`` in the
441+
java-test-bundle fixture project returns from ``Context.awaitCancellation``. ``OnKillHandlers``
442+
there blocks until its ``@Airflow.OnKill`` method releases it. Each task then logs that it is
443+
stopping. A JVM that exited on the signal could not write that log line.
444+
"""
445+
446+
@pytest.mark.parametrize(
447+
("dag_id", "task_id", "expected_events"),
448+
[
449+
pytest.param(_STOP_REQUEST_DAG_ID, _STOP_REQUEST_TASK_ID, [_STOPPING_EVENT], id="context"),
450+
pytest.param(_ON_KILL_DAG_ID, _ON_KILL_TASK_ID, [_ON_KILL_EVENT, _STOPPING_EVENT], id="on-kill"),
451+
],
452+
)
453+
def test_running_task_stops_at_airflows_request(self, dag_id, task_id, expected_events):
454+
client = AirflowClient()
455+
run_id = client.trigger_dag(dag_id, json={"logical_date": datetime.now(UTC).isoformat()})[
456+
"dag_run_id"
457+
]
458+
run = _CompletedRun(client=client, dag_id=dag_id, run_id=run_id)
459+
460+
# The SDK catches the stop signal only once it has started the task, so mark the task instance
461+
# failed only after the task logs that it is waiting.
462+
deadline = time.monotonic() + _JAVA_TASK_TIMEOUT
463+
while (state := _get_task_state(client, dag_id, run_id, task_id)) != "running":
464+
assert state in {None, "scheduled", "queued"}, f"the task ended before it ran; state: {state!r}"
465+
assert time.monotonic() < deadline, f"the task never started; state: {state!r}"
466+
time.sleep(3)
467+
waiting, records = run.wait_for_log_record(
468+
task_id, 1, lambda r: r.get("event") == "Waiting for a stop request"
469+
)
470+
assert waiting is not None, (
471+
f"the task never started waiting; events seen: {[r.get('event') for r in records]}"
472+
)
473+
474+
client.set_task_instance_state(dag_id, run_id, task_id, "failed")
475+
476+
stopping, records = run.wait_for_log_record(task_id, 1, lambda r: r.get("event") == _STOPPING_EVENT)
477+
assert stopping is not None, (
478+
f"the task should see the request to stop; events seen: {[r.get('event') for r in records]}"
479+
)
480+
events = [r.get("event") for r in records if r.get("event") in {_ON_KILL_EVENT, _STOPPING_EVENT}]
481+
assert events == expected_events, f"the task should stop in this order; records: {records}"
482+
caught = [r.get("signal") for r in records if r.get("event") == "Asking the task to stop"]
483+
assert caught == ["SIGTERM"], f"the SDK should log the SIGTERM it caught; records: {records}"
484+
assert client.wait_for_dag_run(dag_id, run_id, timeout=_JAVA_TASK_TIMEOUT) == "failed"
485+
assert _get_task_state(client, dag_id, run_id, task_id) == "failed", (
486+
"Airflow should keep the state the test set"
487+
)
488+
489+
490+
def _get_task_state(client: AirflowClient, dag_id: str, run_id: str, task_id: str) -> str | None:
491+
task_instances = client.get_task_instances(dag_id, run_id).get("task_instances", [])
492+
return next((ti.get("state") for ti in task_instances if ti["task_id"] == task_id), None)
493+
494+
430495
class TestJavaSDKInterfaceExample:
431496
"""Verify tasks written against ``InputTask`` receive their bound arguments."""
432497

0 commit comments

Comments
 (0)