how to reset the status of the job when the container get terminated?

[leonelea]

Our project encountered an issue, the context is as follows:
Our project has some data synchronization jobs. These jobs are deployed in a cluster built by another team. The cluster uses k8s. Recently, the container where the job is located always disappears suddenly during the execution process, resulting in the status of the executing job being unable to be updated (execution status is unknown, step execution is executing), which leads to an error like below:
A job execution for this job is already running: JobExecution: id=9211, version=1, startTime=2024-01-09 02:03:04.677, endTime=null, lastUpdated=2024-01-09 02:03:04.678, status=STARTED, exitStatus=exitCode=UNKNOWN;
After investigation, it was found that the job was terminated because the node where the container was located was terminated, and it would take a long time for the team to fix it.
So, from our own side, is there any elegant way to reset the status of these jobs that executed normally but terminated abnormally because the container was killed, so that they can run normally again?

[hemeda3]

same issue here any solution?

[fmbenhassine]

When a job is killed abruptly, Spring Batch won't have a chance to update its status in the Job repository, so the status is stuck at STARTED. On restart, it is impossible to distinguish between a job that is effectively running and a job that has been killed abruptly (in both cases, the status is STARTED). You need to reset the status to FAILED and the END_TIME to a non null value manually in the database. There is a note about this case in the docs here.

I also wrote about this case with an example in the blog post here: https://spring.io/blog/2021/01/27/spring-batch-on-kubernetes-efficient-batch-processing-at-scale#4-gracefulabrupt-shutdown-implication.

Let me know if this helps.

[Walnussbaer]

Note that updating the status and end_time needs to be done for the job instance inside public.batch_job_execution as well as for the step(s) of the job inside public.batch_step_execution. Doing this only for the job execution instance is not enough!

[fmbenhassine]

That's right, here is the relevant part from the blog post I mentioned in the answer:

> update BATCH_JOB_EXECUTION set status = 'FAILED', END_TIME = '2020-01-15 10:10:28.235' where job_execution_id = X;
> update BATCH_STEP_EXECUTION set status = 'FAILED' where job_execution_id = X and step_name='failed step name';

By the way, we introduced a new API in v6 to allow you to recover a job execution without having to manually update the database (and this API will update the status and end time of both job and step execution as needed), see #4876

[fmbenhassine]

I believe my last comment answers the question, so I am closing this discussion; But If you need more help on this, please add a comment.

[fmbenhassine]

In addition to the new JobOperator#recover API that is designed for abrupt shutdowns, we have introduced some updates in v6 to support graceful shutdowns as well (see #4965, #4023, #5028). It is now possible to register a job execution shutdown hook provided by Spring Batch to gracefully stop a job execution. There is a complete example here: https://github.com/spring-projects/spring-batch/tree/main/spring-batch-samples/src/main/java/org/springframework/batch/samples/shutdown