[JENKINS-52165] Stabilize and better test USE_WATCHING

[jglick]

JENKINS-52165: the performance benefits seem important. Have not yet tried to measure impact on (controller) CPU or Remoting channel bandwidth, but can use inotify to verify this allows a long-running sh step with copious output to only write to log (and maybe log-index), whereas in currently default pull mode there are constant writes to program.dat (via an atomic temp file) due to

workflow-durable-task-step-plugin/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java

Lines 599 to 601 in 6f24fa9

	} else { // legacy mode
	if (controller.writeLog(workspace, listener.getLogger())) {
	getContext().saveState();

which would otherwise be limited to step transitions.

Downstream of jenkinsci/workflow-api-plugin#294 and jenkinsci/durable-task-plugin#186.

• test failures
• pick up Catch errors in TaskListenerDecorator.decorate even when wrapped with merge workflow-api-plugin#290 & Unnecessary use of Guava in StatusCheck durable-task-plugin#185
• handle [JENKINS-68080] DurableTaskStep.HandlerImpl.printStreamDelegate requires --add-opens #212 more cleanly
• check PCT ([JENKINS-52165] Pretesting USE_WATCHING bom#2246 and CloudBees CI)
• log output sometimes lost across restarts (caught in CloudBees CI proprietary tests)
• test against pipeline-cloudwatch-logs
• test against KubernetesPipelineOverridenNamespaceTest
• parameterize relevant tests to run with and without watching mode
• [JENKINS-52165] Stabilize and better test USE_WATCHING #323 (comment)
• have at least some Windows test coverage
• try Add forkCount argument to buildPlugin jenkins-infra/pipeline-library#707

[dwnusbaum]

I have been following this off and on in the background. I have not attempted to fully trace the reasoning behind each individual change, but I know you have tested things reasonably thoroughly, and I do not see anything that looks obviously problematic.