Fix multiple validation jobs

[sjanssen2]

Addressing #3379:
With the proposed changes, both validation jobs of a deblur command successfully run through in my local setup. I suspect two mistakes of the current code:

1. maybe a copy&paste error of lines 1108ff as they seem to be identical to 1091ff
   

   qiita/qiita_db/processing_job.py

   Lines 1091 to 1114 in 5ab3ebb

   	p = Process(target=launcher['function'],
   	args=(plugin_env_script,
   	plugin_start_script,
   	url,
   	self.id,
   	job_dir))
   	p.start()
   	job_id = p.pid
   	if dependent_jobs_list:
   	# for now, treat dependents as independent when
   	# running locally. This means they will not be
   	# organized into n 'queues' or 'chains', and
   	# will all run simultaneously.
   	for dependent in dependent_jobs_list:
   	p = Process(target=launcher['function'],
   	args=(plugin_env_script,
   	plugin_start_script,
   	url,
   	self.id,
   	job_dir))
   	p.start()

2. even though the dependent jobs where "submitted" as child processes, they were not "registered" by Qiita's DB, such that Qiita did not "poll" for successful execution and thus could never set status to "success" of these job(s) even though the actual process did run through and finished with exit 0

[coveralls]

coverage: 92.376% (-0.01%) from 92.389%
when pulling 769e382 on jlab:fix_multiple_validation_jobs
into d49339f on qiita-spots:dev.

[antgonza]

Got it, thank you for the fix. Do you think it will be possible to add a retroactive test that brakes without the fix and passes now with it?

[sjanssen2]

We would need to have a plugin that generates multiple artifacts as a test case. Adding the environment, registering all plugins to be "active", executing the workflow up to this point runs approx 15min on my laptop. Not sure if we want to invest that much compute in github actions.
Maybe we can create a mock plugin that just "touches" files for two ore more artifacts?!

[antgonza]

That's right and after doing a quick search in the code base I think this is happening in test_processing_job.py ProcessingJobTest.test_complete_multiple_outputs. Maybe the only thing missing for the test is changing the value of qiita_config.job_scheduler_dependency_q_cn from 1 to 2; what do you think?

[sjanssen2]

I think the problem with the existing test is that it manually sets the state of the dependent job here to "running":

qiita/qiita_db/test/test_processing_job.py

Line 347 in 5ab3ebb

obs2._set_status('running')

which in a running qiita instance should indirectly be done by the submit() function. But the code could be a primer?!

[sjanssen2]

The existing test does not seem to actually execute jobs. Creation of the biom files is faked here:

qiita/qiita_db/test/test_processing_job.py

Lines 309 to 319 in 5ab3ebb

	fd, fp1 = mkstemp(suffix="_table.biom")
	self._clean_up_files.append(fp1)
	close(fd)
	with open(fp1, 'w') as f:
	f.write('\n')
	fd, fp2 = mkstemp(suffix="_table.biom")
	self._clean_up_files.append(fp2)
	close(fd)
	with open(fp2, 'w') as f:
	f.write('\n')

Since the submit() function is creating true child processes, I am not sure if we really want to invest in writing a mock to test this behaviour. Maybe an alternative is to just test the DB fingerprint of what a submit would change?! But than it is not truly testing submit. I don't see an easy way forward :-/

[antgonza]

OK, thank you for checking. I agree with your assessment.