Build errors and unit failures in CMSSW_12_0_X_2021-07-09-2300

[qliphy]

Since CMSSW_12_0_X_2021-07-09-2300 there are build errors and unit failures in the default ( slc7_amd64_gcc900)

e.g.
https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc7_amd64_gcc900/CMSSW_12_0_X_2021-07-09-2300/Validation/HGCalValidation

Building big object file tmp/slc7_amd64_gcc900/src/Validation/HGCalValidation/src/ValidationHGCalValidation/bigobj/ValidationHGCalValidation.obj
/data/cmsbld/jenkins/workspace/build-any-ib/w/slc7_amd64_gcc900/external/gcc/9.3.0/bin/ld -r -z muldefs -o tmp/slc7_amd64_gcc900/src/Validation/HGCalValidation/src/ValidationHGCalValidation/bigobj/ValidationHGCalValidation.obj tmp/slc7_amd64_gcc900/src/Validation/HGCalValidation/src/ValidationHGCalValidation/bigobj/a/ValidationHGCalValidation_xr.cc.o tmp/slc7_amd64_gcc900/src/Validation/HGCalValidation/src/ValidationHGCalValidation/bigobj/HGVHistoProducerAlgo.cc.o
Copying tmp/slc7_amd64_gcc900/src/Validation/HGCalValidation/src/ValidationHGCalValidation/libValidationHGCalValidation.so to productstore area:
Checking EDM Class Version for src/Validation/HGCalValidation/src/classes_def.xml in libValidationHGCalValidation.so
/usr/bin/echo >> Skipped running EDM Class Version test
File "/usr/bin/echo", line 1
SyntaxError: Non-UTF-8 code starting with '\xff' in file /usr/bin/echo on line 2, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
Suggestion: You can run 'scram build updateclassversion' to generate src/Validation/HGCalValidation/src/classes_def.xml.generated with updated ClassVersion
gmake: *** [tmp/slc7_amd64_gcc900/src/Validation/HGCalValidation/src/ValidationHGCalValidation/libValidationHGCalValidation.so] Error 1
Leaving library rule at Validation/HGCalValidation

And
https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc900/CMSSW_12_0_X_2021-07-09-2300/unitTestLogs/DQMOffline/Configuration#/244-244

Task failed.
Traceback (most recent call last):
File "/cvmfs/cms-ib.cern.ch/nweek-02688/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-09-1100/bin/slc7_amd64_gcc900/cmsswSequenceInfo.py", line 532, in
processseqs(seqset)
File "/cvmfs/cms-ib.cern.ch/nweek-02688/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-09-1100/bin/slc7_amd64_gcc900/cmsswSequenceInfo.py", line 288, in processseqs
for it in t.get(): # should only be one
File "/cvmfs/cms-ib.cern.ch/nweek-02688/slc7_amd64_gcc900/external/python3/3.8.2-cms/lib/python3.8/multiprocessing/pool.py", line 768, in get
raise self._value
File "/cvmfs/cms-ib.cern.ch/nweek-02688/slc7_amd64_gcc900/external/python3/3.8.2-cms/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/cvmfs/cms-ib.cern.ch/nweek-02688/slc7_amd64_gcc900/external/python3/3.8.2-cms/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/cvmfs/cms-ib.cern.ch/nweek-02688/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-09-1100/bin/slc7_amd64_gcc900/cmsswSequenceInfo.py", line 272, in
tasks = [stp.map_async(lambda seq: (seq, inspectsequence(seq)), [seq]) for seq in seqs]
File "/cvmfs/cms-ib.cern.ch/nweek-02688/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-09-1100/bin/slc7_amd64_gcc900/cmsswSequenceInfo.py", line 65, in inspectsequence
subprocess.check_call(driverargs, cwd=wd, stdout=2) # 2: STDERR
File "/cvmfs/cms-ib.cern.ch/nweek-02688/slc7_amd64_gcc900/external/python3/3.8.2-cms/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmsDriver.py', 'step3', '--conditions', 'auto:run2_data', '-s', 'VALIDATION', '--process', 'DUMMY', '--eventcontent', 'DQM', '--datatier', 'DQMIO', '--customise_commands', 'process.Tracer = cms.Service("Tracer")', '--filein', 'file:///data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_12_0_X_2021-07-09-2300/tmp/004D960A-EA4C-E811-A908-FA163ED1F481.root', '-n', '0', '--python_filename', 'cmssw_cfg.py', '--no_exec']' returned non-zero exit status 1.

[cmsbuild]

A new Issue was created by @qliphy Qiang Li.

@Dr15Jones, @perrotta, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[qliphy]

Could it be due to recent python2 -> python3 migration?

e.g. cms-sw/cmsdist#7106 was merged recently.

[qliphy]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[smuzaffar]

@qliphy , cms-sw/cmsdist#7106 was merged in CMSSW_12_0_X_2021-07-09-1100 IB and unit tests ran fine in that IB. I think unit tests failue are due to one of the PRs merged in CMSSW_12_0_X_2021-07-09-2300 .

For build errors, I see that errors are only in patch IBs. Lets which for last night's 00h00 IB ( which will be a full IB) to see if full re-build has fixed the build issue

[smuzaffar]

assign dqm
@jfernan2 , could the change in #34302 now confusing unit tests in DQMOffline/Configuration ?

[cmsbuild]

New categories assigned: dqm

@jfernan2,@kmaeshima,@rvenditti,@andrius-k,@ErnestaP,@ahmad3213 you have been requested to review this Pull request/Issue and eventually sign? Thanks

[smuzaffar]

I confirm that reverting #34302 fixes TestDQMOfflineConfiguration_* tests ( at least TestDQMOfflineConfiguration_110 ran successfully). So if #34302 contains a valid change then DQMOffline/Configuration also need update

[a]

/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_0_X_2021-07-10-1100/bin/slc7_amd64_gcc900/edmConfigDump:6: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
Analyzing 10 seqs...
+ sqlite3 sequences110.db
+ grep EDAnalyzer legacymodules-sequences110.db.txt
+ exit 0

---> test TestDQMOfflineConfiguration_110 succeeded
TestTime:440
^^^^ End Test TestDQMOfflineConfiguration_110 ^^^^