Update LLVM to 11.1 #6691

mrodozov · 2021-03-01T23:02:25Z

Resolves: #6666
This builds the LLVM external

cmsbuild · 2021-03-01T23:02:43Z

A new Pull Request was created by @mrodozov (Mircho Rodozov) for branch IB/CMSSW_11_3_X/master.

@cmsbuild, @smuzaffar, @mrodozov can you please review it and eventually sign? Thanks.
cms-bot commands are listed here

mrodozov · 2021-03-02T00:00:17Z

please test

cmsbuild · 2021-03-02T03:46:37Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c0a525/13182/summary.html
COMMIT: e2ab0c3
CMSSW: CMSSW_11_3_X_2021-03-01-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6691/13182/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

call_subprocess(
File "/build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc900/external/py3-pip/20.3.3/lib/python3.8/site-packages/pip/_internal/utils/subprocess.py", line 240, in call_subprocess
raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc900/external/python3/3.8.2-bcolbf/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/external/py2-llvmlite/0.33.0-00294e2e5be24bf0759475cbe506703a/cmsdist-tmp/pip-req-build-92zvxjsw/setup.py'"'"'; __file__='"'"'/build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/external/py2-llvmlite/0.33.0-00294e2e5be24bf0759475cbe506703a/cmsdist-tmp/pip-req-build-92zvxjsw/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/external/py2-llvmlite/0.33.0-00294e2e5be24bf0759475cbe506703a/cmsdist-tmp/pip-record-n8fwnlri/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/BUILDROOT/00294e2e5be24bf0759475cbe506703a/opt/cmssw/slc7_amd64_gcc900/external/py2-llvmlite/0.33.0-00294e2e5be24bf0759475cbe506703a/include/python3.8/llvmlite Check the logs for full command output.
Removed build tracker: '/build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/external/py2-llvmlite/0.33.0-00294e2e5be24bf0759475cbe506703a/cmsdist-tmp/pip-req-tracker-1xsgj6tj'
error: Bad exit status from /build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.kYud8a (%build)


RPM build errors:
Macro %rpmbuild_libdir defined but not used within scope
Bad exit status from /build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.kYud8a (%build)

cmsbuild · 2021-03-08T16:19:25Z

Pull request #6691 was updated.

mrodozov · 2021-03-08T16:21:07Z

please test
hopefully full toolconf will build.

mrodozov · 2021-03-08T17:19:43Z

please test for slc7_aarch64_gcc9

mrodozov · 2021-03-08T17:19:57Z

please test for slc7_ppc64le_gcc9

smuzaffar · 2021-03-08T17:23:06Z

@mrodozov , you need to issue please test for slc7_aarch64_gcc9 again. In one iteration bot only recognize the last command. Normally you have to wait for the bot to ack your last command before issuing the next one :-)

mrodozov · 2021-03-08T17:25:31Z

please test for slc7_aarch64_gcc9

cmsbuild · 2021-03-08T19:57:58Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c0a525/13338/summary.html
COMMIT: de88813
CMSSW: CMSSW_11_3_X_2021-03-08-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/6691/13338/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test test-clang-tidy had ERRORS
---> test testLLVMLite had ERRORS

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-c0a525/34634.0_TTbar_14TeV+2026D76+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-c0a525/34834.999_TTbar_14TeV+2026D76PU_PMXS1S2PR+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU

Summary:

No significant changes to the logs found
Reco comparison results: 3 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2849195
DQMHistoTests: Total failures: 10
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2849162
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 37 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 160 log files, 37 edm output root files, 38 DQM output files

cmsbuild · 2021-03-08T20:15:41Z

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c0a525/13342/summary.html
COMMIT: de88813
CMSSW: CMSSW_11_3_X_2021-03-07-2300/slc7_ppc64le_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/6691/13342/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test testLLVMLite had ERRORS
---> test test-clang-tidy had ERRORS

RelVals

11634.91111634.911_TTbar_14TeV+2021_DD4hep+TTbar_14TeV_TuneCP5_GenSim+Digi+Reco+HARVEST+ALCA/step1_TTbar_14TeV+2021_DD4hep+TTbar_14TeV_TuneCP5_GenSim+Digi+Reco+HARVEST+ALCA.log

cmsbuild · 2021-03-08T23:11:32Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c0a525/13344/summary.html
COMMIT: de88813
CMSSW: CMSSW_11_3_X_2021-03-08-1100/slc7_aarch64_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/6691/13344/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test test-clang-tidy had ERRORS
---> test testLLVMLite had ERRORS

mrodozov · 2021-03-09T11:32:25Z

please test with cms-sw/cmssw#33110

smuzaffar · 2021-03-13T08:30:23Z

please test

cmsbuild · 2021-03-13T17:52:08Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c0a525/13481/summary.html
COMMIT: 6b600df
CMSSW: CMSSW_11_3_X_2021-03-12-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/6691/13481/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2635087
DQMHistoTests: Total failures: 12
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2635052
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 36 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 155 log files, 37 edm output root files, 37 DQM output files

silviodonato · 2021-03-15T09:51:03Z

@cms-sw/externals-l2 can we merge this PR with cms-sw/cmssw#33110? Do we need cms-sw/cmssw#33136 too?

mrodozov · 2021-03-15T10:04:18Z

cms-sw/cmssw#33110 and cms-sw/cmssw#33136 can and should be merged on their own . 33110 was updated to work with both llvm10 and 11 . I'll quote this (from above comments)

looks like new llvm generate few more warnings for CLANG build ( https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c0a525/13406/build-logs/ ). Once ready we should first get llvm 11. 1 only CLANG builds

for which we have #6720

smuzaffar · 2021-03-15T15:43:50Z

please test for CMSSW_11_3_CLANG_X

smuzaffar · 2021-03-15T15:46:29Z

@mrodozov , @silviodonato , once we have cms-sw/cmssw#33171 merged then all the new build warnings should go away and we can merge it directly in the master IB.

cmsbuild · 2021-03-16T00:51:00Z

-1

Failed Tests: HeaderConsistency
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c0a525/13524/summary.html
COMMIT: 6b600df
CMSSW: CMSSW_11_3_CLANG_X_2021-03-14-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6691/13524/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 4274 lines to the logs
Reco comparison results: 17928 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2635087
DQMHistoTests: Total failures: 79903
DQMHistoTests: Total nulls: 7
DQMHistoTests: Total successes: 2555155
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.012 KiB( 36 files compared)
DQMHistoSizes: changed ( 136.874 ): 0.016 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 155 log files, 37 edm output root files, 37 DQM output files

smuzaffar · 2021-03-16T08:05:51Z

+externals
looks good to go, CLANG IBs will have an extra warnings which should be fixed when cms-sw/cmssw#33171 is merged

cmsbuild · 2021-03-16T08:06:13Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_11_3_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

davidlange6 · 2021-03-17T12:44:00Z

Hi @smuzaffar @mrodozov - did you look at clang-tidy changes due to this upgrade? I've noticed a few unexpected changes in the last day (eg, here was a change needed after an unrelated commit was pushed yesterday cms-sw/cmssw#33189

smuzaffar · 2021-03-17T13:16:33Z

yes I did run clang-tidy and format for this, see the following comments
#6691 (comment)
#6691 (comment)

Most of these are already reported by llvm 10 too.

davidlange6 · 2021-03-17T13:33:48Z

Hum, then I don’t understand the need for cms-sw/cmssw@04a1e0b That came one the second code-checks run of the PR on a file that wasn’t touched during the PR review.. Hopefully isolated:)

…

On Mar 17, 2021, at 2:16 PM, Malik Shahzad Muzaffar ***@***.***> wrote: yes I did run clang-tidy and format for this, see the following comments #6691 (comment) #6691 (comment) Most of these are already reported by llvm 10 too. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

smuzaffar · 2021-03-17T13:51:02Z

cms-sw/cmssw@04a1e0b is code format change. Looks like llvm 11 has improved on few formats . See the below links for code format changes for full cmssw

LLVM 10: https://cmssdt.cern.ch/SDT/jenkins-artifacts/jenkins-test-code-format/CMSSW_11_3_X_2021-03-08-2300/59/code-format.patch (61 changes)
LLVM 11: https://cmssdt.cern.ch/SDT/jenkins-artifacts/jenkins-test-code-format/CMSSW_11_3_X_2021-03-08-1100/63/code-format.patch (112 changes)

smuzaffar · 2021-03-17T13:55:04Z

it could be if google code style has updated few defaults. Let me see the differences

makortel · 2021-03-17T13:59:54Z

Would it make sense to have an automated reformatting campaign? (after double-checking the style definition itself)

smuzaffar · 2021-03-17T14:09:47Z

here is the code format changes which is reasonable. I would not recommend to change the google default. After all it is not a big bang changes. Only 50 new changes.

--- l10.txt     2021-03-17 15:02:22.291374654 +0100
+++ l11.txt     2021-03-17 15:03:58.029633511 +0100
@@ -1,19 +1,21 @@
 >> Local Products Rules ..... started
 >> Local Products Rules ..... done
-clang-format   -dump-config /build/muz/CMSSW_11_3_X_2021-03-07-0000/src/.clang-format
+clang-format   -dump-config /build/muz/CMSSW_11_3_X_2021-03-16-2300/src/.clang-format
 ---
 Language:        Cpp
 AccessModifierOffset: -2
 AlignAfterOpenBracket: Align
 AlignConsecutiveMacros: false
 AlignConsecutiveAssignments: false
+AlignConsecutiveBitFields: false
 AlignConsecutiveDeclarations: false
 AlignEscapedNewlines: Left
-AlignOperands:   true
+AlignOperands:   Align
 AlignTrailingComments: true
 AllowAllArgumentsOnNextLine: true
 AllowAllConstructorInitializersOnNextLine: true
 AllowAllParametersOfDeclarationOnNextLine: true
+AllowShortEnumsOnASingleLine: true
 AllowShortBlocksOnASingleLine: Never
 AllowShortCaseLabelsOnASingleLine: false
 AllowShortFunctionsOnASingleLine: All
@@ -29,7 +31,7 @@
 BraceWrapping:
   AfterCaseLabel:  false
   AfterClass:      false
-  AfterControlStatement: false
+  AfterControlStatement: Never
   AfterEnum:       false
   AfterFunction:   false
   AfterNamespace:  false
@@ -39,6 +41,8 @@
   AfterExternBlock: false
   BeforeCatch:     false
   BeforeElse:      false
+  BeforeLambdaBody: false
+  BeforeWhile:     false
   IndentBraces:    false
   SplitEmptyFunction: true
   SplitEmptyRecord: true
@@ -85,10 +89,13 @@
 IncludeIsMainRegex: '([-_](test|unittest))?$'
 IncludeIsMainSourceRegex: ''
 IndentCaseLabels: true
+IndentCaseBlocks: false
 IndentGotoLabels: true
 IndentPPDirectives: None
+IndentExternBlock: AfterExternBlock
 IndentWidth:     2
 IndentWrappedFunctionNames: false
+InsertTrailingCommas: None
 JavaScriptQuotes: Leave
 JavaScriptWrapImports: true
 KeepEmptyLinesAtTheStartOfBlocks: false
@@ -98,6 +105,7 @@
 NamespaceIndentation: All
 ObjCBinPackProtocolList: Never
 ObjCBlockIndentWidth: 2
+ObjCBreakBeforeNestedBlockParam: true
 ObjCSpaceAfterProperty: false
 ObjCSpaceBeforeProtocolList: true
 PenaltyBreakAssignment: 2
@@ -135,6 +143,8 @@
       - PARSE_TEXT_PROTO
       - ParseTextOrDie
       - ParseTextProtoOrDie
+      - ParseTestProto
+      - ParsePartialTestProto
     CanonicalDelimiter: ''
     BasedOnStyle:    google
 ReflowComments:  false
@@ -166,5 +176,9 @@
 TabWidth:        8
 UseCRLF:         false
 UseTab:          Never
+WhitespaceSensitiveMacros:
+  - STRINGIZE
+  - PP_STRINGIZE
+  - BOOST_PP_STRINGIZE

smuzaffar · 2021-03-17T14:17:17Z

Would it make sense to have an automated reformatting campaign? (after double-checking the style definition itself)

Yes, it is about time to do one. Although for default clang-tidy and format , we have 120 packages (194 files) which needs format/tidy updates but we can enable cms-hanlde check too (needs 6.5K changes)

makortel · 2021-03-17T14:32:05Z

I would do the cms-handle separately as it is conceptually different from "changes from LLVM upgrade" (and try it out on a few packages first to get feedback like we did when we started to use clang-tidy).

Update LLVM to 11.1

e2ab0c3

cmsbuild added externals-pending orp-pending pending-signatures tests-pending labels Mar 1, 2021

cmsbuild added tests-started and removed tests-pending labels Mar 2, 2021

cmsbuild added tests-rejected and removed tests-started labels Mar 2, 2021

Add new patch, modify existing and say goodbye to llvmlite py2

de88813

cmsbuild added tests-pending and removed tests-rejected labels Mar 8, 2021

cmsbuild added tests-started and removed tests-pending labels Mar 8, 2021

cmsbuild added tests-rejected and removed tests-started labels Mar 8, 2021

mrodozov mentioned this pull request Mar 9, 2021

Add EventSetupRecord::get checker cms-externals/llvm-project#10

Merged

cmsbuild added the tests-pending label Mar 13, 2021

cmsbuild added tests-started and removed tests-pending labels Mar 13, 2021

cmsbuild added tests-approved and removed tests-started labels Mar 13, 2021

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Mar 16, 2021

smuzaffar merged commit 4ae5869 into cms-sw:IB/CMSSW_11_3_X/master Mar 16, 2021

smuzaffar mentioned this pull request Mar 16, 2021

LLVM 11 in CLANG IB only #6720

Closed

mrodozov deleted the update-llvm-11.1 branch March 17, 2021 13:22

Update LLVM to 11.1 #6691

Update LLVM to 11.1 #6691

Uh oh!

Conversation

mrodozov commented Mar 1, 2021

Uh oh!

cmsbuild commented Mar 1, 2021

Uh oh!

mrodozov commented Mar 2, 2021

Uh oh!

cmsbuild commented Mar 2, 2021

External Build

Uh oh!

cmsbuild commented Mar 8, 2021

Uh oh!

mrodozov commented Mar 8, 2021

Uh oh!

mrodozov commented Mar 8, 2021

Uh oh!

mrodozov commented Mar 8, 2021

Uh oh!

smuzaffar commented Mar 8, 2021

Uh oh!

mrodozov commented Mar 8, 2021

Uh oh!

cmsbuild commented Mar 8, 2021

Unit Tests

Comparison Summary

Uh oh!

cmsbuild commented Mar 8, 2021

Unit Tests

RelVals

Uh oh!

cmsbuild commented Mar 8, 2021

Unit Tests

Uh oh!

mrodozov commented Mar 9, 2021

Uh oh!

smuzaffar commented Mar 13, 2021

Uh oh!

cmsbuild commented Mar 13, 2021

Comparison Summary

Uh oh!

silviodonato commented Mar 15, 2021

Uh oh!

mrodozov commented Mar 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented Mar 15, 2021

Uh oh!

smuzaffar commented Mar 15, 2021

Uh oh!

cmsbuild commented Mar 16, 2021

Comparison Summary

Uh oh!

smuzaffar commented Mar 16, 2021

Uh oh!

cmsbuild commented Mar 16, 2021

Uh oh!

davidlange6 commented Mar 17, 2021

Uh oh!

smuzaffar commented Mar 17, 2021

Uh oh!

davidlange6 commented Mar 17, 2021 via email

Uh oh!

smuzaffar commented Mar 17, 2021

Uh oh!

smuzaffar commented Mar 17, 2021

Uh oh!

makortel commented Mar 17, 2021

Uh oh!

smuzaffar commented Mar 17, 2021

Uh oh!

smuzaffar commented Mar 17, 2021

Uh oh!

makortel commented Mar 17, 2021

Uh oh!

Reviewers

Assignees

Labels

mrodozov commented Mar 15, 2021 •

edited

Loading