-Ofast and CPU reco timing

[silviodonato]

In the study of the CPU-GPU differences, we found out (again) that -Ofast creates differences between AMD and Intel in the electron reconstruction [1] (~1-2% of events passing the single electron trigger).
We asked PdmV to perform studies on the offline side [2].

Whatever will be the outcome of [2], I think we need to quantify the gain of -Ofast in the offline reconstruction, so we should measure the CPU timing of the reconstruction with and without the -Ofast flag (just deleting the lines with the flag in the BuildFile.xml, see [3]).

What does @cms-sw/reconstruction-l2 think about this? Can you perform this kind of study?

Thank you,
Silvio.

[1] https://its.cern.ch/jira/browse/PDMVRELVALS-159
[2] https://its.cern.ch/jira/browse/CMSHLT-2257
[3] https://cmssdt.cern.ch/lxr/search?%21v=CMSSW_12_5_X_2022-06-07-2300&_filestring=&_string=ofast-flag

cc @cms-sw/hlt-l2 @sanuvarghese

[cmsbuild]

A new Issue was created by @silviodonato Silvio Donato.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign reconstruction

[cmsbuild]

New categories assigned: reconstruction

@mandrenguyen,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks

[clacaputo]

Hi @silviodonato , I've performed this check beginning of July 2022.
Removing the -Ofast compiler option increase the reconstruction time by ~1% percent.

I’ve tested the effect in CMSSW_12_4_1_patch1, on vocms011 (Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz), both on production-like workflow 11834.21 and T0-like workflow 136.889:

• wf 11834.21: increase in reconstruction time of ~1.2%
• wf 136.889: increase in reconstruction time of ~1.1%

20 independent measurements were performed.

Do you need the same study in another 124X release?

[sanuvarghese]

This maybe a naive question : If the increase in reconstruction time is only ~1% and the removal of Ofast removes all the CPU-CPU differences, why is it still there?

[mandrenguyen]

@sanuvarghese A 1% increase in reconstruction time is not insignificant.
We're talking about the cost of an extra 1% of CPU not only for prompt and data reprocessing, but also all of the MC production. That has a real cost in terms of resources, which are under increased pressure these days.
We can certainly have a discussion about whether a 1% CPU-GPU difference is more or less important than a 1% difference in reconstruction time, but it's not as obvious as you seem to be implying.

Following discussion at the RECO meeting last Friday, I understand the nature of certain GPU algorithms makes them inherently unreproducible to some level, meaning even GPU-GPU comparisons will not be exact.
If we are going to have to live with these fluctuations anyway, it may not make sense to remove this compiler flag.
I'm not making the case either way, just trying to provide some extra info.

[makortel]

I'm curious if the "1-2 % differences in electron reconstruction" between AMD and Intel look like random differences or if they show any systematic effect?

[srlantz]

@sanuvarghese I certainly can't speak for reconstruction as a whole, but considering just the mkFit algorithm for Si strip tracking (which is what I've been involved with), certain parts of -Ofast are necessary to enable autovectorization of math functions (sin/cos) with gcc, and this can make a big difference to mkFit, since it is designed to maximize SIMD parallelism. For example, in recent tests on AMD Milan, when -Ofast is specified, I obtain ~25% faster single-core performance in the initial step that finds most of the tracks.*

However, in the case of mkFit, -Ofast is really overkill; we just need -O3 -ffast-math, or to be even more focused, -O3 -fno-math-errno -ffinite-math-only -fno-rounding-math -funsafe-math-optimization, which is the part that unlocks math vectorization via libmvec. In other words, mkFit's performance benefits from a subset of what is included in the -ffast-math umbrella option, which is itself a subset of -Ofast. I find that using the focused subset, as above, gives essentially identical performance to using -Ofast or -O3 -ffast-math. Note that this is all based on a generic AVX2 build that specifies -mavx2 -mfma.

So, maybe numerical differences between CPUs could be minimized by using a more focused set of options in place of -Ofast.

*Note: for testing purposes, I use a special standalone build of mkFit. Full physics validation is not performed, but I can confirm that expected numbers of tracks are being built.

[silviodonato]

@makortel from the latest measurements we see up to 8% difference for double electrons. Yes, I'm worried that there might be a systematic effect. This is something should be checked by PDMVRELVALS-159 , which is somehow related to #30044 (even though it is not strictly necessary)

[silviodonato]

@srlantz thanks for the info, so we could try to make a comparison between

1. -Ofast
2. -O3 -ffast-math
3. -O3 -fno-math-errno -ffinite-math-only -fno-rounding-math -funsafe-math-optimization
4. nothing

In this previous test made by Riccardo S, we see that 4) eliminated the AMD/Intel differences. Please note that are differences both in AMD 1) vs AMD 4) and in Intel 1) vs Intel 4)
For the record, I see that ofast-flag was defined here and should be equivalent to -Ofast

[srlantz]

@silviodonato Yes, I think tests with more limited sets of compiler options compared to -Ofast would be relatively easy to do and might help to determine what specific loosening of arithmetic is affecting the numerical agreement. The full effects of -ffast-math are detailed here. The possibility of hardware dependence is mentioned: "-ffast-math also may disable some features of the hardware IEEE implementation such as the support for denormals or flush-to-zero behavior."

Even the reduced set of options that I have suggested could still cause issues. For example, -ffinite-math-only means that the optimizer does not allow for the possibility of NaN results, so the isnan() C function always returns false (ref). My own quick C++ test at godbolt.org shows that std::isnan() and the common check x == x don't work either; they always return false and true, respectively. This could affect the results of codes that depend on such checks internally.