[CSIT-1956] 2n-icx: trex tests fail to initialize on TB212

[vvalderrv]

Description

Looking at job statistics [0], the job passes on other testbeds, but never on TB212.

Comparing a bad verify run [1] on TB212 with a good verify run [2] on TB214, I see differences only in MAC addresses (should not matter) and PCI addresses (probably matters).

[0] https://csit.fd.io/stats/#eNrLyk-yTS7OLNEtKUqt0C1ILUrTzUspKkgp0i1PTc3OqdTNTSwuSS3SNcrTzUyuAACdRhFi

[1] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-verify-master-2n-icx/7/log.html.gz#s1-s1-s1-s1-s1-k1-k5-k3-k1-k1-k1-k5

[2] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-verify-master-2n-icx/6/log.html.gz#s1-s1-s1-s1-s1-k1-k5-k3-k1-k1-k1-k5

Assignee

Vratko Polak

Reporter

Vratko Polak

Comments

• vrpolak (Mon, 11 Nov 2024 12:45:01 +0000): For quick identification, last line in console log is:

Start T-Rex failed after multiple retries!

• vrpolak (Mon, 11 Nov 2024 11:52:26 +0000): > Fixed long time ago.

The symptom [5] is present again. Either the fix was short lived, or (more probably) this was never fixed, I just made a mistake thinking it is.

[5] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-ndrpdr-weekly-master-2n-icx/106/log.html.gz#s1-s1-s1-s1-s1-k1-k5-k3-k1-k1-k1-k5

• vrpolak (Wed, 14 Aug 2024 11:48:39 +0000): Fixed long time ago.
• vrpolak (Fri, 3 May 2024 15:55:32 +0000):

  False conclusion, the first verify run did not run on TB212.

It seems [4] the old PCI address is correct, and the issue is somewhere else.

[4] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-verify-master-2n-icx/11/log.html.gz#s1-s1-s1-k1-k5

• vrpolak (Fri, 3 May 2024 15:37:28 +0000): > PCI addresses (probably matters).

Indeed, that leads to fix [3] to be merged soon.

[3] 40867: fix(topology): copy PCI address from other 2n-icx | https://gerrit.fd.io/r/c/csit/+/40867

Original issue: https://jira.fd.io/browse/CSIT-1956

[vvalderrv]

For quick identification, last line in console log is:

Start T-Rex failed after multiple retries!

[vvalderrv]

> Fixed long time ago.

The symptom [5] is present again. Either the fix was short lived, or (more probably) this was never fixed, I just made a mistake thinking it is.

[5] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-ndrpdr-weekly-master-2n-icx/106/log.html.gz#s1-s1-s1-s1-s1-k1-k5-k3-k1-k1-k1-k5

[vvalderrv]

Fixed long time ago.

[vvalderrv]

False conclusion, the first verify run did not run on TB212.
It seems [4] the old PCI address is correct, and the issue is somewhere else.

[4] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-trex-perf-verify-master-2n-icx/11/log.html.gz#s1-s1-s1-k1-k5

[vvalderrv]

> PCI addresses (probably matters).

Indeed, that leads to fix [3] to be merged soon.

[3] 40867: fix(topology): copy PCI address from other 2n-icx | https://gerrit.fd.io/r/c/csit/+/40867