[CSIT-1971] investigate two-band structure in nginx 2c cps tests

[vvalderrv]

Description

The structure is not visible in rps tests nor 1c tests. It is less pronounced in 2k tests, more pronounced on 0B tests. DMA does no affect the gap between bands (the worse band performance is below 1c), but it seems to increase the probability of performance being on the worse band. Trending: [0].

[0] https://csit.fd.io/trending/#eNrtlMFqwzAMhp8mvQxB7DXktMO6vkdwba02c2JhaW3ap29aytxSGD0MRtkuttEv6RcfwiwpY8cYX6pmUbWLSrfBTUf1_Po0XRsi0AMwZdB1vUZNyo7tBo1AH8cGfGJhMfYD6hUoCygeAs3FkhchcL2B6ChjTMbBsA7DCKrTqmvAEh9t9NvRxn3KlWdRyO-K8s0kpcJkNKXka8CSIMgXbndPXBq8Z9Mjhz2WLvWqyHYiWhRlr51lRxfqmUK7PGX8NP7HQv-g2PX_1t_i17-I_w9v_b3Ym-VsSLk__fnTm33aguRgIp9DB0_Uw1k

Assignee

Vratko Polak

Reporter

Vratko Polak

Comments

• vrpolak (Thu, 31 Oct 2024 09:17:58 +0000): Among NICs used in trending, only cx7 is affected (e180cq and e810xxv are not affected), so this issue may be specific to that NIC or mlx5 driver in general.
• vrpolak (Wed, 30 Oct 2024 13:00:50 +0000): Comparing vpp-show-stats between good [1] and bad [2] performance, the values in "show runtime" look as if VPP had less work to do per iteration, so maybe AB tool sends traffic slower? The "show node counters" portion has interesting differences. The bad case does not have any "Pure ACKs received" for "tcp4-established" (only for " tcp4-rcv-process") but the bad run also has "Connection already e" and "Resets received" not present in the good run. So VPP definitely behaves differently between runs, I am just not sure if this is a driver issue or hoststack code issue.

[1] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-hoststack-daily-master-2n-spr/329/log.html.gz#s1-s1-s1-s1-s5-t2-k2-k11-k5-k1-k1-k1-k1

[2] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-hoststack-daily-master-2n-spr/328/log.html.gz#s1-s1-s1-s1-s5-t2-k2-k11-k5-k1-k1-k1-k1

Original issue: https://jira.fd.io/browse/CSIT-1971

[vvalderrv]

Among NICs used in trending, only cx7 is affected (e180cq and e810xxv are not affected), so this issue may be specific to that NIC or mlx5 driver in general.

[vvalderrv]

Comparing vpp-show-stats between good [1] and bad [2] performance, the values in "show runtime" look as if VPP had less work to do per iteration, so maybe AB tool sends traffic slower? The "show node counters" portion has interesting differences. The bad case does not have any "Pure ACKs received" for "tcp4-established" (only for " tcp4-rcv-process") but the bad run also has "Connection already e" and "Resets received" not present in the good run. So VPP definitely behaves differently between runs, I am just not sure if this is a driver issue or hoststack code issue.

[1] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-hoststack-daily-master-2n-spr/329/log.html.gz#s1-s1-s1-s1-s5-t2-k2-k11-k5-k1-k1-k1-k1
[2] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-hoststack-daily-master-2n-spr/328/log.html.gz#s1-s1-s1-s1-s5-t2-k2-k11-k5-k1-k1-k1-k1