roachperf: regression around 2022-09-09 [Go 1.19]

[erikgrinaker]

Most pronounced on kv95/enc=false/nodes=3/cpu=96, but also shows up on many other benchmarks.

Jira issue: CRDB-19655

[blathers-crl]

cc @cockroachdb/test-eng

[erikgrinaker]

Bisected this down to the Go 1.19 upgrade in #86049. Raw data:

$ roachprod create -n 4 --gce-machine-type n2-standard-32 --gce-local-ssd-count 4 --gce-zones europe-west1-b grinaker-bench
$ ./workload run kv --init --splits 1000 --concurrency 192 --read-percent 95 --ramp 120s --duration 180s

 d68046c799 2022-09-07: Merge #87448 [craig[bot]] = 118756.4
 7c01c817e5 2022-09-09: Merge #86451 [craig[bot]] = 113676.4
 1a637b0d7b 2022-09-08: Merge #87531 [craig[bot]] = 113873.7
 c83a2e8972 2022-09-07: server: fix error format message [Yahor Yuzefovich] = 108224.2 (bad run)
 76a5f6383f 2022-09-07: Merge #87174 #87318 #87403 #87443 #87450 #87461 #87493 #87504 [craig[bot]] = 112652.3
 2675c7c048 2022-08-12: *: upgrade to go 1.19.1 [Ricky Stewart] = 112210.3
 1ac25dcebf 2022-09-07: Merge #87497 [craig[bot]] = 118695.0

[blathers-crl]

Hi @erikgrinaker, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[blathers-crl]

cc @cockroachdb/replication

[erikgrinaker]

Reassigning to repl until we find an owner to investigate the Go 1.19 regression.

[ajwerner]

It'd be good to get CPU profiles before and after the upgrade for some workload to compare.