kv/kvclient/kvstreamer: TestLargeKeys failed

[cockroach-teamcity]

kv/kvclient/kvstreamer.TestLargeKeys failed with artifacts on master @ ed02f237c2039209084ac2877ce37a389e03508d:

Failed with:

=== RUN   TestLargeKeys
    test_log_scope.go:172: test logs captured to: /artifacts/tmp/_tmp/cfc927495bcc48713af623bdbb6e7a90/logTestLargeKeys22639275
    test_log_scope.go:82: use -show-logs to present logs inline
    test_server_shim.go:168: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
*
* WARNING: disk slowness detected: unable to sync log files within 10s
*

Fatal entries found in Cockroach logs:

F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1  disk stall detected: unable to sync log files within 20s
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !goroutine 19341 [running]:
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !runtime/debug.Stack()
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !	GOROOT/src/runtime/debug/stack.go:26 +0x64
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !github.com/cockroachdb/cockroach/pkg/util/debugutil.Stack(...)
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !	pkg/util/debugutil/debugutil.go:63
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !	pkg/util/log/clog.go:292 +0xc4
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x5b80fa0, 0xa700860}, 0x2, 0x4, 0x1, 0x1?, {0x588dad2, 0x37}, {0x1b845b11fb8, 0x1, ...})
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !	pkg/util/log/channels.go:104 +0x484
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !github.com/cockroachdb/cockroach/pkg/util/log.shoutfDepth(...)
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !	pkg/util/log/channels.go:111
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Shoutf(...)
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !	bazel-out/aarch64-fastbuild/bin/pkg/util/log/log_channels_generated.go:1072
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).flushAndMaybeSyncLocked.func1()
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !	pkg/util/log/file.go:270 +0xa0
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !created by time.goFunc
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !	GOROOT/src/time/sleep.go:215 +0x38
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !For more context, check log files in: /artifacts/tmp/_tmp/cfc927495bcc48713af623bdbb6e7a90/logTestLargeKeys22639275
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !****************************************************************************
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !This node experienced a fatal error (printed above), and as a result the
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !process is terminating.
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !will try to determine the root cause, recommend next steps, and we can
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !improve CockroachDB based on your report.
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !Please submit a crash report by following the instructions here:
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !If you would rather not post publicly, please contact us directly at:
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !    support@cockroachlabs.com
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !
F260624 15:22:35.958838 19341 1@util/log/file.go:270  [-] 1 !The Cockroach Labs team appreciates your feedback.

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

Jira issue: CRDB-65109