owner manager hang in BreakCampaignLoop

[tiancaiamao]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tikv exit first, and then tidb fail to exit when I press Ctrl+C

2. What did you expect to see? (Required)

tidb should not hang when exit.

3. What did you see instead (Required)

It hang here:

 0  0x00000000020b972e in runtime.gopark
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:425
 1  0x0000000002094ac5 in runtime.goparkunlock
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:430
 2  0x0000000002094ac5 in runtime.semacquire1
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/sema.go:178
 3  0x00000000020bb165 in sync.runtime_Semacquire
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/sema.go:71
 4  0x00000000020d5c68 in sync.(*WaitGroup).Wait
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/sync/waitgroup.go:118
 5  0x0000000003c82ab0 in github.com/pingcap/tidb/pkg/owner.(*ownerManager).BreakCampaignLoop
    at ./pkg/owner/manager.go:349
 6  0x0000000004a6dcc4 in github.com/pingcap/tidb/pkg/ddl.(*ddl).close
    at ./pkg/ddl/ddl.go:1032
 7  0x0000000004a6b6d7 in github.com/pingcap/tidb/pkg/ddl.(*ddl).Stop
    at ./pkg/ddl/ddl.go:765
 8  0x0000000004d0df03 in github.com/pingcap/tidb/pkg/domain.(*Domain).Close
    at ./pkg/domain/domain.go:1248
 9  0x0000000005a461d0 in main.closeDDLOwnerMgrDomainAndStorage
    at ./cmd/tidb-server/main.go:904
10  0x0000000005a462de in main.cleanup
    at ./cmd/tidb-server/main.go:929
11  0x0000000005a421aa in main.main.func1
    at ./cmd/tidb-server/main.go:330
12  0x0000000005a2f0a7 in github.com/pingcap/tidb/pkg/util/signal.SetupSignalHandler.func2
    at ./pkg/util/signal/signal_posix.go:55
13  0x00000000020c2721 in runtime.goexit
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700

ddl close call d.ownerManager.BreakCampaignLoop() and the owner manager hang.
It's waiting for the campaign loop to exit, but the campaign loop never break:

 0  0x00000000020b972e in runtime.gopark
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:425
 1  0x00000000020938a5 in runtime.selectgo
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:335
 2  0x0000000002743cd8 in google.golang.org/grpc.(*pickerWrapper).pick
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/picker_wrapper.go:114
 3  0x000000000273e64f in google.golang.org/grpc.(*ClientConn).getTransport
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:1047
 4  0x00000000027596c5 in google.golang.org/grpc.(*csAttempt).getTransport
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/stream.go:467
 5  0x0000000002758ca5 in google.golang.org/grpc.newClientStreamWithParams.func2
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/stream.go:350
 6  0x000000000275a97a in google.golang.org/grpc.(*clientStream).withRetry
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/stream.go:777
 7  0x0000000002758768 in google.golang.org/grpc.newClientStreamWithParams
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/stream.go:362
 8  0x0000000002757b27 in google.golang.org/grpc.newClientStream.func3
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/stream.go:219
 9  0x00000000027574f9 in google.golang.org/grpc.newClientStream
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/stream.go:254
10  0x0000000002738d77 in google.golang.org/grpc.invoke
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/call.go:66
11  0x0000000002da2a98 in go.etcd.io/etcd/client/v3.(*Client).unaryClientInterceptor.func1
    at /home/genius/go/pkg/mod/go.etcd.io/etcd/client/[email protected]/retry_interceptor.go:58
12  0x0000000002738c45 in google.golang.org/grpc.(*ClientConn).Invoke
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/call.go:35
13  0x0000000002d2ef65 in go.etcd.io/etcd/api/v3/etcdserverpb.(*leaseClient).LeaseGrant
    at /home/genius/go/pkg/mod/go.etcd.io/etcd/api/[email protected]/etcdserverpb/rpc.pb.go:6812
14  0x0000000002da03d5 in go.etcd.io/etcd/client/v3.(*retryLeaseClient).LeaseGrant
    at /home/genius/go/pkg/mod/go.etcd.io/etcd/client/[email protected]/retry.go:144
15  0x0000000002d99452 in go.etcd.io/etcd/client/v3.(*lessor).Grant
    at /home/genius/go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:216
16  0x0000000003273c73 in go.etcd.io/etcd/client/v3/concurrency.NewSession
    at /home/genius/go/pkg/mod/go.etcd.io/etcd/client/[email protected]/concurrency/session.go:46
17  0x000000000327e565 in github.com/pingcap/tidb/pkg/util.NewSession
    at ./pkg/util/etcd.go:72
18  0x0000000003c83c65 in github.com/pingcap/tidb/pkg/owner.(*ownerManager).refreshSession
    at ./pkg/owner/manager.go:446
19  0x0000000003c82cbb in github.com/pingcap/tidb/pkg/owner.(*ownerManager).campaignLoop
    at ./pkg/owner/manager.go:367
20  0x0000000003c827c8 in github.com/pingcap/tidb/pkg/owner.(*ownerManager).CampaignOwner.gowrap1
    at ./pkg/owner/manager.go:300
21  0x00000000020c2721 in runtime.goexit
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700

Similiar to #58418

4. What is your TiDB version? (Required)

master
70393b5

[Defined2014]

Same as #58620, close this one