Skip to content

fix: both atelet and ate-apiserver leak gRPC connections #480

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Eric Bishop (ericdbishop) wants to merge 4 commits into
base: main
Choose a base branch
from
+139 −4
Open

fix: both atelet and ate-apiserver leak gRPC connections #480

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5b11a10
fix: both atelet and ate-apiserver leak gRPC connections
ericdbishop Jul 21, 2026
7ec609e
add debug log; move EvictionFunction to separate var
ericdbishop Jul 21, 2026
ea71d91
first test; conn enviction in atelet
ericdbishop Jul 21, 2026
6a05325
had llm help write similar test for ateapi
ericdbishop Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion cmd/ateapi/internal/controlapi/dialer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,13 @@ func NewAteletDialer(workerIndexer cache.Indexer, ateletIndexer cache.Indexer) *
return &AteletDialer{
workerIndexer: workerIndexer,
ateletIndexer: ateletIndexer,
ateletConns: lru.New(1024),
ateletConns: lru.NewWithEvictionFunc(1024, func(key lru.Key, value any) {
// Close connection when evicting from cache.
conn, ok := value.(*grpc.ClientConn)

@keithmattix Keith Mattix II (keithmattix) Jul 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should we log when either the assertion or Close() fails?

if ok {
conn.Close()
}
}),
}
}

Expand Down
9 changes: 8 additions & 1 deletion cmd/atelet/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,13 @@ func main() {
go serverboot.StartMetricsServer(ctx, serverboot.MetricsServerOptions{Addr: *metricsListenAddr})

ateomDialer := &AteomDialer{
conns: lru.New(256),
conns: lru.NewWithEvictionFunc(256, func(key lru.Key, value any) {
// Close connection when evicting from cache.
conn, ok := value.(*grpc.ClientConn)
if ok {
conn.Close()
Comment thread
ericdbishop marked this conversation as resolved.
Outdated
}
}),

@ericdbishop Eric Bishop (ericdbishop) Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming this looks ok, not sure if there is a common util file I could move this function to so it's not duplicated across files.

@juli4n Julian Gutierrez Oschmann (juli4n) Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we blindly close on eviction, can this close a connection which is already in use?

@ericdbishop Eric Bishop (ericdbishop) Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, how long lived are these connections? Maybe a delayed close might help? I also was considering if we should make the cache size configurable for both dialers.

}

var gcpRegistryAuthn authn.Authenticator
Expand Down Expand Up @@ -806,6 +812,7 @@ func toAteomReadyz(in *ateletpb.Readyz) *ateompb.Readyz {
return out
}

// AteomDialer handles gRPC connections to Ateom pods.

@keithmattix Keith Mattix II (keithmattix) Jul 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this comment?

@ericdbishop Eric Bishop (ericdbishop) Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had just seen a similar comment in the ateapi file: https://github.com/ericdbishop/substrate/blob/5b11a10073e97f0c425bb7bee23955b2d5762dc3/cmd/ateapi/internal/controlapi/dialer.go#L31. I could remove it if preferred.

@keithmattix Keith Mattix II (keithmattix) Jul 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries just curious

type AteomDialer struct {
conns *lru.Cache
}
Expand Down