Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unclear with Container garbage collection configuration #127157

Open
T-Lakshmi opened this issue Sep 5, 2024 · 9 comments · May be fixed by kubernetes/website#48001 or #129391
Open

Unclear with Container garbage collection configuration #127157

T-Lakshmi opened this issue Sep 5, 2024 · 9 comments · May be fixed by kubernetes/website#48001 or #129391
Assignees
@zhifei92 @esotsal
Labels
area/kubelet help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/documentation Categorizes issue or PR as related to documentation. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. sig/node Categorizes an issue or PR as relevant to SIG Node. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@T-Lakshmi
Copy link

What happened?

Iam trying to disable maxPerPodContainer feature in my cluster. To achieve this followed the docs Container garbage collection and gone through code related to garbage collection.

Configured Container garbage collection parameters in the kubelet config file and restarted the kubelet sucessfully.
After that if i check, values in kubelet configuration file are persist but those are actually not reflected in cluster.

What did you expect to happen?

Container garbage collection parameters which i added in config file should reflect in cluster.

minAge:
maxPerPodContainer:
maxContainers:

How can we reproduce it (as minimally and precisely as possible)?

Ex:

  1. Add below values in kubelet configuration file: /var/lib/kubelet/config.yaml
minAge: "5m"
maxPerPodContainer: 1
maxContainers: 50

  1. Restart kubelet service:
    sudo systemctl restart kubelet
    Make sure kubelet restarted sucessfully: sudo systemctl status kubelet

  2. Run: kubectl proxy

  3. Open another terminal and check whether added parameters are present in configuration or not:
    curl http://localhost:8001/api/v1/nodes/<hostname/proxy/configz

Anything else we need to know?

I tried many ways to add these parameters but no use. even these are not listed in api-resources.
https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

Is it possible to define these values in kubelet as mentioned in docs? if possible how can we configure them?

Kubernetes version

$ kubectl version
Client Version: v1.30.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.3

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)
@T-Lakshmi T-Lakshmi added the kind/bug Categorizes issue or PR as related to a bug. label Sep 5, 2024
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 5, 2024
@T-Lakshmi
Copy link
Author

/remove-kind bug
/sig node
/area kubelet

@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. area/kubelet needs-kind Indicates a PR lacks a `kind/foo` label and requires one. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. kind/bug Categorizes issue or PR as related to a bug. labels Sep 5, 2024
@saschagrunert
Copy link
Member

saschagrunert commented Sep 6, 2024 •
edited
Loading

It looks like we don't populate those values in the kubelet configuration any more after #53088:

minAge: "5m"
maxPerPodContainer: 1
maxContainers: 50

It's possible to set them using the CLI, though:

--minimum-container-ttl-duration duration                  Minimum age for a finished container before it is garbage collected.  Examples: '300ms', '10s' or '2h45m' (DEPRECATED: Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.)
--maximum-dead-containers int32                            Maximum number of old instances of containers to retain globally.  Each container takes up some disk space. To disable, set to a negative number. (default -1) (DEPRECATED: Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.)
--maximum-dead-containers-per-container int32              Maximum number of old instances to retain per container.  Each container takes up some disk space. (default 1) (DEPRECATED: Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.)

I think the whole container GC configuration is deprecated in favor of the eviction feature and I'm not sure we should do anything here.

@T-Lakshmi
Copy link
Author

@saschagrunert,
Thanks for responding.

--minimum-container-ttl-duration duration                  Minimum age for a finished container before it is garbage collected.  Examples: '300ms', '10s' or '2h45m' (DEPRECATED: Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.)
--maximum-dead-containers int32                            Maximum number of old instances of containers to retain globally.  Each container takes up some disk space. To disable, set to a negative number. (default -1) (DEPRECATED: Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.)
--maximum-dead-containers-per-container int32              Maximum number of old instances to retain per container.  Each container takes up some disk space. (default 1) (DEPRECATED: Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.)
I think the whole container GC configuration is deprecated in favor of the eviction feature and I'm not sure we should do anything here.

Yes, i was going through these values before. There was no information these parameters are direct related with container GC configuration. Is these are for container GC configuration?

@haircommander
Copy link
Contributor

TBH I think we probably should drop these CLI flags now. We just hadn't gotten around to it yet, but they're marked as DEPRECATED. We should also update the documentation to deter folks from using them

/triage accepted
/priority important-longterm
/kind documentation

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. kind/documentation Categorizes issue or PR as related to documentation. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 11, 2024
@haircommander
Copy link
Contributor

/help

@k8s-ci-robot
Copy link
Contributor

@haircommander:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

  • Why are we solving this issue?
  • To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
  • Does this issue have zero to low barrier of entry?
  • How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Sep 11, 2024
@haircommander
Copy link
Contributor

/assign @esotsal

@haircommander haircommander moved this from Triage to Triaged in SIG Node Bugs Sep 11, 2024
@T-Lakshmi T-Lakshmi linked a pull request Sep 19, 2024 that will close this issue
Open
@T-Lakshmi
Copy link
Author

Raised PR in website repo for docs fix.
PTAL!

@haircommander @esotsal

@zhifei92 zhifei92 linked a pull request Dec 25, 2024 that will close this issue
Open
@zhifei92
Copy link
Contributor

/assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhifei92 zhifei92

@esotsal esotsal

Labels
area/kubelet help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/documentation Categorizes issue or PR as related to documentation. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. sig/node Categorizes an issue or PR as relevant to SIG Node. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Node Bugs
Status: Triaged
Milestone
No milestone
6 participants
@saschagrunert @haircommander @k8s-ci-robot @zhifei92 @T-Lakshmi @esotsal