New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

More intelligent concurrency limits #241

Open

blester125 opened this issue Apr 24, 2024 · 0 comments

Labels

Collaborator

blester125 commented Apr 24, 2024

Currently we can limit concurrency by specifying a number of parameters to process at once which is pretty rough.

It should be possible to automatically limit concurrency based on things like memory usage (see https://github.com/google-research/t5x/blob/6b02d25cd67a397c6cfffe90ad2cca4b343535ae/t5x/checkpoints.py#L461), compute used, number of open files, etc. Automatic limiting could even help avoid write contention on disk.

https://death.andgravity.com/limit-concurrency includes ways to implement concurrency limits (Note: I don't think their issue with our Semaphore solution isn't really relevant to us)

blester125 added the enhancement label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment