Memory-bounded clone/fetch for large repositories

[mjgil]

Summary 💡

I'd like to be able to clone or fetch large repositories with gitoxide while keeping process memory bounded and predictable, instead of having memory usage scale with pack shape or repository size.

The concrete thing I’m after is some form of memory-budgeted clone/fetch path, where pack receive, indexing, and delta resolution either:

1. stay within a configured memory budget, or
2. fail cleanly with a dedicated error,

instead of risking an OOM kill on large or adversarial repos.

I’ve been exploring this in a fork and would be interested in upstreaming it in smaller pieces if this direction makes sense.

Motivation 🔦

I’m using gitoxide in a service that mirrors remote repositories on relatively small machines. The tricky case is not correctness, it’s survivability: on a 1 GB-ish VPS, a sufficiently large repo, large pack, or deep delta chain can push clone/fetch memory high enough that the host gets OOM-killed.

What I want from gitoxide is roughly the same operational property you’d want from any server-side storage primitive: memory use should be controllable, and pathological inputs should degrade into a normal error instead of taking the process down.

The work I’ve done locally points to a plausible direction:

• a shared process/repository-scoped memory budget
• bounded decoded-object / delta caches
• spilling large intermediates to disk when the budget is exhausted
• external sorting / streaming in places that currently retain large in-memory structures
• surfacing OutOfBudget-style failures as ordinary errors

In that fork, this was enough to get stress clones of very large repos into a much smaller and more predictable memory envelope, which makes gitoxide much more viable for mirroring and other server-side automation.

Before I invest more in polishing this for upstream, I wanted to check whether this is a direction the project would want to support.

[Byron]

Thanks for following up #2526 with an issue to describe the bigger picture - t's much appreciated.

Also, it's great to hear that this already works in your fork.

While I think everyone would be interested in having software that fails gracefully under certain possibly even configurable memory limits, I also fear the complexity that comes with it.

So starting this off with streaming objects seems like a good idea because this is one of the weak points of the current implementation. There is no world where Gitoxide should use 5GB in memory just because it tries to write a 5GB object to disk.

So I think there are generally two goals in described here.
One is to do more under a constrained memory envelope, and two is to fail gracefully if failure is needed. With the second point, maybe there are other ways to prevent unrelated processes to be killed as well.

GPT 5.4

Is it possible to run a Linux process in such a way that it will automatically be killed if it tries to go beyond a certain maximum memory bound. The idea is to avoid OOM killing unrelated processes.

Yes. The Linux mechanism you want is a memory cgroup, not classic ulimit.

Set the process in its own cgroup with a hard limit:

• On cgroup v2: memory.max=<limit>
• Optional: memory.oom.group=1 to kill the whole process tree together

That confines OOM handling to that cgroup, so the kernel kills the offending process(es) there instead of unrelated system processes.

Examples:

systemd-run --user --scope -p MemoryMax=500M your_command

If you want the whole subprocess tree killed on cgroup OOM:

systemd-run --user --scope -p MemoryMax=500M -p OOMPolicy=kill your_command

Or directly with cgroup v2:

mkdir /sys/fs/cgroup/myjob
echo 500M > /sys/fs/cgroup/myjob/memory.max
echo 1 > /sys/fs/cgroup/myjob/memory.oom.group
echo $$ > /sys/fs/cgroup/myjob/cgroup.procs
exec your_command

Important caveats:

• RLIMIT_RSS is basically useless on Linux.
• ulimit -v / RLIMIT_AS limits virtual address space, not actual memory usage, so it is often the wrong tool.
• Containers (docker run --memory=500m ...) also use cgroups underneath and work well for this.

So the short answer is: yes, use a dedicated cgroup with memory.max. That is the standard way to prevent a runaway process from causing a system-wide OOM that kills unrelated processes.

[mjgil]

I was thinking along these lines, but it seems like a substantial change.

https://github.com/mjgil/gitoxide/tree/pr-2527-anon
https://github.com/mjgil/gitoxide/tree/pr-2527-pread

[Byron]

I don't really know why there are two links that look so similar without any context.

Having looked just at the first one, I keep thinking that the allocator should play the major role in limiting memory.
This approach would mean that gitoxide will play nicer in any kind of OOM situation, should it come to that, while allowing to set explicit bounds with custom allocators. This would mean gitoxide has to use APIs that won't panic on OOM, which adds complexity at the very least, but I think it's worth it. The hard part is to validate that this is respected everywhere, but I think this could be done by clippy-forbidding any APIs that aren't OOM-safe.

From there, it would be interesting to explore to what extent more can be done with the same amount or with less memory, so that less memory is needed to handle even big repositories. To my mind the extension to the above paragraph is that when it's out of memory it can fall back to another mechanism to be able to complete the work albeit slower. After all, nothing is free and there will be a trade-off.

Lastly, the Delta tree is great and I think it has similar memory requirements as Git for the same task. And from all I can tell, it's going to be hard to not use this much memory. But again, there might be ways of bypassing high RAM needs with disk-backed memory maps, as a backup mechanism for when it runs out of memory.

I can even imagine certain git configuration values that will force it to use the low-memory alternative implementation by default. This should be useful for when you know it's going to run out of memory, or you can predict it.

So in a way, what I'm proposing is a more general version of what you might have implemented.

I may also say that at this point I'm not convinced something like an explicit memory cap should be added. To me this is work of the allocator and it's all about using OOM safe APIs.