Podman freezes when pulling image

[karlobratko]

Hi,

I am experiencing an issue with Podman where attempts to pull an image from quay.io (or any other image repository) consistently freeze during the GET request for the image manifest. My network connection is stable, and I can access quay.io without any issues through a web browser and using tools like curl. The problem persists even after ensuring there are no network-related issues on my end.

Interesting thing is that before 3 days everything was working properly. There was no problems at all.

Steps to Reproduce:

Run the command podman --log-level=debug pull quay.io/podman/hello in terminal.

Actual Behavior:

The pull command freezes during the GET request for the image manifest, specifically at this step:

DEBU[0002] GET https://quay.io/v2/podman/hello/manifests/sha256:70d0485c87afcdb2c69082f7dba4dc173aeec596c1dbf3dff9ec08b200ef70aa

The command does not progress beyond this point, and no further output is provided, even after a significant wait time.

Attempts to Resolve:

• Verified network connectivity and DNS resolution.
• Accessed quay.io successfully using a browser and curl.
• Updated Podman to the latest version (on Fedora it is 4.9.4.).
• Cleared Podman's system and storage.
• Performed a clean reinstallation of Podman, including removal and reinstallation of all configuration files.
• Executed podman system reset to clear Podman's system and storage.
• Attempted pulling different images/tags from quay.io.
• Increased verbosity for more detailed logs but didn't identify any specific errors related to the freeze.

Environment:

• Podman version: 4.9.4.
• OS: Fedora Linux 39 (Workstation Edition) x86_64
• Kernel: 6.8.4-200.fc39.x86_64

Full command output:

INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman run --log-level=debug hello-world)
DEBU[0000] Using conmon: "/usr/bin/conmon"
INFO[0000] Using sqlite as database backend
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/kbratko/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /home/kbratko/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/kbratko/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 49
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image hello-world (policy: missing)
DEBU[0000] Looking up image "hello-world" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] reference "[overlay@/home/kbratko/.local/share/containers/storage+/run/user/1000/containers]quay.io/podman/hello:latest" does not resolve to an image ID
DEBU[0000] Trying "localhost/hello-world:latest" ...
DEBU[0000] reference "[overlay@/home/kbratko/.local/share/containers/storage+/run/user/1000/containers]localhost/hello-world:latest" does not resolve to an image ID
DEBU[0000] Trying "registry.fedoraproject.org/hello-world:latest" ...
DEBU[0000] reference "[overlay@/home/kbratko/.local/share/containers/storage+/run/user/1000/containers]registry.fedoraproject.org/hello-world:latest" does not resolve to an image ID
DEBU[0000] Trying "registry.access.redhat.com/hello-world:latest" ...
DEBU[0000] reference "[overlay@/home/kbratko/.local/share/containers/storage+/run/user/1000/containers]registry.access.redhat.com/hello-world:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/library/hello-world:latest" ...
DEBU[0000] reference "[overlay@/home/kbratko/.local/share/containers/storage+/run/user/1000/containers]docker.io/library/hello-world:latest" does not resolve to an image ID
DEBU[0000] Trying "quay.io/hello-world:latest" ...
DEBU[0000] reference "[overlay@/home/kbratko/.local/share/containers/storage+/run/user/1000/containers]quay.io/hello-world:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/library/hello-world:latest" ...
DEBU[0000] reference "[overlay@/home/kbratko/.local/share/containers/storage+/run/user/1000/containers]docker.io/library/hello-world:latest" does not resolve to an image ID
DEBU[0000] Trying "hello-world" ...
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Attempting to pull candidate quay.io/podman/hello:latest for hello-world
DEBU[0000] parsed reference into "[overlay@/home/kbratko/.local/share/containers/storage+/run/user/1000/containers]quay.io/podman/hello:latest"
DEBU[0000] Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull quay.io/podman/hello:latest...
DEBU[0000] Copying source image //quay.io/podman/hello:latest to destination image [overlay@/home/kbratko/.local/share/containers/storage+/run/user/1000/containers]quay.io/podman/hello:latest
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "quay.io/podman/hello:latest"
DEBU[0000] No credentials matching quay.io/podman/hello found in /run/user/1000/containers/auth.json
DEBU[0000] No credentials matching quay.io/podman/hello found in /home/kbratko/.config/containers/auth.json
DEBU[0000] No credentials matching quay.io/podman/hello found in /home/kbratko/.docker/config.json
DEBU[0000] No credentials matching quay.io/podman/hello found in /home/kbratko/.dockercfg
DEBU[0000] No credentials for quay.io/podman/hello found
DEBU[0000]  No signature storage configuration found for quay.io/podman/hello:latest, using built-in default file:///home/kbratko/.local/share/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io
DEBU[0000] GET https://quay.io/v2/
DEBU[0000] Ping https://quay.io/v2/ status 401
DEBU[0000] GET https://quay.io/v2/auth?scope=repository%3Apodman%2Fhello%3Apull&service=quay.io
DEBU[0001] Increasing token expiration to: 60 seconds
DEBU[0001] GET https://quay.io/v2/podman/hello/manifests/latest
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json"
DEBU[0001] Using SQLite blob info cache at /home/kbratko/.local/share/containers/cache/blob-info-cache-v1.sqlite
DEBU[0001] Source is a manifest list; copying (only) instance sha256:70d0485c87afcdb2c69082f7dba4dc173aeec596c1dbf3dff9ec08b200ef70aa for current system
DEBU[0001] GET https://quay.io/v2/podman/hello/manifests/sha256:70d0485c87afcdb2c69082f7dba4dc173aeec596c1dbf3dff9ec08b200ef70aa

[TheB0tt]

Hi,

Did you manage to resolve this? We are facing what appears to be the same problem where podman pull freezes on the GET. If we use dig, curl or wget then it works fine.

[sbrivio-rh]

Can you all try with updated versions and report back? Thanks.

[karlobratko]

I am currently on Podman version 5.4.0 and everything works correctly.
In my case problem was in network infrastructure through which I was sending pull request. At least that is explanation that I found to make most sense.

[KlausLinzner]

We ran into this situation last week in a customer environment. After checking with tcp dump we saw TCP Dup Ack followed by a (seemingly) infinite loop of TCP retransmissions.
We managed to get out of by configuring TCP settings:

sudo vi /etc/sysctl.conf

net.core.rmem_max = 12582912
net.core.wmem_max = 4194304
net.ipv4.tcp_rmem = 2048 87380 4194304
net.ipv4.tcp_wmem = 2048 16384 4194304
net.ipv4.tcp_retries2 = 2

sudo sysctl -p

Although no deeper tests were made, the most important part of it was setting tcp_retries2.
Maybe as a sidenote: we could reproduce it only with registry.redhat.io and quay.io. While pulling from azure container registry we initially ran into the same problem - but as ACR kills the connection after one minute idle, it eventually resolved itself.

[sbrivio-rh]

We ran into this situation last week in a customer environment. After checking with tcp dump we saw TCP Dup Ack followed by a (seemingly) infinite loop of TCP retransmissions.

If you're running a recent version of Podman, rootless, you probably need this fix for pasta(1): https://passt.top/passt/commit/tcp.c?id=238c69f9af458e41dea5ad8c988dbf65b05b5172. But without any details about environments, configuration, versions, etc. it's a bit difficult to evaluate.

If it's a bug, it should be reported as such, with the required details.

[Luap99]

Note pasta is not involved at all when pulling images as that happens from the host netns unless we are talking about nested containers.

[KlausLinzner]

Did open case 04106200 at redhat as it happened during openshift installation. Unfortunately can't give any/many details about environment as we're running in a heavily restricted customer environment were only very little is visible to us.

(Just wanted to share a possible way out for those suffering from the problem)