[Bug]: HttpRequestException during PostgreSqlContainer.StartAsync() #13
Labels
Comments
|
Thank you for creating the issue. I believe this is an issue with Docker.DotNet. I moved the issue to the upstream repository. It looks like that Docker.DotNet fails to send the HTTP request properly. If you look at the exception message, the body (JSON payload) seems incomplete (the second one looks incomplete too). |
|
We should probably look into the HTTP client message handler and check why it is failing to send the request. Docker.DotNet sets the required handler in its constructor. |
|
It appears that the request was sent correctly, and the 'Invalid chunk header' exception is related to the response from the GET containers/{id}/json request Unfortunately, I am unable to reproduce this issue on my local machine. As usual everything works as expected. |
|
You are right, it looks like:
Do you think you could run a custom version of Docker.DotNet in your CI that logs additional information, such as the |
|
Yeah, it's possible for me to run a custom version. When debugging locally it seems like I am able to reproduce the issue, or at least something very similar.
Where responseLines in the HttpConnection had the following values:
Can you potentially guide me towards some insights? It seems like I'm overlooking the problem. |
|
I started yesterday adding a logger to the library. Hopefully, this will help us identify and address issues that occur when running with Podman, Rancher, etc. I also created a simple test (note that the output from the first run might get truncated due to the size of the pull log messages; however, the second run should contain the interesting and necessary information). When I run the test against Docker Desktop, I receive the following data. The RequestUri="http://docker_engine/containers/create"
Headers="HTTP/1.1 201 Created, Api-Version: 1.47, Content-Type: application/json, Date: Tue, 18 Mar 2025 11:58:16 GMT, Docker-Experimental: false, Ostype: linux, Server: Docker/27.5.1 (linux), Connection: close, Transfer-Encoding: chunked"
HeaderLine="58"
HeaderLine="0"
Content="{"Id":"c326db38cfa70232485be88f54b17541618436c0dba1c8d653b88422add23830","Warnings":[]}"
RequestUri="http://docker_engine/containers/c326db38cfa70232485be88f54b17541618436c0dba1c8d653b88422add23830/json"
Headers="HTTP/1.1 200 OK, Api-Version: 1.47, Content-Type: application/json, Date: Tue, 18 Mar 2025 11:58:16 GMT, Docker-Experimental: false, Ostype: linux, Server: Docker/27.5.1 (linux), Connection: close, Transfer-Encoding: chunked"
HeaderLine="eed"
HeaderLine="0"
Content="{"Id":"c326db38cfa70232485be88f54b17541618436c0dba1c8d653b88422add23830","Created":"2025-03-18T11:58:16.735439603Z","Path":"docker-entrypoint.sh","Args":["postgres"],"State":{"Status":"created","Running":false,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":0,"ExitCode":0,"Error":"","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z"},"Image":"sha256:7062a2109c4b51f3c792c7ea01e83ed12ef9a980886e3b3d380a7d2e5f6ce3f5","ResolvConfPath":"","HostnamePath":"","HostsPath":"","LogPath":"","Name":"/frosty_volhard","RestartCount":0,"Driver":"overlayfs","Platform":"linux","MountLabel":"","ProcessLabel":"","AppArmorProfile":"","ExecIDs":null,"HostConfig":{"Binds":null,"ContainerIDFile":"","LogConfig":{"Type":"json-file","Config":{}},"NetworkMode":"bridge","PortBindings":{},"RestartPolicy":{"Name":"no","MaximumRetryCount":0},"AutoRemove":false,"VolumeDriver":"","VolumesFrom":null,"ConsoleSize":[0,0],"CapAdd":null,"CapDrop":null,"CgroupnsMode":"host","Dns":null,"DnsOptions":null,"DnsSearch":null,"ExtraHosts":null,"GroupAdd":null,"IpcMode":"private","Cgroup":"","Links":null,"OomScoreAdj":0,"PidMode":"","Privileged":false,"PublishAllPorts":false,"ReadonlyRootfs":false,"SecurityOpt":null,"UTSMode":"","UsernsMode":"","ShmSize":67108864,"Runtime":"runc","Isolation":"","CpuShares":0,"Memory":0,"NanoCpus":0,"CgroupParent":"","BlkioWeight":0,"BlkioWeightDevice":null,"BlkioDeviceReadBps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteIOps":null,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpusetCpus":"","CpusetMems":"","Devices":null,"DeviceCgroupRules":null,"DeviceRequests":null,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":null,"OomKillDisable":false,"PidsLimit":null,"Ulimits":null,"CpuCount":0,"CpuPercent":0,"IOMaximumIOps":0,"IOMaximumBandwidth":0,"MaskedPaths":["/proc/asound","/proc/acpi","/proc/kcore","/proc/keys","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug","/proc/scsi","/sys/firmware","/sys/devices/virtual/powercap"],"ReadonlyPaths":["/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"]},"GraphDriver":{"Data":null,"Name":"overlayfs"},"Mounts":[{"Type":"volume","Name":"218ec5bc0bd77591ba76821389beeebf68ef7d6f4b67c4e8de2625d5dfe35e8c","Source":"/var/lib/docker/volumes/218ec5bc0bd77591ba76821389beeebf68ef7d6f4b67c4e8de2625d5dfe35e8c/_data","Destination":"/var/lib/postgresql/data","Driver":"local","Mode":"","RW":true,"Propagation":""}],"Config":{"Hostname":"c326db38cfa7","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"ExposedPorts":{"5432/tcp":{}},"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","GOSU_VERSION=1.17","LANG=en_US.utf8","PG_MAJOR=17","PG_VERSION=17.4","PG_SHA256=c4605b73fea11963406699f949b966e5d173a7ee0ccaef8938dec0ca8a995fe7","DOCKER_PG_LLVM_DEPS=llvm19-dev \t\tclang19","PGDATA=/var/lib/postgresql/data"],"Cmd":["postgres"],"Image":"postgres:alpine","Volumes":{"/var/lib/postgresql/data":{}},"WorkingDir":"/","Entrypoint":["docker-entrypoint.sh"],"OnBuild":null,"Labels":{},"StopSignal":"SIGINT"},"NetworkSettings":{"Bridge":"","SandboxID":"","SandboxKey":"","Ports":{},"HairpinMode":false,"LinkLocalIPv6Address":"","LinkLocalIPv6PrefixLen":0,"SecondaryIPAddresses":null,"SecondaryIPv6Addresses":null,"EndpointID":"","Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","MacAddress":"","Networks":{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"MacAddress":"","DriverOpts":null,"NetworkID":"","EndpointID":"","Gateway":"","IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DNSNames":null}}}}"
RequestUri="http://docker_engine/containers/c326db38cfa70232485be88f54b17541618436c0dba1c8d653b88422add23830/json"
Headers="HTTP/1.1 200 OK, Api-Version: 1.47, Content-Type: application/json, Date: Tue, 18 Mar 2025 11:58:16 GMT, Docker-Experimental: false, Ostype: linux, Server: Docker/27.5.1 (linux), Connection: close, Transfer-Encoding: chunked"
HeaderLine="eed"
HeaderLine="0"
Content="{"Id":"c326db38cfa70232485be88f54b17541618436c0dba1c8d653b88422add23830","Created":"2025-03-18T11:58:16.735439603Z","Path":"docker-entrypoint.sh","Args":["postgres"],"State":{"Status":"created","Running":false,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":0,"ExitCode":0,"Error":"","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z"},"Image":"sha256:7062a2109c4b51f3c792c7ea01e83ed12ef9a980886e3b3d380a7d2e5f6ce3f5","ResolvConfPath":"","HostnamePath":"","HostsPath":"","LogPath":"","Name":"/frosty_volhard","RestartCount":0,"Driver":"overlayfs","Platform":"linux","MountLabel":"","ProcessLabel":"","AppArmorProfile":"","ExecIDs":null,"HostConfig":{"Binds":null,"ContainerIDFile":"","LogConfig":{"Type":"json-file","Config":{}},"NetworkMode":"bridge","PortBindings":{},"RestartPolicy":{"Name":"no","MaximumRetryCount":0},"AutoRemove":false,"VolumeDriver":"","VolumesFrom":null,"ConsoleSize":[0,0],"CapAdd":null,"CapDrop":null,"CgroupnsMode":"host","Dns":null,"DnsOptions":null,"DnsSearch":null,"ExtraHosts":null,"GroupAdd":null,"IpcMode":"private","Cgroup":"","Links":null,"OomScoreAdj":0,"PidMode":"","Privileged":false,"PublishAllPorts":false,"ReadonlyRootfs":false,"SecurityOpt":null,"UTSMode":"","UsernsMode":"","ShmSize":67108864,"Runtime":"runc","Isolation":"","CpuShares":0,"Memory":0,"NanoCpus":0,"CgroupParent":"","BlkioWeight":0,"BlkioWeightDevice":null,"BlkioDeviceReadBps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteIOps":null,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpusetCpus":"","CpusetMems":"","Devices":null,"DeviceCgroupRules":null,"DeviceRequests":null,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":null,"OomKillDisable":false,"PidsLimit":null,"Ulimits":null,"CpuCount":0,"CpuPercent":0,"IOMaximumIOps":0,"IOMaximumBandwidth":0,"MaskedPaths":["/proc/asound","/proc/acpi","/proc/kcore","/proc/keys","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug","/proc/scsi","/sys/firmware","/sys/devices/virtual/powercap"],"ReadonlyPaths":["/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"]},"GraphDriver":{"Data":null,"Name":"overlayfs"},"Mounts":[{"Type":"volume","Name":"218ec5bc0bd77591ba76821389beeebf68ef7d6f4b67c4e8de2625d5dfe35e8c","Source":"/var/lib/docker/volumes/218ec5bc0bd77591ba76821389beeebf68ef7d6f4b67c4e8de2625d5dfe35e8c/_data","Destination":"/var/lib/postgresql/data","Driver":"local","Mode":"","RW":true,"Propagation":""}],"Config":{"Hostname":"c326db38cfa7","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"ExposedPorts":{"5432/tcp":{}},"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","GOSU_VERSION=1.17","LANG=en_US.utf8","PG_MAJOR=17","PG_VERSION=17.4","PG_SHA256=c4605b73fea11963406699f949b966e5d173a7ee0ccaef8938dec0ca8a995fe7","DOCKER_PG_LLVM_DEPS=llvm19-dev \t\tclang19","PGDATA=/var/lib/postgresql/data"],"Cmd":["postgres"],"Image":"postgres:alpine","Volumes":{"/var/lib/postgresql/data":{}},"WorkingDir":"/","Entrypoint":["docker-entrypoint.sh"],"OnBuild":null,"Labels":{},"StopSignal":"SIGINT"},"NetworkSettings":{"Bridge":"","SandboxID":"","SandboxKey":"","Ports":{},"HairpinMode":false,"LinkLocalIPv6Address":"","LinkLocalIPv6PrefixLen":0,"SecondaryIPAddresses":null,"SecondaryIPv6Addresses":null,"EndpointID":"","Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","MacAddress":"","Networks":{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"MacAddress":"","DriverOpts":null,"NetworkID":"","EndpointID":"","Gateway":"","IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DNSNames":null}}}}"
RequestUri="http://docker_engine/containers/c326db38cfa70232485be88f54b17541618436c0dba1c8d653b88422add23830/json"
Headers="HTTP/1.1 200 OK, Api-Version: 1.47, Content-Type: application/json, Date: Tue, 18 Mar 2025 11:58:16 GMT, Docker-Experimental: false, Ostype: linux, Server: Docker/27.5.1 (linux), Connection: close, Transfer-Encoding: chunked"
HeaderLine="eed"
HeaderLine="0"
Content="{"Id":"c326db38cfa70232485be88f54b17541618436c0dba1c8d653b88422add23830","Created":"2025-03-18T11:58:16.735439603Z","Path":"docker-entrypoint.sh","Args":["postgres"],"State":{"Status":"created","Running":false,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":0,"ExitCode":0,"Error":"","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z"},"Image":"sha256:7062a2109c4b51f3c792c7ea01e83ed12ef9a980886e3b3d380a7d2e5f6ce3f5","ResolvConfPath":"","HostnamePath":"","HostsPath":"","LogPath":"","Name":"/frosty_volhard","RestartCount":0,"Driver":"overlayfs","Platform":"linux","MountLabel":"","ProcessLabel":"","AppArmorProfile":"","ExecIDs":null,"HostConfig":{"Binds":null,"ContainerIDFile":"","LogConfig":{"Type":"json-file","Config":{}},"NetworkMode":"bridge","PortBindings":{},"RestartPolicy":{"Name":"no","MaximumRetryCount":0},"AutoRemove":false,"VolumeDriver":"","VolumesFrom":null,"ConsoleSize":[0,0],"CapAdd":null,"CapDrop":null,"CgroupnsMode":"host","Dns":null,"DnsOptions":null,"DnsSearch":null,"ExtraHosts":null,"GroupAdd":null,"IpcMode":"private","Cgroup":"","Links":null,"OomScoreAdj":0,"PidMode":"","Privileged":false,"PublishAllPorts":false,"ReadonlyRootfs":false,"SecurityOpt":null,"UTSMode":"","UsernsMode":"","ShmSize":67108864,"Runtime":"runc","Isolation":"","CpuShares":0,"Memory":0,"NanoCpus":0,"CgroupParent":"","BlkioWeight":0,"BlkioWeightDevice":null,"BlkioDeviceReadBps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteIOps":null,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpusetCpus":"","CpusetMems":"","Devices":null,"DeviceCgroupRules":null,"DeviceRequests":null,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":null,"OomKillDisable":false,"PidsLimit":null,"Ulimits":null,"CpuCount":0,"CpuPercent":0,"IOMaximumIOps":0,"IOMaximumBandwidth":0,"MaskedPaths":["/proc/asound","/proc/acpi","/proc/kcore","/proc/keys","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug","/proc/scsi","/sys/firmware","/sys/devices/virtual/powercap"],"ReadonlyPaths":["/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"]},"GraphDriver":{"Data":null,"Name":"overlayfs"},"Mounts":[{"Type":"volume","Name":"218ec5bc0bd77591ba76821389beeebf68ef7d6f4b67c4e8de2625d5dfe35e8c","Source":"/var/lib/docker/volumes/218ec5bc0bd77591ba76821389beeebf68ef7d6f4b67c4e8de2625d5dfe35e8c/_data","Destination":"/var/lib/postgresql/data","Driver":"local","Mode":"","RW":true,"Propagation":""}],"Config":{"Hostname":"c326db38cfa7","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"ExposedPorts":{"5432/tcp":{}},"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","GOSU_VERSION=1.17","LANG=en_US.utf8","PG_MAJOR=17","PG_VERSION=17.4","PG_SHA256=c4605b73fea11963406699f949b966e5d173a7ee0ccaef8938dec0ca8a995fe7","DOCKER_PG_LLVM_DEPS=llvm19-dev \t\tclang19","PGDATA=/var/lib/postgresql/data"],"Cmd":["postgres"],"Image":"postgres:alpine","Volumes":{"/var/lib/postgresql/data":{}},"WorkingDir":"/","Entrypoint":["docker-entrypoint.sh"],"OnBuild":null,"Labels":{},"StopSignal":"SIGINT"},"NetworkSettings":{"Bridge":"","SandboxID":"","SandboxKey":"","Ports":{},"HairpinMode":false,"LinkLocalIPv6Address":"","LinkLocalIPv6PrefixLen":0,"SecondaryIPAddresses":null,"SecondaryIPv6Addresses":null,"EndpointID":"","Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","MacAddress":"","Networks":{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"MacAddress":"","DriverOpts":null,"NetworkID":"","EndpointID":"","Gateway":"","IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DNSNames":null}}}}"
Did you do anything special to break/reproduce it? It seems that in your environment, the library is reading the wrong parts of the stream. It appears to be reading the body instead of the header. Have you tried using an older version of Testcontainers for .NET as well? I noticed you’re using an outdated version of Docker. Have you tried updating it? We recently updated the library to support the new Docker API. Although, this alone does not explain the flakiness. |
|
Thank you, I've just checked out the branch and ran the test creating the following result: The HeaderLine value seems to be correct, other than that it's significantly longer than your response.
I placed some breakpoints in the stacktrace starting at StartAsync, nothing special. |
|
I'll try with an older version of testcontainers as well as an older version of rancher desktop. |
Hmm, the output looks good so far. How does it behave when you try to break it? I'm curious if this issue only occurs in conjunction with Testcontainers. 🤔 Are you running multiple containers? Perhaps it is a race condition? |
|
When placing a single breakpoint on l186 in the HttpConnection.cs of your custom branch I can reproduce the following 100% of the time:The following then always hits:
|
|
Not sure if this is a good sign, but maybe we've made a step forward 😅. Unfortunately, I still cannot reproduce the issue using Docker Desktop or Podman Desktop. What's interesting is that the first time it reads the numeric hex value [Docker.DotNet 00:00:03.02] HeaderLine="1451"
[Docker.DotNet 00:00:03.02] HeaderLine="Once[...]"This is how the successful request looks: [Docker.DotNet 00:00:02.84] HeaderLine="1451"
[Docker.DotNet 00:00:02.84] HeaderLine="0"It seems like we read too much and start extracting the body too early, or more specifically, we begin reading the body when we should actually receive It probably makes sense to debug into these lines and check what's going wrong. I suspect we're missing a case here. Or If you add |
|
Can the issue be the following if (read == 0) check? When _chunkBytesRemaining is 0, can we safely set _done on true instead of waiting on the next iteration from Stream.cs CopyToAsync? It still doesn't make sense to me why this only happens during debugging. |
|
If I read line by line after reading the first "_chunkBytesRemaining" I get the following. |
|
I believe this is heading in the right direction, but I still don't understand why it fails in your environment. Unfortunately, there is no previous maintainer available that we can ask AFAIK, but I've fixed a couple of similar issues in the past, and I think I'm seeing some familiar behavior here. As you mentioned, it seems like we're calling or processing the InstanceCount=1
{"status":"Pulling from library/postgres","id":"alpine"}
InstanceCount=2
{"status":"Digest: sha256:7062a2109c4b51f3c792c7ea01e83ed12ef9a980886e3b3d380a7d2e5f6ce3f5"}
{"status":"Status: Image is up to date for postgres:alpine"}
InstanceCount=3
{"status":"Digest: sha256:7062a2109c4b51f3c792c7ea01e83ed12ef9a980886e3b3d380a7d2e5f6ce3f5"}
{"status":"Status: Image is up to date for postgres:alpine"}
InstanceCount=4
{"Id":"fba67eede3cacd9383049b12daf22de8aa5b45f11099c58f5ac845b95c43147f","Warnings":[]}
f5"}
{"status":"Status: Image is up to date for postgres:alpine"}
InstanceCount=5
{"Id":"fba67eede3cacd9383049b12daf22de8aa5b45f11099c58f5ac845b95c43147f","Warnings":[]}
f5"}
{"status":"Status: Image is up to date for postgres:alpine"}
InstanceCount=6
{"Id":"fba67eede3cacd9383049b12daf22de8aa5b45f11099c58f5ac845b95c43147f","Created":"2025-03-19T07:56:32.493819596Z","Path":"docker-entrypoint.sh","Args":["postgres"],"State":{"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":1560,"ExitCode":0,"Error":"","StartedAt":"2025-03-19T07:56:32.574025732Z","FinishedAt":"0001-01-01T00:00:00Z"},"Image":"sha256:7062a2109c4b51f3c792c7ea01e83ed12ef9a980886e3b3d380a7d2e5f6ce3f5","ResolvConfPath":"/var/lib/docker/containers/fba67eede3cacd9383049b12daf22de8aa5b45f11099c58f5ac845b95c43147f/resolv.conf","HostnamePath":"/var/lib/docker/containers/fba67eede3cacd9383049b12daf22de8aa5b45f11099c58f5ac845b95c43147f/hostname","HostsPath":"/var/lib/docker/containers/fba67eede3cacd9383049b12daf22de8aa5b45f11099c58f5ac845b95c43147f/hosts","LogPath":"/var/lib/docker/containers/fba67eede3cacd9383049b12daf22de8aa5b45f11099c58f5ac845b95c43147f/fba67eede3cacd9383049b12daf22de8aa5b45f11099c58f5ac845b95c43147f-json.log","Name":"/musin
InstanceCount=7
g_bhaskara","RestartCount":0,"Driver":"overlayfs","Platform":"linux","MountLabel":"","ProcessLabel":"","AppArmorProfile":"","ExecIDs":null,"HostConfig":{"Binds":null,"ContainerIDFile":"","LogConfig":{"Type":"json-file","Config":{}},"NetworkMode":"bridge","PortBindings":{},"RestartPolicy":{"Name":"no","MaximumRetryCount":0},"AutoRemove":false,"VolumeDriver":"","VolumesFrom":null,"ConsoleSize":[0,0],"CapAdd":null,"CapDrop":null,"CgroupnsMode":"host","Dns":null,"DnsOptions":null,"DnsSearch":null,"ExtraHosts":null,"GroupAdd":null,"IpcMode":"private","Cgroup":"","Links":null,"OomScoreAdj":0,"PidMode":"","Privileged":false,"PublishAllPorts":false,"ReadonlyRootfs":false,"SecurityOpt":null,"UTSMode":"","UsernsMode":"","ShmSize":67108864,"Runtime":"runc","Isolation":"","CpuShares":0,"Memory":0,"NanoCpus":0,"CgroupParent":"","BlkioWeight":0,"BlkioWeightDevice":null,"BlkioDeviceReadBps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteIOps":null,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpusetCpus":"","CpusetMems":"","Devices":null,"DeviceCgroupRules":null,"DeviceRequests":null,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":null,"OomKillDisable":false,"PidsLimit":null,"Ulimits":null,"CpuCount":0,"CpuPercent":0,"IOMaximumIOps":0,"IOMaximumBandwidth":0,"MaskedPaths":["/proc/asound","/proc/acpi","/proc/kcore","/proc/keys","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug","/proc/scsi","/sys/firmware","/sys/devices/virtual/powercap"],"ReadonlyPaths":["/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"]},"GraphDriver":{"Data":null,"Name":"overlayfs"},"Mounts":[{"Type":"volume","Name":"0d4b3ef8430c8a37a47574415688de9f62474b12614fc6f456846f42ee96c622","Source":"/var/lib/docker/volumes/0d4b3ef8430c8a37a47574415688de9f62474b12614fc6f456846f42ee96c622/_data","Destination":"/var/lib/postgresql/data","Driver":"local","Mode":"","RW":true,"Propagation":""}],"Config":{"Hostname":"fba67eede3ca","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"ExposedPorts":{"5432/tcp":{}},"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["POSTGRES_PASSWORD=password","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","GOSU_VERSION=1.17","LANG=en_US.utf8","PG_MAJOR=17","PG_VERSION=17.4","PG_SHA256=c4605b73fea11963406699f949b966e5d173a7ee0ccaef8938dec0ca8a995fe7","DOCKER_PG_LLVM_DEPS=llvm19-dev \t\tclang19","PGDATA=/var/lib/postgresql/data"],"Cmd":["postgres"],"Image":"postgres:alpine","Volumes":{"/var/lib/postgresql/data":{}},"WorkingDir":"/","Entrypoint":["docker-entrypoint.sh"],"OnBuild":null,"Labels":{},"StopSignal":"SIGINT"},"NetworkSettings":{"Bridge":"","SandboxID":"f67503d0505c06e18c636e60772c38287e6ab8d795c61b2dd79459dc00659c32","SandboxKey":"/var/run/docker/netns/f67503d0505c","Ports":{"5432/tcp":null},"HairpinMode":false,"LinkLocalIPv6Address":"","LinkLocalIPv6PrefixLen":0,"SecondaryIPAddresses":null,"SecondaryIPv6Addresses":null,"EndpointID":"3a400848514ac1e99cd21047b7fdbcaf83bb72027388f6dd3c999a146f497999","Gateway":"172.17.0.1","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"IPAddress":"172.17.0.8","IPPrefixLen":16,"IPv6Gateway":"","MacAddress":"02:42:ac:11:00:08","Networks":{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"MacAddress":"02:42:ac:11:00:08","DriverOpts":null,"NetworkID":"ab4fca983793173a23c56eccdd4d280b5a0ae86e4b72f212a74edfc637ec990b","EndpointID":"3a400848514ac1e99cd21047b7fdbcaf83bb72027388f6dd3c999a146f497999","Gateway":"172.17.0.1","IPAddress":"172.17.0.8","IPPrefixLen":16,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DNSNames":null}}}}
InstanceCount=8
g_bhaskara","RestartCount":0,"Driver":"overlayfs","Platform":"linux","MountLabel":"","ProcessLabel":"","AppArmorProfile":"","ExecIDs":null,"HostConfig":{"Binds":null,"ContainerIDFile":"","LogConfig":{"Type":"json-file","Config":{}},"NetworkMode":"bridge","PortBindings":{},"RestartPolicy":{"Name":"no","MaximumRetryCount":0},"AutoRemove":false,"VolumeDriver":"","VolumesFrom":null,"ConsoleSize":[0,0],"CapAdd":null,"CapDrop":null,"CgroupnsMode":"host","Dns":null,"DnsOptions":null,"DnsSearch":null,"ExtraHosts":null,"GroupAdd":null,"IpcMode":"private","Cgroup":"","Links":null,"OomScoreAdj":0,"PidMode":"","Privileged":false,"PublishAllPorts":false,"ReadonlyRootfs":false,"SecurityOpt":null,"UTSMode":"","UsernsMode":"","ShmSize":67108864,"Runtime":"runc","Isolation":"","CpuShares":0,"Memory":0,"NanoCpus":0,"CgroupParent":"","BlkioWeight":0,"BlkioWeightDevice":null,"BlkioDeviceReadBps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteIOps":null,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpusetCpus":"","CpusetMems":"","Devices":null,"DeviceCgroupRules":null,"DeviceRequests":null,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":null,"OomKillDisable":false,"PidsLimit":null,"Ulimits":null,"CpuCount":0,"CpuPercent":0,"IOMaximumIOps":0,"IOMaximumBandwidth":0,"MaskedPaths":["/proc/asound","/proc/acpi","/proc/kcore","/proc/keys","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug","/proc/scsi","/sys/firmware","/sys/devices/virtual/powercap"],"ReadonlyPaths":["/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"]},"GraphDriver":{"Data":null,"Name":"overlayfs"},"Mounts":[{"Type":"volume","Name":"0d4b3ef8430c8a37a47574415688de9f62474b12614fc6f456846f42ee96c622","Source":"/var/lib/docker/volumes/0d4b3ef8430c8a37a47574415688de9f62474b12614fc6f456846f42ee96c622/_data","Destination":"/var/lib/postgresql/data","Driver":"local","Mode":"","RW":true,"Propagation":""}],"Config":{"Hostname":"fba67eede3ca","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"ExposedPorts":{"5432/tcp":{}},"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["POSTGRES_PASSWORD=password","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","GOSU_VERSION=1.17","LANG=en_US.utf8","PG_MAJOR=17","PG_VERSION=17.4","PG_SHA256=c4605b73fea11963406699f949b966e5d173a7ee0ccaef8938dec0ca8a995fe7","DOCKER_PG_LLVM_DEPS=llvm19-dev \t\tclang19","PGDATA=/var/lib/postgresql/data"],"Cmd":["postgres"],"Image":"postgres:alpine","Volumes":{"/var/lib/postgresql/data":{}},"WorkingDir":"/","Entrypoint":["docker-entrypoint.sh"],"OnBuild":null,"Labels":{},"StopSignal":"SIGINT"},"NetworkSettings":{"Bridge":"","SandboxID":"f67503d0505c06e18c636e60772c38287e6ab8d795c61b2dd79459dc00659c32","SandboxKey":"/var/run/docker/netns/f67503d0505c","Ports":{"5432/tcp":null},"HairpinMode":false,"LinkLocalIPv6Address":"","LinkLocalIPv6PrefixLen":0,"SecondaryIPAddresses":null,"SecondaryIPv6Addresses":null,"EndpointID":"3a400848514ac1e99cd21047b7fdbcaf83bb72027388f6dd3c999a146f497999","Gateway":"172.17.0.1","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"IPAddress":"172.17.0.8","IPPrefixLen":16,"IPv6Gateway":"","MacAddress":"02:42:ac:11:00:08","Networks":{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"MacAddress":"02:42:ac:11:00:08","DriverOpts":null,"NetworkID":"ab4fca983793173a23c56eccdd4d280b5a0ae86e4b72f212a74edfc637ec990b","EndpointID":"3a400848514ac1e99cd21047b7fdbcaf83bb72027388f6dd3c999a146f497999","Gateway":"172.17.0.1","IPAddress":"172.17.0.8","IPPrefixLen":16,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DNSNames":null}}}}Could you please try setting if (_chunkBytesRemaining == 0)
{
// End of chunk, read the terminator CRLF
var trailer = await _inner.ReadLineAsync(cancellationToken).ConfigureAwait(false);
if (trailer.Length > 0)
{
throw new IOException("Invalid chunk trailer");
}
_done = true;
} |
|
I refactored some parts related to reading from chunked streams and added additional trace messages to identify the issue. I may have already fixed it by refactoring the code. It would be great if you could try to reproduce the issue again using the following test (branch: feature/add-logger-interface): Docker.DotNet/test/Docker.DotNet.Tests/DebugHttpRequestResponse.cs Lines 17 to 28 in 6767e01 I will finalize it and try to integrate it into Testcontainers for .NET to enable tracing of the raw communication. |
|
I've managed to test it again using commit 6767e01. Running without breakpoints (no issue) Placing a single breakpoint on if (_done) and continue on break produces following exception: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Testcontainers version
4.3.0
Using the latest Testcontainers version?
Yes
Host OS
Windows
Host arch
x64
.NET version
8.0.405
Docker version
Client: Version: 27.5.1-rd API version: 1.45 (downgraded from 1.47) Go version: go1.22.11 Git commit: 0c97515 Built: Thu Jan 23 18:14:31 2025 OS/Arch: windows/amd64 Context: default Server: Engine: Version: 26.1.5 API version: 1.45 (minimum version 1.24) Go version: go1.22.5 Git commit: 411e817ddf710ff8e08fa193da80cb78af708191 Built: Fri Jul 26 17:51:06 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.7.17 GitCommit: 3a4de459a68952ffb703bbe7f2290861a75b6b67 runc: Version: 1.1.14 GitCommit: 2c9f5602f0ba3d9da1c2596322dfc4e156844890 docker-init: Version: 0.19.0 GitCommit:Docker info
What happened?
Executing Testcontainers.PostgreSql: PostgreSqlContainer.StartAsync sometimes fails. This results in a flaky test startup.
We're using Rancher Desktop v 1.18.2 with moby.
Following code is executed during our NUnit OneTimeSetup:
It's always the same exception: HttpRequestException from ContainerOperations.InspectContainerAsync.
Producing one of the following HttpRequestException
Exception 1 during TestcontainersClient.StartAsync
With Stacktrace:
OR
Exception 2 during DockerContainer.CheckReadinessAsync
With stacktrace:
Relevant log output
Additional information
No response
The text was updated successfully, but these errors were encountered: