[Bug]: Creating a container on BitBucket pipelines fails

[validide]

Testcontainers version

4.1.0

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host arch

x86

.NET version

9.0.1

Docker version

Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:17:06 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Fri Feb 23 02:38:33 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Context:    default
 Debug Mode: false
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Authorization: pipelines
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7c3aca7a610df76212171d200ca3811ff6096eb8
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  userns
  cgroupns
 Kernel Version: 6.1.79
 Operating System: Alpine Linux v3.19 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 5.953GiB
 Name: 0677b265-5571-4bdc-b0b0-0acba8eb9136-8mp2b
 ID: ea36e815-88af-4581-9763-400c44674ec1
 Docker Root Dir: /var/lib/docker/165536.165536
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  10.61.166.30:5000
  127.0.0.0/8
 Registry Mirrors:
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
  http://10.61.166.30:5000/
 Live Restore Enabled: false
 Product License: Community Engine

What happened?

After upgrading to version 4.1.0 the BitBucket pipeline is failing with the following error:

The same code with the4.0.0 version runs without any issues.

I tried to look into the possible cause in the TestContainers code but I got stuck when looking for the Docker.DotNet.Enhanced and Docker.DotNet.Enhanced.X509 sources, can you point me in the right direction?

Relevant log output

Error Message:
   System.AggregateException : One or more errors occurred. (One or more errors occurred. (Docker API responded with status code=Forbidden, response={"message":"authorization denied by plugin pipelines: Invalid request"}
)) (One or more errors occurred. (Docker API responded with status code=Forbidden, response={"message":"authorization denied by plugin pipelines: Invalid request"}
))
---- System.AggregateException : One or more errors occurred. (Docker API responded with status code=Forbidden, response={"message":"authorization denied by plugin pipelines: Invalid request"}
)
-------- Docker.DotNet.DockerApiException : Docker API responded with status code=Forbidden, response={"message":"authorization denied by plugin pipelines: Invalid request"}
---- System.AggregateException : One or more errors occurred. (Docker API responded with status code=Forbidden, response={"message":"authorization denied by plugin pipelines: Invalid request"}
)
-------- Docker.DotNet.DockerApiException : Docker API responded with status code=Forbidden, response={"message":"authorization denied by plugin pipelines: Invalid request"}
  Stack Trace:
  
----- Inner Stack Trace #1 (System.AggregateException) -----
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at MyLibrary.Tests.Fixtures.MariaDbFixture..ctor() in /opt/atlassian/pipelines/agent/build/test/MyLibrary.EntityFramework.Tests/Fixtures/MariaDbFixture.cs:line 18
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
----- Inner Stack Trace -----
   at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers) in /_/src/Docker.DotNet/DockerClient.cs:line 492
   at Docker.DotNet.DockerClient.MakeRequestAsync[T](IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token) in /_/src/Docker.DotNet/DockerClient.cs:line 257
   at Docker.DotNet.ContainerOperations.CreateContainerAsync(CreateContainerParameters parameters, CancellationToken cancellationToken) in /_/src/Docker.DotNet/Endpoints/ContainerOperations.cs:line 64
   at DotNet.Testcontainers.Clients.DockerContainerOperations.RunAsync(IContainerConfiguration configuration, CancellationToken ct) in /_/src/Testcontainers/Clients/DockerContainerOperations.cs:line 213
   at DotNet.Testcontainers.Clients.TestcontainersClient.RunAsync(IContainerConfiguration configuration, CancellationToken ct) in /_/src/Testcontainers/Clients/TestcontainersClient.cs:line 317
   at DotNet.Testcontainers.Containers.DockerContainer.UnsafeCreateAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 416
   at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 280
----- Inner Stack Trace #2 (System.AggregateException) -----
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at MyLibrary.Tests.Fixtures.PgSqlFixture..ctor() in /opt/atlassian/pipelines/agent/build/test/MyLibrary.EntityFramework.Tests/Fixtures/PgSqlFixture.cs:line 18
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
----- Inner Stack Trace -----
   at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers) in /_/src/Docker.DotNet/DockerClient.cs:line 492
   at Docker.DotNet.DockerClient.MakeRequestAsync[T](IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token) in /_/src/Docker.DotNet/DockerClient.cs:line 257
   at Docker.DotNet.ContainerOperations.CreateContainerAsync(CreateContainerParameters parameters, CancellationToken cancellationToken) in /_/src/Docker.DotNet/Endpoints/ContainerOperations.cs:line 64
   at DotNet.Testcontainers.Clients.DockerContainerOperations.RunAsync(IContainerConfiguration configuration, CancellationToken ct) in /_/src/Testcontainers/Clients/DockerContainerOperations.cs:line 213
   at DotNet.Testcontainers.Clients.TestcontainersClient.RunAsync(IContainerConfiguration configuration, CancellationToken ct) in /_/src/Testcontainers/Clients/TestcontainersClient.cs:line 317
   at DotNet.Testcontainers.Containers.DockerContainer.UnsafeCreateAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 416
   at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 280

Additional information

No response

[HofmeisterAn]

Thank you for creating this issue. The root cause of this issue is likely in Docker.DotNet. There is already an issue in the upstream repository: testcontainers/Docker.DotNet#10.

Someone reached out to me on Slack and mentioned they were in contact with Atlassian support. The support team indicated that the issue is related to incompatible Docker versions. Bitbucket is using an older version (25.0.3), whereas we recently upgraded to version 27.3.1. Docker 25.0 supports a maximum API version of 1.44, while Docker 27.3 supports API version 1.47.

The Docker client allows specifying an API version. If someone can verify that downgrading the API version resolves the issue, we can enhance Testcontainers to support the DOCKER_API_VERSION environment variable.

var dockerClient = new DockerClientConfiguration(new Uri("<host>")).CreateClient(new Version(1, 44));
// TODO: Pull an image and create a container instance.

I will close this issue for now. Let's track the progress in the upstream repository until we identify the root cause and determine the next steps.