[Bug]: cannot hijack chunked or content length stream with podman

[victor-lambret]

Testcontainers version

4.5.0

Using the latest Testcontainers version?

Yes

Host OS

MacOs

Host arch

ARM

.NET version

9.0.203

Docker version

Client:        Podman Engine
Version:       5.5.0
API Version:   5.5.0
Go Version:    go1.24.3
Built:         Tue May 13 20:59:13 2025
Build Origin:  brew
OS/Arch:       darwin/arm64

Server:       Podman Engine
Version:      5.2.4
API Version:  5.2.4
Go Version:   go1.22.7
Built:        Mon Oct  7 02:00:00 2024
OS/Arch:      linux/arm64

Docker info

Client:
  APIVersion: 5.5.0
  BuildOrigin: brew
  Built: 1747162753
  BuiltTime: Tue May 13 20:59:13 2025
  GitCommit: ""
  GoVersion: go1.24.3
  Os: darwin
  OsArch: darwin/arm64
  Version: 5.5.0
host:
  arch: arm64
  buildahVersion: 1.37.4
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-2.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 98.93
    systemPercent: 0.6
    userPercent: 0.46
  cpus: 5
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2041
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.10.10-200.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1079021568
  memTotal: 2043764736
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1.20241007140227477357.main.38.g08fbf82.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.0-dev
    package: netavark-1.12.1-1.20241007131025236895.main.62.g47632d8.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.13.0-dev
  ociRuntime:
    name: crun
    package: crun-1.17-1.20241007140634150540.main.7.g7c194cb.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 4f2c23486977b381fd9461150d2c0038b7d918b3
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240906.g6b38f07-1.fc40.aarch64
    version: |
      pasta 0^20240906.g6b38f07-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 3m 41.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 16722006016
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 12
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.2.4
  Built: 1728259200
  BuiltTime: Mon Oct  7 02:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/arm64
  Version: 5.2.4

What happened?

Hi !

Happy Testcontainers user here, thanks for this excellent tool :)

I have a regression that was introduced by an upgrade of Testcontainers from 4.4.0 to 4.5.0:

-    <PackageVersion Include="Testcontainers.MsSql" Version="4.4.0" />
-    <PackageVersion Include="Testcontainers.RabbitMq" Version="4.4.0" />
-    <PackageVersion Include="Testcontainers.PostgreSql" Version="4.4.0" />
+    <PackageVersion Include="Testcontainers.MsSql" Version="4.5.0" />
+    <PackageVersion Include="Testcontainers.RabbitMq" Version="4.5.0" />
+    <PackageVersion Include="Testcontainers.PostgreSql" Version="4.5.0" />

The update works fine for my other colleagues that are using docker, but it fails on my environment and I suspect it's because I'm using podman.

Looking at the error message cannot hijack chunked or content length stream like testcontainers/testcontainers-dotnet#431 or testcontainers/testcontainers-dotnet#1281. Both are related to users with Docker.

On this project we're using both SqlServer and PostGre containers and both are affected by this problem. Here is the code that is use to Build them:

    private readonly MsSqlContainer _msSqlContainer = new MsSqlBuilder()
        .WithImage(MssqlDockerImage)
        .Build();

    private readonly PostgreSqlContainer _pgContainer = new PostgreSqlBuilder()
        .WithPassword("***")
        .WithUsername("***")
        .WithDatabase("***")
        .WithImage(PgDockerImage)
        .WithExposedPort(64268)
        .WithAutoRemove(true)
        .WithCommand("-c", $"max_connections={PostgresContainerMaxConnections}")
        .Build();

I don't really known how to investigate the problem further so I would be very happy for any lead on how to proceed from here. The problem is deterministic on my setup and I can compare it with the previous versions to provide additional informations.

Thanks !

Relevant log output

System.InvalidOperationException
cannot hijack chunked or content length stream
   at Microsoft.Net.Http.Client.HttpConnectionResponseContent.HijackStream() in /_/src/Docker.DotNet/Microsoft.Net.Http.Client/HttpConnectionResponseContent.cs:line 37
   at Docker.DotNet.DockerClient.MakeRequestForHijackedStreamAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken cancellationToken) in /_/src/Docker.DotNet/DockerClient.cs:line 400
   at Docker.DotNet.ExecOperations.StartContainerExecAsync(String id, ContainerExecStartParameters parameters, CancellationToken cancellationToken) in /_/src/Docker.DotNet/Endpoints/ExecOperations.cs:line 58
   at DotNet.Testcontainers.Clients.DockerContainerOperations.ExecAsync(String id, IList`1 command, CancellationToken ct) in /_/src/Testcontainers/Clients/DockerContainerOperations.cs:line 165
   at Testcontainers.PostgreSql.PostgreSqlBuilder.WaitUntil.UntilAsync(IContainer container) in /_/src/Testcontainers.PostgreSql/PostgreSqlBuilder.cs:line 152
   at DotNet.Testcontainers.Containers.DockerContainer.CheckReadinessAsync(WaitStrategy waitStrategy, CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 646
   at DotNet.Testcontainers.Configurations.WaitStrategy.<>c__DisplayClass24_0.<<WaitUntilAsync>g__UntilAsync|0>d.MoveNext() in /_/src/Testcontainers/Configurations/WaitStrategies/WaitStrategy.cs:line 184
--- End of stack trace from previous location ---
   at DotNet.Testcontainers.Configurations.WaitStrategy.WaitUntilAsync(Func`1 wait, TimeSpan interval, TimeSpan timeout, Int32 retries, CancellationToken ct) in /_/src/Testcontainers/Configurations/WaitStrategies/WaitStrategy.cs:line 213
   at DotNet.Testcontainers.Containers.DockerContainer.CheckReadinessAsync(IEnumerable`1 waitStrategies, CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 664
   at DotNet.Testcontainers.Containers.DockerContainer.UnsafeStartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 526
   at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 309
   at Refinement.Tests.Unit.TestHelpers.Database.RefinementDatabaseFixture.InitializeAsync() in /Users/victorlambret/git/cm-refinement-app/test/Refinement.Tests.Unit/TestHelpers/Database/RefinementDatabaseFixture.cs:line 64

Additional information

No response

[HofmeisterAn]

Thanks for creating the issue. This is likely related to Docker.DotNet and the recent update. Before updating the dependency, I tested Podman on Windows and didn't encounter any issues. Could you please try the following two tests on your machine?

Docker.DotNet/test/Docker.DotNet.Tests/IContainerOperationsTests.cs

Line 711 in 6317bf0

public async Task WriteAsync_OnMultiplexedStream_ForwardsInputToPid1Stdin_CompletesPid1Process()

Docker.DotNet/test/Docker.DotNet.Tests/IContainerOperationsTests.cs

Line 748 in 6317bf0

public async Task WriteAsync_OnMultiplexedStream_ForwardsInputToExecStdin_CompletesExecProcess()

[ahaeber]

I've got the same issue with podman on my machine (Fedora Linux 42). Both of the tests you mentioned fail, but in total most of the tests actually fail:

Failed! - Failed: 80, Passed: 9, Skipped: 0, Total: 89, Duration: 137 ms - Docker.DotNet.Tests.dll (net9.0)

Here is the output for the two tests you mentinoed:

Failed Docker.DotNet.Tests.IContainerOperationsTests.WriteAsync_OnMultiplexedStream_ForwardsInputToExecStdin_CompletesExecProcess [1 ms]
  Error Message:
   System.Net.Http.HttpRequestException : Connection failed.
---- System.Net.Sockets.SocketException : Cannot assign requested address
  Stack Trace:
     at Microsoft.Net.Http.Client.ManagedHandler.ProcessRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs:line 165
   at Microsoft.Net.Http.Client.ManagedHandler.SendAsync(HttpRequestMessage httpRequestMessage, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs:line 77
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/DockerClient.cs:line 424
   at Docker.DotNet.DockerClient.MakeRequestForRawResponseAsync(HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, CancellationToken token) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/DockerClient.cs:line 328
   at Docker.DotNet.Models.StreamUtil.MonitorResponseForMessagesAsync[T](Task`1 responseTask, DockerClient client, CancellationToken cancel, IProgress`1 progress) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Endpoints/StreamUtil.cs:line 80
   at Docker.DotNet.Tests.TestFixture.InitializeAsync() in /home/haband/workspace/opensource/Docker.DotNet/test/Docker.DotNet.Tests/TestFixture.cs:line 70
----- Inner Stack Trace -----
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP)
   at Docker.DotNet.DockerClient.<>c__DisplayClass5_0.<<-ctor>b__1>d.MoveNext() in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/DockerClient.cs:line 96
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Docker.DotNet.DockerClient.<>c__DisplayClass5_0.<.ctor>b__1(String host, Int32 port, CancellationToken cancellationToken)
   at Microsoft.Net.Http.Client.ManagedHandler.ProcessRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs:line 154
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Net.Http.Client.ManagedHandler.ProcessRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Net.Http.Client.ManagedHandler.SendAsync(HttpRequestMessage httpRequestMessage, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs:line 77
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Net.Http.Client.ManagedHandler.SendAsync(HttpRequestMessage httpRequestMessage, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/DockerClient.cs:line 424
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken)
   at Docker.DotNet.DockerClient.MakeRequestForRawResponseAsync(HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, CancellationToken token) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/DockerClient.cs:line 328
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Docker.DotNet.DockerClient.MakeRequestForRawResponseAsync(HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, CancellationToken token)
   at Docker.DotNet.ImageOperations.CreateImageAsync(ImagesCreateParameters parameters, Stream imageStream, AuthConfig authConfig, IDictionary`2 headers, IProgress`1 progress, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Endpoints/ImageOperations.cs:line 137
   at Docker.DotNet.ImageOperations.CreateImageAsync(ImagesCreateParameters parameters, Stream imageStream, AuthConfig authConfig, IProgress`1 progress, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Endpoints/ImageOperations.cs:line 107
   at Docker.DotNet.ImageOperations.CreateImageAsync(ImagesCreateParameters parameters, AuthConfig authConfig, IProgress`1 progress, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Endpoints/ImageOperations.cs:line 97
   at Docker.DotNet.Tests.TestFixture.InitializeAsync() in /home/haband/workspace/opensource/Docker.DotNet/test/Docker.DotNet.Tests/TestFixture.cs:line 70
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Docker.DotNet.Tests.TestFixture.InitializeAsync()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Linq.Enumerable.SelectEnumerableIterator`2.ToList()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
--- End of stack trace from previous location ---
   at Docker.DotNet.DockerClient.<>c__DisplayClass5_0.<<-ctor>b__1>d.MoveNext() in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/DockerClient.cs:line 96
--- End of stack trace from previous location ---
   at Microsoft.Net.Http.Client.ManagedHandler.ProcessRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs:line 154

  Failed Docker.DotNet.Tests.IContainerOperationsTests.WriteAsync_OnMultiplexedStream_ForwardsInputToPid1Stdin_CompletesPid1Process [1 ms]
  Error Message:
   System.Net.Http.HttpRequestException : Connection failed.
---- System.Net.Sockets.SocketException : Cannot assign requested address
  Stack Trace:
     at Microsoft.Net.Http.Client.ManagedHandler.ProcessRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs:line 165
   at Microsoft.Net.Http.Client.ManagedHandler.SendAsync(HttpRequestMessage httpRequestMessage, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs:line 77
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/DockerClient.cs:line 424
   at Docker.DotNet.DockerClient.MakeRequestForRawResponseAsync(HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, CancellationToken token) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/DockerClient.cs:line 328
   at Docker.DotNet.Models.StreamUtil.MonitorResponseForMessagesAsync[T](Task`1 responseTask, DockerClient client, CancellationToken cancel, IProgress`1 progress) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Endpoints/StreamUtil.cs:line 80
   at Docker.DotNet.Tests.TestFixture.InitializeAsync() in /home/haband/workspace/opensource/Docker.DotNet/test/Docker.DotNet.Tests/TestFixture.cs:line 70
----- Inner Stack Trace -----
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket, Boolean saeaCancelable)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP)
   at Docker.DotNet.DockerClient.<>c__DisplayClass5_0.<<-ctor>b__1>d.MoveNext() in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/DockerClient.cs:line 96
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Docker.DotNet.DockerClient.<>c__DisplayClass5_0.<.ctor>b__1(String host, Int32 port, CancellationToken cancellationToken)
   at Microsoft.Net.Http.Client.ManagedHandler.ProcessRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs:line 154
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Net.Http.Client.ManagedHandler.ProcessRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Net.Http.Client.ManagedHandler.SendAsync(HttpRequestMessage httpRequestMessage, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs:line 77
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Net.Http.Client.ManagedHandler.SendAsync(HttpRequestMessage httpRequestMessage, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/DockerClient.cs:line 424
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken)
   at Docker.DotNet.DockerClient.MakeRequestForRawResponseAsync(HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, CancellationToken token) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/DockerClient.cs:line 328
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Docker.DotNet.DockerClient.MakeRequestForRawResponseAsync(HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, CancellationToken token)
   at Docker.DotNet.ImageOperations.CreateImageAsync(ImagesCreateParameters parameters, Stream imageStream, AuthConfig authConfig, IDictionary`2 headers, IProgress`1 progress, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Endpoints/ImageOperations.cs:line 137
   at Docker.DotNet.ImageOperations.CreateImageAsync(ImagesCreateParameters parameters, Stream imageStream, AuthConfig authConfig, IProgress`1 progress, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Endpoints/ImageOperations.cs:line 107
   at Docker.DotNet.ImageOperations.CreateImageAsync(ImagesCreateParameters parameters, AuthConfig authConfig, IProgress`1 progress, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Endpoints/ImageOperations.cs:line 97
   at Docker.DotNet.Tests.TestFixture.InitializeAsync() in /home/haband/workspace/opensource/Docker.DotNet/test/Docker.DotNet.Tests/TestFixture.cs:line 70
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Docker.DotNet.Tests.TestFixture.InitializeAsync()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Linq.Enumerable.IEnumerableWhereSelectIterator`2.ToList()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
--- End of stack trace from previous location ---
   at Docker.DotNet.DockerClient.<>c__DisplayClass5_0.<<-ctor>b__1>d.MoveNext() in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/DockerClient.cs:line 96
--- End of stack trace from previous location ---
   at Microsoft.Net.Http.Client.ManagedHandler.ProcessRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /home/haband/workspace/opensource/Docker.DotNet/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs:line 154

[HofmeisterAn]

but in total most of the tests actually fail:

Ah, I forgot to mention that you need to configure the container runtime accordingly. It does not detect it automatically. Pass the endpoint to the following line:

Docker.DotNet/test/Docker.DotNet.Tests/TestFixture.cs

Line 24 in 6317bf0

DockerClientConfiguration = new DockerClientConfiguration();

E.g. for Podman on Windows:

_ = new DockerClientConfiguration(new Uri("npipe://./pipe/podman-machine-default"));

[HofmeisterAn]

Okay, I believe I can reproduce the issue. It seems like the Upgrade header is missing, and instead of hijacking the connection, we're using the chunked stream reader. For Docker isConnectionUpgrade is true. For Podman it is false. I need to check their docs.

Docker.DotNet/src/Docker.DotNet/Microsoft.Net.Http.Client/HttpConnection.cs

Lines 163 to 173 in 6317bf0

	var isConnectionUpgrade = response.Headers.TryGetValues("Upgrade", out var responseHeaderValues)
	&& responseHeaderValues.Any(header => "tcp".Equals(header));
	var isStream = content.Headers.TryGetValues("Content-Type", out var contentHeaderValues)
	&& contentHeaderValues.Any(header => DockerStreamHeaders.Contains(header));
	var isChunkedTransferEncoding = (response.Headers.TransferEncodingChunked.GetValueOrDefault() && !isStream) || (isStream && !isConnectionUpgrade);
	content.ResolveResponseStream(chunked: isChunkedTransferEncoding);
	return response;

[ahaeber]

Thanks for the tip for running the tests properly.

I've tried to debug this code and noticed that the headers sent for the upgrade request do not match what the Podman and Docker Engine APIs request:

Docker.DotNet/src/Docker.DotNet/DockerClient.cs

Lines 389 to 393 in 6317bf0

	headers.Add("Connection", "tcp");
	headers.Add("Upgrade", "Upgrade");
	var response = await PrivateMakeRequestAsync(timeout, HttpCompletionOption.ResponseHeadersRead, method, path, queryString, headers, body, cancellationToken)
	.ConfigureAwait(false);

The headers should be:

Upgrade: tcp
Connection: Upgrade

Ref https://docs.podman.io/en/latest/_static/api.html#tag/containers/operation/ContainerAttachLibpod

To inform potential proxies about connection hijacking, the client can also optionally send connection upgrade headers.

For example, the client sends this request to upgrade the connection:

POST /v4.6.0/libpod/containers/16253994b7c4/attach?stream=1&stdout=1 HTTP/1.1
Upgrade: tcp
Connection: Upgrade

The service will respond with a 101 UPGRADED response, and will similarly follow with the raw stream:

HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp

[STREAM]

The docs at https://docs.docker.com/reference/api/engine/version/v1.50/#tag/Container/operation/ContainerAttach seem to be the same.

[HofmeisterAn]

The headers should be:

Upgrade: tcp
Connection: Upgrade

Good catch 🦅 thanks. I took a quick look and updated it, but Podman still isn't responding with the expected headers.

Docker:
  HTTP/1.1 101 UPGRADED
  Api-Version: 1.50
  Connection: Upgrade
  Content-Type: application/vnd.docker.multiplexed-stream
  Docker-Experimental: false
  Ostype: linux
  Server: Docker/28.2.2 (linux)
  Upgrade: tcp

Podman:
  HTTP/1.1 200 OK
  Content-Type: application/vnd.docker.raw-stream

[ahaeber]

Thanks @HofmeisterAn !

I also tried to run it with the headers updated and see that it still fails. The Podman API logs shows it like this:

@ - - [10/Jun/2025:12:10:47 +0200] "POST /containers/f342e7d9a8b792cf0807e86b905bf7db6bb3cf68b192b199473d20fb4969f438/attach?stream=1&stdin=1&stdout=1&stderr=1&logs=1 HTTP/1.1" 200 0 "" "Docker.DotNet"                         

I tried to modify the test slightly to echo text at the beginning to ensure some data is available in stdin, but that did not change the response.

Curiously, by removing the check for content response type the tests complete successfully.

Docker.DotNet/src/Docker.DotNet/Microsoft.Net.Http.Client/HttpConnectionResponseContent.cs

Lines 35 to 38 in 6317bf0

	if (_responseStream != _connection.Transport)
	{
	throw new InvalidOperationException("cannot hijack chunked or content length stream");
	}

But that is probably not the best fix since the upgrade actually fails.

[HofmeisterAn]

The Podman log above shows the HTTP request, right? How does the response log look (I can check it as well; I just need some time)?

[ahaeber]

That is the request + response from Podman, but I haven't been able to see the actual response data in the Podman logs. Here is what I could find in the logs related to this:

INFO[0225] Performing HTTP Hijack attach to container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67                                                               
DEBU[0225] Successfully connected to container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 attach socket /home/xxxx/.local/share/containers/storage/overlay-containers/9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67/userdata/attach 
DEBU[0225] Going to hijack container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 attach connection 
DEBU[0225] IdleTracker:hijacked 1m+0h/19t connection(s)  X-Reference-Id=0xc0005a0e28
DEBU[0225] Hijack for container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 attach session done, ready to stream 
DEBU[0225] Will stream logs for container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 attach session 
DEBU[0225] Writing logs for container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 to HTTP attach 
DEBU[0225] Done reading logs for container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67, 0 bytes 
DEBU[0225] Forwarding attach output for container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 
DEBU[0225] Performing non-terminal HTTP attach for container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 
DEBU[0251] Removing container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 
DEBU[0251] Stopping ctr 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 (timeout 10) 
DEBU[0251] Stopping container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 (PID 625179) 
DEBU[0251] Sending signal 15 to container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 
DEBU[0261] Timed out stopping container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 with SIGTERM, resorting to SIGKILL: given PID did not die within timeout 
WARN[0261] StopSignal SIGTERM failed to stop container keen_germain in 10 seconds, resorting to SIGKILL 
DEBU[0261] Sending signal 9 to container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 
DEBU[0261] STDOUT/ERR copy completed                    
DEBU[0261] STDIN copy completed                         
DEBU[0261] Attach for container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 completed successfully 
DEBU[0261] IdleTracker:closed 1m+1h/22t connection(s)    X-Reference-Id="%!p(<nil>)"
@ - - [10/Jun/2025:12:54:32 +0200] "POST /containers/9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67/attach?stream=1&stdin=1&stdout=1&stderr=1&logs=1 HTTP/1.1" 200 0 "" "Docker.DotNet"
DEBU[0261] Container "9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67" state changed from "stopping" to "stopped" while waiting for it to be stopped: discontinuing stop procedure as another process interfered 
DEBU[0261] Cleaning up container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 
DEBU[0261] Tearing down network namespace at /run/user/1000/netns/netns-f71d3db8-8e1d-5ae5-1574-d536fd11492f for container 9748b6820174349c539d59d4d9ac4e2d88a6688f344b23df5b5f386030089b67 

What looks weird to me is that at the beginning the hijack operation looks to be handled correctly, but then the container changes to "stopping" before the attach operation is logged ("Attach for container xyz completed successfully") and the HTTP log below it.

I noticed that the documentation specifies that a 200 OK is returned if no upgrade header is found, which could be the reason for this confusion:

Responses
101 no error, hints proxy about hijacking
200 no error, no upgrade header found

Relevant code in Podman (libpod) for checking for the Upgrade header is:
https://github.com/containers/podman/blob/5ff067cdcdf69a877caa9ed03adb2e78916ce58f/libpod/util.go#L214-L225

But I don't understand why the following headers do not match these headers:

        headers.Add("Connection", "Upgrade");
        headers.Add("Upgrade", "tcp");

One detail I noticed in the documentation is that the API does not seem to work differently if it returns 200 OK or 101 UPGRADED:

// To inform potential proxies about connection hijacking, the client
// can also optionally send connection upgrade headers.

I believe that is why the tests succeeded when I removed the check in Microsoft.Net.Http.Client.HttpConnectionResponseContent.HijackStream as I did earlier.

[victor-lambret]

Thanks to the two of you for the investigation. So far there is no difference between what you describe and what I observe locally on my setup.

[HofmeisterAn]

I think I've found the issue, but I'm not totally sure about the implications. After checking the raw communication, I noticed that the Connection header has two values: Upgrade, close. This probably explains:

What looks weird to me is that at the beginning the hijack operation looks to be handled correctly, but then the container changes to "stopping" before the attach operation is logged ("Attach for container xyz completed successfully") and the HTTP log below it.

But I can't figure out why it works when throw new InvalidOperationException(string) is simply commented out.

@ahaeber could you please test the following patch?

diff --git a/src/Docker.DotNet/DockerClient.cs b/src/Docker.DotNet/DockerClient.cs
index aab4e70..2c32db0 100644
--- a/src/Docker.DotNet/DockerClient.cs
+++ b/src/Docker.DotNet/DockerClient.cs
@@ -386,8 +386,8 @@ public sealed class DockerClient : IDockerClient
             headers = new Dictionary<string, string>();
         }
 
-        headers.Add("Connection", "tcp");
-        headers.Add("Upgrade", "Upgrade");
+        headers.Add("Upgrade", "tcp");
+        headers.Add("Connection", "upgrade");
 
         var response = await PrivateMakeRequestAsync(timeout, HttpCompletionOption.ResponseHeadersRead, method, path, queryString, headers, body, cancellationToken)
             .ConfigureAwait(false);
diff --git a/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs b/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs
index eee9198..728cf6e 100644
--- a/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs
+++ b/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs
@@ -141,7 +141,7 @@ public class ManagedHandler : HttpMessageHandler
 
         ProcessUrl(request);
         ProcessHostHeader(request);
-        request.Headers.ConnectionClose = true; // TODO: Connection re-use is not supported.
+        request.Headers.ConnectionClose = !request.Headers.Contains("Connection"); // TODO: Connection reuse is not supported.
 
         ProxyMode proxyMode = DetermineProxyModeAndAddressLine(request);
         Socket socket;

BTW, this is the related raw request:

read(10, "POST /exec/fbe6812b8b19a02dd36027bdebf0ec5bde1cde1466303b1f488261c68db12184/start HTTP/1.1\r\nUser-Agent: Docker.DotNet\r\nUpgrade: tcp\r\nConnection: Upgrade, close\r\nHost: podman-machine-default:80\r\nContent-Type: application/json; charset=utf-8\r\nTransfer-Encoding: chunked\r\n\r\n", 4096) = 271

[ahaeber]

Thanks for the quick patch and all your efforts @HofmeisterAn !

I've made the changes and most tests are OK at my PC :-)

For IContainerOperationsTests the test ListContainersAsync_ContainerExists_Succeeds is the only test that fails, but it also fails when I run it on main so that is likely not related to this change. This is the output from xunit:

Xunit.Sdk.NotEmptyException
Assert.NotEmpty() Failure: Collection was empty
   at Docker.DotNet.Tests.IContainerOperationsTests.ListContainersAsync_ContainerExists_Succeeds() in /home/haband/workspace/opensource/Docker.DotNet/test/Docker.DotNet.Tests/IContainerOperationsTests.cs:line 549
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass47_0.<<InvokeTestMethodAsync>b__1>d.MoveNext() in /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 259
--- End of stack trace from previous location ---
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in /_/src/xunit.core/Sdk/ExceptionAggregator.cs:line 90

Besides that, all Swarm-related tests fail because I don't have that available. In addition there are two test errors for ISystemOperationsTests:

• Docker_IsRunning: Assert.NotNull() Failure: Value is null
• MonitorEventsFiltered_Succeeds: Assert.Equal() Failure: Values differ; Expected: 2; Actual: 0

[HofmeisterAn]

I've made the changes and most tests are OK at my PC :-)

Yes, I noticed that some tests fail with Podman too. Here's what I suggest:

1. Apply the patch mentioned above and release a new patch version. Then, you can explicitly update the transitive dependency (of Docker.DotNet) in your project, and you should be good.
2. Double-check whether closing the connection is truly necessary. I believe it isn't. Currently, the client implementation closes the connection, but perhaps we can remove this entirely.
3. Add Podman to the CI and begin fixing the failing tests.