EPIPE error on Postgres snapshot on Github Actions

[PieterJanVdb]

Expected Behaviour
For the snapshot to execute successfully in a Github Actions environment.

Actual Behaviour
When executing await postgresContainer.snapshot(); during CI, it throws an EPIPE error. It works fine locally on my machine (MacOS M1 with Colima)

Testcontainer Logs
(Patched the @testcontainers/postgresql package to have this.exec include { logs: true } to have the log output of exec for this)
(I've redacted the db name)

2025-04-02T11:35:33.864Z testcontainers:containers [800f65a7b637] The files belonging to this database system will be owned by user "postgres".
2025-04-02T11:35:33.864Z testcontainers:containers [800f65a7b637] This user must also own the server process.
2025-04-02T11:35:33.865Z testcontainers:containers [800f65a7b637] 
2025-04-02T11:35:33.865Z testcontainers:containers [800f65a7b637] The database cluster will be initialized with locale "en_US.utf8".
2025-04-02T11:35:33.865Z testcontainers:containers [800f65a7b637] The default database encoding has accordingly been set to "UTF8".
2025-04-02T11:35:33.865Z testcontainers:containers [800f65a7b637] The default text search configuration will be set to "english".
2025-04-02T11:35:33.865Z testcontainers:containers [800f65a7b637] 
2025-04-02T11:35:33.865Z testcontainers:containers [800f65a7b637] Data page checksums are disabled.
2025-04-02T11:35:33.865Z testcontainers:containers [800f65a7b637] 
2025-04-02T11:35:33.865Z testcontainers:containers [800f65a7b637] fixing permissions on existing directory /var/lib/postgresql/data ... ok
2025-04-02T11:35:33.866Z testcontainers:containers [800f65a7b637] creating subdirectories ... ok
2025-04-02T11:35:33.866Z testcontainers:containers [800f65a7b637] selecting dynamic shared memory implementation ... posix
2025-04-02T11:35:33.874Z testcontainers:containers [800f65a7b637] selecting default max_connections ... 100
2025-04-02T11:35:33.889Z testcontainers:containers [800f65a7b637] selecting default shared_buffers ... 128MB
2025-04-02T11:35:33.961Z testcontainers:containers [800f65a7b637] selecting default time zone ... UTC
2025-04-02T11:35:33.963Z testcontainers:containers [800f65a7b637] creating configuration files ... ok
2025-04-02T11:35:34.108Z testcontainers:containers [800f65a7b637] running bootstrap script ... ok
2025-04-02T11:35:34.328Z testcontainers:containers [800f65a7b637] sh: locale: not found
2025-04-02T11:35:34.329Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:34.328 UTC [36] WARNING:  no usable system locales were found
2025-04-02T11:35:34.881Z testcontainers:containers [800f65a7b637] performing post-bootstrap initialization ... ok
2025-04-02T11:35:34.977Z testcontainers:containers [800f65a7b637] syncing data to disk ... ok
2025-04-02T11:35:34.977Z testcontainers:containers [800f65a7b637] 
2025-04-02T11:35:34.978Z testcontainers:containers [800f65a7b637] 
2025-04-02T11:35:34.978Z testcontainers:containers [800f65a7b637] Success. You can now start the database server using:
2025-04-02T11:35:34.978Z testcontainers:containers [800f65a7b637] 
2025-04-02T11:35:34.978Z testcontainers:containers [800f65a7b637] pg_ctl -D /var/lib/postgresql/data -l logfile start
2025-04-02T11:35:34.978Z testcontainers:containers [800f65a7b637] 
2025-04-02T11:35:34.978Z testcontainers:containers [800f65a7b637] initdb: warning: enabling "trust" authentication for local connections
2025-04-02T11:35:34.978Z testcontainers:containers [800f65a7b637] You can change this by editing pg_hba.conf or using the option -A, or
2025-04-02T11:35:34.978Z testcontainers:containers [800f65a7b637] --auth-local and --auth-host, the next time you run initdb.
2025-04-02T11:35:35.001Z testcontainers:containers [800f65a7b637] waiting for server to start....2025-04-02 11:35:35.001 UTC [53] LOG:  starting PostgreSQL 13.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20210424) 10.3.1 20210424, 64-bit
2025-04-02T11:35:35.002Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.002 UTC [53] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-04-02T11:35:35.006Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.006 UTC [54] LOG:  database system was shut down at 2025-04-02 11:35:34 UTC
2025-04-02T11:35:35.009Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.009 UTC [53] LOG:  database system is ready to accept connections
2025-04-02T11:35:35.052Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.051 UTC [67] FATAL:  database "[our db name]" does not exist
2025-04-02T11:35:35.084Z testcontainers:containers [800f65a7b637] done
2025-04-02T11:35:35.084Z testcontainers:containers [800f65a7b637] server started
2025-04-02T11:35:35.156Z testcontainers:containers [800f65a7b637] CREATE DATABASE
2025-04-02T11:35:35.157Z testcontainers:containers [800f65a7b637] 
2025-04-02T11:35:35.157Z testcontainers:containers [800f65a7b637] 
2025-04-02T11:35:35.158Z testcontainers:containers [800f65a7b637] /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
2025-04-02T11:35:35.158Z testcontainers:containers [800f65a7b637] 
2025-04-02T11:35:35.158Z testcontainers:containers [800f65a7b637] waiting for server to shut down....2025-04-02 11:35:35.158 UTC [53] LOG:  received fast shutdown request
2025-04-02T11:35:35.160Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.159 UTC [53] LOG:  aborting any active transactions
2025-04-02T11:35:35.160Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.160 UTC [53] LOG:  background worker "logical replication launcher" (PID 60) exited with exit code 1
2025-04-02T11:35:35.161Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.161 UTC [55] LOG:  shutting down
2025-04-02T11:35:35.171Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.171 UTC [53] LOG:  database system is shut down
2025-04-02T11:35:35.259Z testcontainers:containers [800f65a7b637] done
2025-04-02T11:35:35.259Z testcontainers:containers [800f65a7b637] server stopped
2025-04-02T11:35:35.259Z testcontainers:containers [800f65a7b637] 
2025-04-02T11:35:35.259Z testcontainers:containers [800f65a7b637] PostgreSQL init process complete; ready for start up.
2025-04-02T11:35:35.259Z testcontainers:containers [800f65a7b637] 
2025-04-02T11:35:35.276Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.275 UTC [1] LOG:  starting PostgreSQL 13.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20210424) 10.3.1 20210424, 64-bit
2025-04-02T11:35:35.276Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.275 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2025-04-02T11:35:35.276Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.275 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2025-04-02T11:35:35.277Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.277 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-04-02T11:35:35.280Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.280 UTC [74] LOG:  database system was shut down at 2025-04-02 11:35:35 UTC
2025-04-02T11:35:35.284Z testcontainers:containers [800f65a7b637] 2025-04-02 11:35:35.283 UTC [1] LOG:  database system is ready to accept connections
2025-04-02T11:22:03.368Z testcontainers:exec [1dd5a6b94e09] UPDATE 0

2025-04-02T11:22:03.429Z testcontainers:exec [1dd5a6b94e09] DROP DATABASE

2025-04-02T11:22:03.430Z testcontainers:exec [1dd5a6b94e09] NOTICE:  database "migrated_template" does not exist, skipping

stderr | [test name]
Failed to execute command: CREATE DATABASE "migrated_template" WITH TEMPLATE "[our db name]" OWNER "postgres" Error: write EPIPE
  at afterWriteDispatched (node:internal/stream_base_commons:159:15)
  at writeGeneric (node:internal/stream_base_commons:150:3)
  at Socket._writeGeneric (node:net:971:11)
  at Socket._write (node:net:983:8)
  at doWrite (node:internal/streams/writable:598:12)
  at clearBuffer (node:internal/streams/writable:783:7)
  at onwrite (node:internal/streams/writable:653:7)
  at afterWriteDispatched (node:internal/stream_base_commons:162:9)
  at writeGeneric (node:internal/stream_base_commons:150:3)
  at Socket._writeGeneric (node:net:971:11) {
errno: -32,
code: 'EPIPE',
syscall: 'write'
}

Steps to Reproduce

1. In this environment: Github Actions
2. With this config: nothing to specify here
3. Run:

const postgresContainer = await new PostgreSqlContainer()
    .withDatabase(config.name)
    .withUsername(config.user)
    .withPassword(config.password)
    .start();

await postgresContainer.snapshot();

5. See error:

Failed to execute command: CREATE DATABASE "migrated_template" WITH TEMPLATE "[our table name]" OWNER "postgres" Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:159:15)
    at writeGeneric (node:internal/stream_base_commons:150:3)
    at Socket._writeGeneric (node:net:971:11)
    at Socket._write (node:net:983:8)
    at doWrite (node:internal/streams/writable:598:12)
    at clearBuffer (node:internal/streams/writable:783:7)
    at onwrite (node:internal/streams/writable:653:7)
    at afterWriteDispatched (node:internal/stream_base_commons:162:9)
    at writeGeneric (node:internal/stream_base_commons:150:3)
    at Socket._writeGeneric (node:net:971:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

Environment Information

• Operating System: ubuntu-latest
• Docker Version: 24.0.7
• Node version: 22
• Testcontainers version: 10.23.0

[cristianrgreco]

@Nikola-Milovic don't suppose you could look into this one?

We also run the PG snapshot tests in GHA on ubuntu runners, though 22.04, not 24.04 which I assume is latest. @PieterJanVdb could you please try running on ubuntu-22.04? These are the tests we run:

testcontainers-node/packages/modules/postgresql/src/postgresql-container.test.ts

Line 117 in 17ae9b5

describe("PostgreSqlContainer snapshot and restore", { timeout: 180_000 }, () => {

Perhaps you could debug to see where's the difference?

[Nikola-Milovic]

@cristianrgreco this seems like an issue with executing command inside the container in general, not really related to snapshotting functionality.

I am guessing actions are using docker in docker which could pose a problem.

If there is no way to solve it, we can always look into providing an interface for executing the sql commands that you can pass into the container. I prefer executing queries with a client as opposed to the docker exec, but nodejs in general has no prefered postgres client and I didnt want to introduce a pg or any transient dependency to users.

So please look into the docker exec issue since it might affect other containers as well. If not I can take a look and try and introduce a solution

[cristianrgreco]

this seems like an issue with executing command inside the container in general, not really related to snapshotting functionality.

@Nikola-Milovic It doesn't look like an issue with container.exec (I'm also not aware of any issues relating to container.exec, regardless of container runtime, DIND, etc).

Calling container.snapshot() invokes these commands:

await this.execCommandsSQL([
  // Update pg_database to remove the template flag, then drop the database if it exists.
  // This is needed because dropping a template database will fail.
  `UPDATE pg_database SET datistemplate = FALSE WHERE datname = '${snapshotName}'`,
  `DROP DATABASE IF EXISTS "${snapshotName}"`,
  // Create a copy of the database to another database to use as a template now that it was fully migrated
  `CREATE DATABASE "${snapshotName}" WITH TEMPLATE "${this.getDatabase()}" OWNER "${this.getUsername()}"`,
  // Snapshot the template database so we can restore it onto our original database going forward
  `ALTER DATABASE "${snapshotName}" WITH is_template = TRUE`,
])

The error is:

Failed to execute command: CREATE DATABASE "migrated_template" WITH TEMPLATE "[our table name]" OWNER "postgres" Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:159:15)
    at writeGeneric (node:internal/stream_base_commons:150:3)
    at Socket._writeGeneric (node:net:971:11)
    at Socket._write (node:net:983:8)
    at doWrite (node:internal/streams/writable:598:12)
    at clearBuffer (node:internal/streams/writable:783:7)
    at onwrite (node:internal/streams/writable:653:7)
    at afterWriteDispatched (node:internal/stream_base_commons:162:9)
    at writeGeneric (node:internal/stream_base_commons:150:3)
    at Socket._writeGeneric (node:net:971:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

Meaning that these commands succeeded:

`UPDATE pg_database SET datistemplate = FALSE WHERE datname = '${snapshotName}'`,
`DROP DATABASE IF EXISTS "${snapshotName}"`,

We can see the same in the logs:

2025-04-02T11:22:03.368Z testcontainers:exec [1dd5a6b94e09] UPDATE 0
2025-04-02T11:22:03.429Z testcontainers:exec [1dd5a6b94e09] DROP DATABASE

And this one failed with EPIPE error:

`CREATE DATABASE "${snapshotName}" WITH TEMPLATE "${this.getDatabase()}" OWNER "${this.getUsername()}"`,

Indicating it isn't a general issue with exec'ing commands against a container. Looks more that PG wasn't able to create the template.

[Nikola-Milovic]

Good catch, my bad, was going over this on my phone, will check it out when I get to my desktop and see what is happening here

[PieterJanVdb]

I might have been a bit too quick on the trigger, I disabled the msw server on a hunch and at first glance the commands seem to be executed correctly. However I previously assumed I'd fixed any issues between msw & testcontainers by:

1. overriding msw to use @mswjs/interceptors version 0.37.5
2. ignoring any requests to localhost & 127.0.0.1

Doing some additional testing (what I don't necessarily understand is the discrepancy between local & CI if it's related to msw)

[cristianrgreco]

I'm not sure about msw here. If it were the cause I'd expect to see issues earlier on, e.g initialising Testcontainers (connecting to the Docker host). Hopefully it is just that but I'd be surprised that in the middle of executing a series of commands against the container, that is would suddenly make it stop working 🤷

Please keep us posted!

[PieterJanVdb]

I somehow got it to fail on a fresh repo after bumbling around for a while trying to replicate the codebase the issue is happening in: https://github.com/PieterJanVdb/testcontainers-msw-repro

I have no idea what is even going on, it doesn't always fail, and it only seems to fail when I run two containers, while it fails consistently on the codebase I originally had the issue in.

Failing run: https://github.com/PieterJanVdb/testcontainers-msw-repro/actions/runs/14230342775/job/39879519770

[cristianrgreco]

I see in the repro you're setting up msw. Are there any failures if you don't use msw?

[PieterJanVdb]

Can't seem to make it error out when there's no msw server running.

[Nikola-Milovic]

@cristianrgreco I still think my initial hypothesis is correct, the migration commands are not failing, the exec itself is. If the any of the commands failed I'd expect to see them logged in postgres as well, but they aren't even reaching the postgres instance.

From the examples Pieter provided, the commands are failing at different parts as well.

I'll look into the sample Pieter provided as well, but I'd look into replicating his setup with any other container and exec commands

[PieterJanVdb]

Apologies for the flaky reproduction code, appreciate you guys looking into it. My first hunch would be that it has something to do with the way docker-modem extends/patches the native http module for redirection purposes.

[cristianrgreco]

My first hunch would be that it has something to do with the way docker-modem extends/patches the native http module for redirection purposes.

The latest version of msw includes an update to the interceptors where this override is no longer an issue: mswjs/interceptors#731. Could you give a try?