Got "iptables: bulk-prepend-rules: iptables-restore: line 3 failed" when run hello-world pipeline

[vanviethieuanh]

Hi, I'm following this README guide to run the containers using docker-compose on my PC. I generated the key and the containers work fine.

But when I follow the tutorial to create a hello world pipeline and trigger it. The pipeline run failed with the message:

started hello-world/hello-world-job #3

initializing
initializing check: image
selected worker: 9e876159591d
run check: find or create container on worker 9e876159591d: iptables: bulk-prepend-rules: iptables-restore: line 3 failed

run check: find or create container on worker 9e876159591d: iptables: bulk-prepend-rules: iptables-restore: line 3 failed

errored

This is the log from worker container:

{"timestamp":"2024-06-26T16:13:35.787389378Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.get-volume.volume-not-found","data":{"session":"4.1.162.1","volume":"c17fc114-6231-405c-74c1-f4de14a79edf"}}
{"timestamp":"2024-06-26T16:13:35.787417297Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.volume-not-found","data":{"session":"4.1.162","volume":"c17fc114-6231-405c-74c1-f4de14a79edf"}}
{"timestamp":"2024-06-26T16:13:36.069840675Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.get-volume.volume-not-found","data":{"session":"4.1.167.1","volume":"f7797376-6248-45d1-695d-6863100090f9"}}
{"timestamp":"2024-06-26T16:13:36.069872765Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.volume-not-found","data":{"session":"4.1.167","volume":"f7797376-6248-45d1-695d-6863100090f9"}}
{"timestamp":"2024-06-26T16:13:36.076611097Z","level":"info","source":"guardian","message":"guardian.create.start","data":{"handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181"}}
{"timestamp":"2024-06-26T16:13:36.076716783Z","level":"info","source":"guardian","message":"guardian.create.network-depot-setup-bindmounts.start","data":{"handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.2"}}
{"timestamp":"2024-06-26T16:13:36.076878107Z","level":"info","source":"guardian","message":"guardian.create.network-depot-setup-bindmounts.finished","data":{"handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.2"}}
{"timestamp":"2024-06-26T16:13:36.076909031Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.start","data":{"handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.3"}}
{"timestamp":"2024-06-26T16:13:36.078355522Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.depot-create.started","data":{"handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.3.1"}}
{"timestamp":"2024-06-26T16:13:36.078496613Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.depot-create.finished","data":{"handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.3.1"}}
{"timestamp":"2024-06-26T16:13:36.078523110Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.creating","data":{"bundle":"/worker-state/depot/8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","bundlePath":"/worker-state/depot/8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","id":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","logPath":"/worker-state/depot/8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f/create.log","pidFilePath":"/worker-state/depot/8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f/pidfile","runc":"goci.RuncBinary{Path:\"/var/gdn/assets/linux/bin/runc\", Root:\"/run/runc\"}","session":"181.3.2"}}
{"timestamp":"2024-06-26T16:13:36.131620051Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.completing","data":{"bundle":"/worker-state/depot/8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.3.2"}}
{"timestamp":"2024-06-26T16:13:36.131820187Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.finished","data":{"bundle":"/worker-state/depot/8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.3.2"}}
{"timestamp":"2024-06-26T16:13:36.131839946Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.finished","data":{"handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.3"}}
{"timestamp":"2024-06-26T16:13:36.131906771Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.watch.watching","data":{"bundle":"/worker-state/depot/8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.3.2.1"}}
{"timestamp":"2024-06-26T16:13:36.138172406Z","level":"info","source":"guardian","message":"guardian.create.network.started","data":{"handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.7","spec":""}}
{"timestamp":"2024-06-26T16:13:36.138211127Z","level":"info","source":"guardian","message":"guardian.create.network.config-create","data":{"config":{"ContainerHandle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","HostIntf":"wn4o1q4to3ub-0","ContainerIntf":"wn4o1q4to3ub-1","IPTablePrefix":"w--","IPTableInstance":"n4o1q4to3ub","BridgeName":"wbrdg-0a500000","BridgeIP":"10.80.0.1","ContainerIP":"10.80.0.2","ExternalIP":"172.27.0.4","Subnet":{"IP":"10.80.0.0","Mask":"/////A=="},"Mtu":1500,"PluginNameservers":null,"OperatorNameservers":["172.27.0.4"],"AdditionalNameservers":[],"AdditionalHostEntries":null,"PluginSearchDomains":null},"handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.7","spec":""}}
{"timestamp":"2024-06-26T16:13:36.234702466Z","level":"error","source":"guardian","message":"guardian.iptables-runner.command.failed","data":{"argv":["/var/gdn/assets/linux/sbin/iptables-restore","--noflush"],"error":"exit status 1","exit-status":1,"session":"1.30","stderr":"iptables-restore: line 3 failed\n","stdout":"","took":"8.760171ms"}}
{"timestamp":"2024-06-26T16:13:36.234755148Z","level":"info","source":"guardian","message":"guardian.create.network.finished","data":{"handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.7","spec":""}}
{"timestamp":"2024-06-26T16:13:36.234765942Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.start","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8"}}
{"timestamp":"2024-06-26T16:13:36.234771797Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.destroy.started","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8.1"}}
{"timestamp":"2024-06-26T16:13:36.234776268Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.destroy.delete.started","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8.1.1"}}
{"timestamp":"2024-06-26T16:13:36.239186247Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.destroy.delete.state","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8.1.1","state":{"Pid":255,"Status":"running"}}}
{"timestamp":"2024-06-26T16:13:36.239218903Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.destroy.delete.runc-delete.started","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8.1.1.2"}}
{"timestamp":"2024-06-26T16:13:36.344690146Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.destroy.delete.runc-delete.finished","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8.1.1.2"}}
{"timestamp":"2024-06-26T16:13:36.344743845Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.destroy.delete.finished","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8.1.1"}}
{"timestamp":"2024-06-26T16:13:36.344756202Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.destroy.finished","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8.1"}}
{"timestamp":"2024-06-26T16:13:36.400893561Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.watch.done","data":{"bundle":"/worker-state/depot/8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.3.2.1"}}
{"timestamp":"2024-06-26T16:13:36.697791845Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.network-depot-destroy.start","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8.2"}}
{"timestamp":"2024-06-26T16:13:36.698033459Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.network-depot-destroy.finished","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8.2"}}
{"timestamp":"2024-06-26T16:13:36.698080507Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.remove-bundle.destroy.started","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8.4.1"}}
{"timestamp":"2024-06-26T16:13:36.698425486Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.remove-bundle.destroy.finished","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8.4.1"}}
{"timestamp":"2024-06-26T16:13:36.698484553Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.remove-bundle.destroy.started","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8.4.2"}}
{"timestamp":"2024-06-26T16:13:36.698506262Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.remove-bundle.destroy.finished","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8.4.2"}}
{"timestamp":"2024-06-26T16:13:36.698520665Z","level":"info","source":"guardian","message":"guardian.create.create-failed-cleaningup.cleanedup","data":{"cause":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","session":"181.8"}}
{"timestamp":"2024-06-26T16:13:36.698662509Z","level":"error","source":"guardian","message":"guardian.api.garden-server.create.failed","data":{"error":"iptables: bulk-prepend-rules: iptables-restore: line 3 failed\n","request":{"Handle":"8b20f56c-4474-4f79-7ff5-6cbcbabb2a9f","GraceTime":0,"RootFSPath":"raw:///worker-state/volumes/live/c17fc114-6231-405c-74c1-f4de14a79edf/volume","BindMounts":[{"src_path":"/worker-state/volumes/live/f7797376-6248-45d1-695d-6863100090f9/volume","dst_path":"/scratch","mode":1}],"Network":"","Privileged":false,"Limits":{"bandwidth_limits":{},"cpu_limits":{},"disk_limits":{},"memory_limits":{},"pid_limits":{}}},"session":"3.1.144"}}

Did I miss something? I clone the repo and run docker-compose.yml in the example so the worker should have privileged.

Thank you!

[taylorsilva]

This sounds like an error coming from the guardian runtime. I'd try changing the runtime to containerd:

CONCOURSE_RUNTIME=containerd

[taylorsilva]

The error is coming from here in guadian: https://github.com/cloudfoundry/guardian/blob/e40f722656526fe3e6ec214d1653d1535e770ef4/kawasaki/iptables/iptables.go#L96-L114
Looks like they make a script and then try to pass it to iptables-restore on stdin