How to use pasta with (rootless) play kube and the [email protected]

[andrew-kennedy]

I have a bunch of existing containers I generated using a process like this:

1. First, create a docker compose file:

# APPNICENAME=Caddy
# APPDESCRIPTION=A fast HTTP2 webserver written in go.
---
version: '3.8'
services:
  caddy:
    image: caddy:latest
    labels:
      io.containers.autoupdate: registry
    container_name: caddy
    deploy:
      resources:
        limits:
          memory: 256M
    ports:
      - 8080:8080
      - 8443:8443
    environment:
      HOST_IP: 10.0.0.2
      TZ: America/Los_Angeles
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/user/services/caddy/data:/data:Z
      - /home/user/services/caddy/config:/config:Z
      - /home/user/services/caddy/caddyfile:/etc/caddy:Z
networks:
  default:
    name: podman
    external: true

I then launch this container using the podman docker api socket:

docker-compose up -d

2. Generate a kube yaml file:

podman generate kube caddy > caddy.yml

which yields:

# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.4.2

# NOTE: If you generated this yaml from an unprivileged and rootless podman container on an SELinux
# enabled system, check the podman generate kube man page for steps to follow to ensure that your pod/container
# has the right permissions to access the volumes added.
---
apiVersion: v1
kind: Pod
metadata:
  annotations:
    bind-mount-options: /home/user/services/caddy/caddyfile:Z
    io.containers.autoupdate/caddy: registry
  creationTimestamp: "2023-03-14T08:51:00Z"
  labels:
    app: caddy-pod
  name: caddy-pod
spec:
  # I add these next 3 lines to correctly support rootless volumes.
  securityContext:
    seLinuxOptions:
      type: spc_t
  containers:
  - env:
    - name: TZ
      value: America/Los_Angeles
    - name: HOST_IP
      value: 10.0.0.2
    image: docker.io/library/caddy:latest
    name: caddy
    ports:
    - containerPort: 8080
      hostPort: 8080
    - containerPort: 8443
      hostPort: 8443
    resources:
      limits:
        memory: 256Mi
    volumeMounts:
    - mountPath: /etc/caddy
      name: home-user-services-caddy-caddyfile-host-0
    - mountPath: /config
      name: home-user-services-caddy-config-host-1
    - mountPath: /data
      name: home-user-services-caddy-data-host-2
    - mountPath: /etc/localtime
      name: etc-localtime-host-3
      readOnly: true
  volumes:
  - hostPath:
      path: /home/user/services/caddy/caddyfile
      type: Directory
    name: home-user-services-caddy-caddyfile-host-0
  - hostPath:
      path: /home/user/services/caddy/config
      type: Directory
    name: home-user-services-caddy-config-host-1
  - hostPath:
      path: /home/user/services/caddy/data
      type: Directory
    name: home-user-services-caddy-data-host-2
  - hostPath:
      path: /etc/localtime
      type: File
    name: etc-localtime-host-3

3. I then launch this file with a slightly modified [email protected] file that attaches it to the default podman network:

[Unit]
Description=A template for running K8s workloads via podman-kube-play
Documentation=man:podman-kube-play(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
TimeoutStopSec=70
ExecStart=/usr/bin/podman kube play --replace --network podman --service-container=true %I
ExecStop=/usr/bin/podman kube down %I
Type=notify
NotifyAccess=all

[Install]
WantedBy=default.target

So the launch looks like this:

systemctl --user enable --now podman-kube@$(systemd-escape ~/compose/caddy/caddy.yml)

4. If I do this with another container, such as plex, they can resolve each other at podname:port, such as caddy-pod:8080 and plex-pod:23400, which is very desirable. However, the default podman network mode (bridged?) does not handle original source IP correctly in packets.

Finally, the question:

I heard pasta supports correct source IP in its port forwarding by default, but I'm unsure of how to integrate pasta into my workflow above, while also keeping DNS podname lookup/resolution.

Any advice?

[Luap99]

I heard pasta supports correct source IP in its port forwarding by default, but I'm unsure of how to integrate pasta into my workflow above, while also keeping DNS podname lookup/resolution.

This is not possible at the moment. You either slirp4netns/pasta or the named networks which will always force rootlesskit as port forwarder, see #8193

That said I am going to fix this soon to fully integrate pasta with the rootless netns and I will make sure that it will keep the correct source ip.

[Luap99]

I haven't really started working on it. There are some potential problems that I need to figure out.
Following #8193 should work, alternatively keep an eye out for my network related PRs here.
And of course once it is released it will be mentioned in the release notes.

[andrew-kennedy]

@Luap99 I've been following those release notes, I don't really care about source IP anymore, and would prefer pasta's performance, but I can't seem to get container name DNS resolution working with the pasta network. Is there something special you have to do to enable this? the docs are sparse.

[hitesh-pathak]

@Luap99 Are there any updates on this? I'm looking at migrating from docker to podman. And getting source IPs is important for some containers...

[Luap99]

No real update, a lot of infrastructure code has been moved to better enable this but the actual pasta integration is not implemented yet and port forward likely requires much more changes to work correctly.

[eriksjolund]

Recently socket activation support landed in the git master branch of Caddy.
I verified that it's possible to run a socket-activated Caddy container with rootless Podman (and Pasta)
in a custom network and get the correct source IP address.

I wrote some documentation about it:
https://github.com/eriksjolund/podman-caddy-socket-activation