TalosOS on Podman

[rmvangun]

I'm working towards creating a smooth development experience using TalosOS on Podman for Mac & Windows. I'm new to Podman, and somewhat new to lower-level Linux concepts, and hope this project will help shore up my general knowledge in these areas. I'd like to contain this journey in a single thread and hope this discussion post is the appropriate place.

I'm developing currently on MacOS. I'm running Podman 4.9.2. I've managed to spin up Talos nodes using a docker-compose like this:

version: '3.7'
x-common-config: &common-config
  image: ghcr.io/siderolabs/talos:v1.6.5
  environment:
    - PLATFORM=container
    - TALOSSKU=2CPU-2048RAM
  privileged: true
  security_opt:
    - seccomp=unconfined
  tmpfs:
    - /run
    - /system
    - /tmp
  volumes:
    - type: volume
      target: /system/state
    - type: volume
      target: /var
    - type: volume
      target: /etc/cni
    - type: volume
      target: /etc/kubernetes
    - type: volume
      target: /usr/libexec/kubernetes
    - type: volume
      target: /usr/etc/udev
    - type: volume
      target: /opt

networks:
  cluster-network:
    driver: bridge
    ipam:
      config:
        - subnet: 10.5.0.0/16

services:
  talos-controlplane-1:
    <<: *common-config
    container_name: talos-controlplane-1
    hostname: talos-controlplane-1
    read_only: true
    ports:
      - 50000:50000     # Talos API
      - 6443:6443         # k8s API
    networks:
      cluster-network:
        ipv4_address: 10.5.1.1

  talos-worker-1:
    <<: *common-config
    container_name: talos-worker-1
    hostname: talos-worker-1
    ports:
      - 50001:50000     # Talos API
    volumes:
      - ./data:/data
    networks:
      cluster-network:
        ipv4_address: 10.5.10.1

I've also configured the Podman machine to run as rootful. With this configuration, podman compose up works and I can successfully bootstrap the cluster. However, pods running in the cluster are failing to communicate, at least over HTTP. For instance, I'm running Flux, and the kustomize-controller can't communicate with the source-controller:

dial tcp 10.111.8.208:80: i/o timeout","method":"GET","url":"http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/blueprints/2a35e8e87363c31a4477e2c930e70612a18a8c09.tar.gz

DNS is working, there are no blocking network policies in place, and I can ping pod IPs inside the cluster, so this seems specific to inter-pod HTTP traffic. Where's the next place I should look? This identical configuration works fine using Docker.

[baude]

proxy, vpns?

[rmvangun]

No proxies or VPNs outside of a pretty vanilla setup

[rmvangun]

I've not added anything to the setup concerning additional proxies or VPNs.

Interestingly, my network configuration does not explicitly call out Netavark, so in /usr/share/containers/containers.conf I see:

#network_backend = ""

I then set it to:

network_backend = "netavark"

In /etc/containers/containers.conf

And with this setting, I also can no longer communicate outside the cluster. I'm not sure how to see which network backend I'm using by default, other than looking at the config?

[Luap99]

podman info shows you what network backend is used

[rmvangun]

Ah, OK, confirmed that it defaults to netavark. Sorry, still learning my way around the podman CLI.

[rmvangun]

I managed to create a minimal reproduction of a setup that reproduces this issue. This uses Terraform to perform the Talos bootstrap. There are very specific instructions and it creates a diagnostics pod to illustrate. https://gist.github.com/rmvangun/880224067b2a9f6ac32f89ac8904f4b7

[rmvangun]

Does the demo I posted give enough details to help troubleshoot this matter? I apologize as even this simple reproduction requires a bit of tooling. Any suggestions as to where I can go from here? I'm about to dive in to deeper network troubleshooting on my own but this is probably something very basic for those who have deep familiarity with Podman and container networking.