Ansible playbook failing at "Deploy Database Directly" (Chap 01 / non-standard Vagrant box)

[jeff-tillinghast]

Hi all-- Because I'm running on an M1 Mac, I'm trying to make a non-standard local Vagrant configuration work. I'm running Vagrant with Parallels using box bento/ubuntu-22.04-arm64 (the recommended box per Parallels). In the Vagrantfile, I've changed line 12 to point to the according box, and line 18 to change from :virtual_box to :parallels. The box builds, and the playbook initiates through 'TASK [todo : python prerequisites]', but then it loses the plot-- it skips the next three steps and finally hits failure loop with multiple failures in 'deploy database directly', all referencing 'Invalid kube-config file. No configuration found.':

Output
TASK [todo : python prerequisites] *********************************************
changed: [host01]

TASK [todo : manifests directory] **********************************************
skipping: [host01]

TASK [todo : stage manifests] **************************************************
skipping: [host01] => (item=database.yaml)
skipping: [host01] => (item=application.yaml)
skipping: [host01] => (item=service.yaml)
skipping: [host01] => (item=scaler.yaml)
skipping: [host01]

TASK [todo : deploy database using operator] ***********************************
skipping: [host01]

TASK [todo : deploy database directly] *****************************************
FAILED - RETRYING: [host01]: deploy database directly (10 retries left).
[...]

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible_collections.kubernetes.core.plugins.module_utils.k8s.exceptions.CoreException: Could not create API client: Invalid kube-config file. No configuration found.
failed: [host01] (item=pvc.yaml.j2) => {"ansible_loop_var": "item", "attempts": 10, "changed": false, "item": "pvc.yaml.j2", "msg": "Could not create API client: Invalid kube-config file. No configuration found."}

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible_collections.kubernetes.core.plugins.module_utils.k8s.exceptions.CoreException: Could not create API client: Invalid kube-config file. No configuration found.
failed: [host01] (item=database-secret.yaml.j2) => {"ansible_loop_var": "item", "attempts": 10, "changed": false, "item": "database-secret.yaml.j2", "msg": "Could not create API client: Invalid kube-config file. No configuration found."}

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible_collections.kubernetes.core.plugins.module_utils.k8s.exceptions.CoreException: Could not create API client: Invalid kube-config file. No configuration found.
failed: [host01] (item=database-deploy.yaml.j2) => {"ansible_loop_var": "item", "attempts": 10, "changed": false, "item": "database-deploy.yaml.j2", "msg": "Could not create API client: Invalid kube-config file. No configuration found."}

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible_collections.kubernetes.core.plugins.module_utils.k8s.exceptions.CoreException: Could not create API client: Invalid kube-config file. No configuration found.
failed: [host01] (item=database-service.yaml.j2) => {"ansible_loop_var": "item", "attempts": 10, "changed": false, "item": "database-service.yaml.j2", "msg": "Could not create API client: Invalid kube-config file. No configuration found."}

PLAY RECAP *********************************************************************
host01 : ok=11 changed=10 unreachable=0 failed=1 skipped=3 rescued=0 ignored=0

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

/Output

The only other modification I made to the Vagrant was adding 'ansible.compatibility_mode = "2.0" to both of the 'config.vm.provision' blocks in response to output during my first test run. But adding/removing that condition doesn't affect the results or the output above.

Thanks for any help!

[jeff-tillinghast]

Adding: I was able to use the image to continue through Chapter 1 until I got to the sample application. As a result of the failures above, K3S did not install properly. I manually installed K3S using the script at http://get.k3s.io, and it runs satisfactorily. But obviously the sample application did not install and start, so there are no pods running with the application or database.

[disposedtrolley]

@jeff-tillinghast I think it'll be a challenge to get this working, if it's at all possible.

The following change will allow k3s to install successfully and vagrant up to run without erroring:

diff --git a/setup/roles/k3s/defaults/main.yaml b/setup/roles/k3s/defaults/main.yaml
index 73fc135..5248305 100644
--- a/setup/roles/k3s/defaults/main.yaml
+++ b/setup/roles/k3s/defaults/main.yaml
@@ -1,3 +1,3 @@
 ---
-k3s_version: "v1.23.4+k3s1"
-k3s_url: https://github.com/k3s-io/k3s/releases/download/{{ k3s_version }}/k3s
\ No newline at end of file
+k3s_version: "v1.30.1+k3s1"
+k3s_url: https://github.com/k3s-io/k3s/releases/download/{{ k3s_version }}/k3s-arm64
\ No newline at end of file

The next issue occurs within the VM when K8s attempts to start the todo app:

vagrant@host01:~$ sudo k3s kubectl get pods
NAME                       READY   STATUS             RESTARTS      AGE
todo-db-66d6d87b67-psf98   1/1     Running            0             3m10s
todo-7c94fcc877-6zh69      0/1     CrashLoopBackOff   4 (88s ago)   3m9s
todo-7c94fcc877-5w5gd      0/1     CrashLoopBackOff   4 (80s ago)   3m9s
todo-7c94fcc877-xh7ph      0/1     CrashLoopBackOff   4 (73s ago)   3m9s

Examining the logs:

vagrant@host01:~$ sudo k3s kubectl logs todo-7c94fcc877-6zh69
exec /usr/local/bin/docker-entrypoint.sh: exec format error

which is due to the todo app's container image (referenced in setup/roles/todo/templates/application.yaml.j2) being built for amd64 only: https://hub.docker.com/r/bookofkubernetes/todo

[jdieg0]

Hi @disposedtrolley, @jeff-tillinghast,

thanks for the discussion. This is exactly where I got too with Vagrant and Parallels and the box bento/ubuntu-22.04 on an M1Pro before I started searching through the issues here in the GitHub repo.

vagrant@host01:~$ sudo su -
root@host01:~# k3s kubectl get pods
NAME                       READY   STATUS             RESTARTS      AGE
todo-db-585649889f-msvcg   1/1     Running            0             8m27s
todo-6bd859fdd5-s27ws      0/1     CrashLoopBackOff   6 (86s ago)   8m26s
todo-6bd859fdd5-zssqz      0/1     CrashLoopBackOff   6 (82s ago)   8m26s
todo-6bd859fdd5-bn7br      0/1     CrashLoopBackOff   6 (66s ago)   8m26s

root@host01:~# k3s kubectl logs todo-6bd859fdd5-s27ws
standard_init_linux.go:228: exec user process caused: exec format error
root@host01:~# k3s kubectl describe pod todo-6bd859fdd5-s27ws
Name:         todo-6bd859fdd5-s27ws
Namespace:    default
Priority:     0
Node:         host01/10.211.55.4
Start Time:   Thu, 12 Sep 2024 12:11:17 -0500
Labels:       app=todo
              pod-template-hash=6bd859fdd5
Annotations:  <none>
Status:       Running
IP:           10.42.0.6
IPs:
  IP:           10.42.0.6
Controlled By:  ReplicaSet/todo-6bd859fdd5
Containers:
  todo:
    Container ID:   containerd://adfcb2802330ee236c2da4f3f3735f702324322ff2c0376c689907041f07d955
    Image:          bookofkubernetes/todo:1.0.2
    Image ID:       docker.io/bookofkubernetes/todo@sha256:4e2d05617cce02695d0c8a0b8514176cbc586b3fce9f544cca2e7f263509df77
    Port:           5000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 12 Sep 2024 12:23:23 -0500
      Finished:     Thu, 12 Sep 2024 12:23:23 -0500
    Ready:          False
    Restart Count:  7
    Limits:
      cpu:     50m
      memory:  128Mi
    Requests:
      cpu:     50m
      memory:  128Mi
    Environment:
      NODE_ENV:    production
      PREFIX:      /todo
      PGHOST:      todo-db
      PGDATABASE:  todo
      PGUSER:      <set to the key 'POSTGRES_USER' in secret 'todo-db'>      Optional: false
      PGPASSWORD:  <set to the key 'POSTGRES_PASSWORD' in secret 'todo-db'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-54bt5 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-54bt5:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  16m                 default-scheduler  Successfully assigned default/todo-6bd859fdd5-s27ws to host01
  Normal   Pulling    15m                 kubelet            Pulling image "bookofkubernetes/todo:1.0.2"
  Normal   Pulled     14m                 kubelet            Successfully pulled image "bookofkubernetes/todo:1.0.2" in 1m12.166504064s
  Normal   Pulled     13m (x4 over 14m)   kubelet            Container image "bookofkubernetes/todo:1.0.2" already present on machine
  Normal   Created    13m (x5 over 14m)   kubelet            Created container todo
  Normal   Started    13m (x5 over 14m)   kubelet            Started container todo
  Warning  BackOff    48s (x65 over 14m)  kubelet            Back-off restarting failed container

➜  nostarch-kubernetes git:(main) ✗ docker run --platform linux/arm64 -it bookofkubernetes/todo:1.0.2 /bin/sh
Unable to find image 'bookofkubernetes/todo:1.0.2' locally
1.0.2: Pulling from bookofkubernetes/todo
Digest: sha256:4e2d05617cce02695d0c8a0b8514176cbc586b3fce9f544cca2e7f263509df77
Status: Image is up to date for bookofkubernetes/todo:1.0.2
docker: image with reference docker.io/bookofkubernetes/todo:1.0.2 was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64.
See 'docker run --help'.

@AlanHohn Are there any ambitions to provide multi-architecture images, e.g. built with docker buildx? 😇

Best regards,
Diego

[tobias-wilfert]

Sadge, was trying the same thing with VM Fusion.

[cog1to]

In case anyone is still trying to follow the book's examples on M-series Macs or any other arm64 machines: you can create and deploy your own arm64 version of the Todo app pretty easily. I did create one and uploaded it to Docker.io as burn3rk1tty/todo:arm64, so you should be able to use it by modifying the template to point to this image instead of the default one:

diff --git a/setup/roles/todo/templates/application.yaml.j2 b/setup/roles/todo/templates/application.yaml.j2
index 1f1440f..2c61e35 100644
--- a/setup/roles/todo/templates/application.yaml.j2
+++ b/setup/roles/todo/templates/application.yaml.j2
@@ -21,7 +21,7 @@ spec:
     spec:
       containers:
       - name: todo
-        image: bookofkubernetes/todo:{{ todo_version }}
+        image: burn3rk1tty/todo:arm64
         env:
         - name: NODE_ENV
           value: production

For anyone finding this comment when the image is no longer available, here's a quick instruction how to create and upload your own image:

1. Install Docker and create a Docker.io account/login into your existing account
2. In the terminal, do docker login and follow the instructions to be able to communicate with the Docker.io's repositories.
3. Clone the Todo app repo
4. Inside the repo, execute docker build --platform linux/arm64 -t todo:arm64 .. The :arm64 is optional. I used :arm64 to explicitly identify it as a arm64 version of the todo app.
5. Add tag to your build with your username attached: docker tag todo:arm64 <username>/todo:arm64
6. Push it into the docker registry: docker push <username>/todo:arm64
7. Replace the image value in the setup/roles/todo/templates/application.yaml.j2 config with your uploaded image tag, similar to the patch above

Now your Vagrant deployments should be able to pull this image into the deployed VMs.