Skip to content

Race condition when running multiple kameleon builds on the same machine #43

@jonglezb

Description

@jonglezb

When running multiple builds in parallel, one of them often fails when setting up port forwarding in qemu:

Step 24 : bootstrap/start_qemu/start_vm
--> Running the step...
[local] Qemu VM shutdown: in setup section cleaning
[local] Starting qemu...
[local] VNC port: 3
[local] qemu-system-ppc64: Could not set up host forwarding rule 'tcp::50002-:22'

This is caused by a race condition involving two steps:

  • the step bootstrap/prepare_ssh_to_out_context finds a free TCP port and stores it in $SSH_FWD_PORT
  • the step bootstrap/start_qemu.yaml then later uses this variable $SSH_FWD_PORT and causes qemu to bind to this port

Between these two steps, there is a race window of 5-10 seconds during which two kameleon instances can choose the same port:

--> Running the step...
[local] SSH forwarding port: 50001
Step 20 : bootstrap/prepare_ssh_to_out_context/prepare_ssh_config
Step prepare_ssh_to_out_context took: 0 secs
Step 21 : bootstrap/prepare_appliance/generate_ssh_keys
--> Running the step...
...
Step prepare_appliance took: **7 secs**
Step 24 : bootstrap/start_qemu/start_vm
--> Running the step...
[local] Qemu VM shutdown: in setup section cleaning
[local] Starting qemu...
[local] VNC port: 2
[local] qemu-system-ppc64: Could not set up host forwarding rule 'tcp::50001-:22'

A quick & dirty solution would be to randomize the port chosen by __find_free_port. I have no better idea right now.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions