[QUESTION] Can icecc be used on Docker?

[ericriff]

Sorry if this is explained somewhere, I looked around and I couldn't find any info.

I currently have a working icecc setup, but it is based on native (aka non-docker) builds.

I'm trying to migrate my workflow to use Docker instead, specifically vscode's devcontainers.

I have icecc installed and running on my container and iceccmon can see it. Yet when I launch a build from within the container, no jobs are assigned to the slave PCs.

I was wondering if this is a supported workflow and if any specific setup is required.

Thanks,

Eric.

[ericriff]

Just in case, this is the complete setup described on the picture.

• Laptop 1: Ubuntu 20.04, has the icecc scheduler installed. It is running the docker container which I'm using to build, shows up on icecc-mon as the 1a7.. node. This container is based on Ubuntu 22.04.
• Laptop 2: Ubuntu 22.04. Has iceccd installed and runnign.
  Both laptops are directly connected to the router through ethernet.

[ericriff]

Well, as soon as I posted this I made some small progress. Turns out iceccd was not running as root on the container.
Now it looks better

sudo iceccd -vvv
[5944] 2025-02-18 18:52:40: ICECREAM daemon 1.3.1 starting up (nice level 5) 
[5944] 2025-02-18 18:52:40: 16 CPU(s) online on this server
[5944] 2025-02-18 18:52:40: allowing up to 16 active jobs
[5944] 2025-02-18 18:52:40: supported features: env_xz env_zstd
[5944] 2025-02-18 18:52:40: not detaching
[5944] 2025-02-18 18:52:40: already a process group leader
[5944] 2025-02-18 18:52:40: ignoring localhost lo for broadcast
[5944] 2025-02-18 18:52:40: broadcast eth0 172.17.255.255
[5944] 2025-02-18 18:52:40: Netnames:
[5944] 2025-02-18 18:52:40: ICECREAM
[5944] 2025-02-18 18:52:40: starting to listen on all interfaces
[5944] 2025-02-18 18:52:40: ignoring localhost lo for broadcast
[5944] 2025-02-18 18:52:40: broadcast eth0 172.17.255.255
[5944] 2025-02-18 18:52:40: scheduler not yet found/selected.
[5944] 2025-02-18 18:52:40: Suitable scheduler found at 172.17.0.1:8765 (version: 42)
[5944] 2025-02-18 18:52:40: scheduler not yet found/selected.
[5944] 2025-02-18 18:52:43: scheduler is on 172.17.0.1:8765 (net ICECREAM)
[5944] 2025-02-18 18:52:43: Connected to scheduler (I am known as 172.17.0.2)

And icecc mon now properly lists the 16 cores of my container. Just to make sure I killed the iceccd which was running natively on the laptop where the container is running.
But no jobs are being assigned to the slave laptop anyways.

[ericriff]

This is the scheduler log. Nothing relevant to my untrained eye

icecc-scheduler -vvv
[1124312] 2025-02-18 15:57:59: ICECREAM scheduler 1.3.1 starting up, port 8765
[1124312] 2025-02-18 15:57:59: scheduler ready
[1124312] 2025-02-18 15:57:59: ignoring localhost lo for broadcast
[1124312] 2025-02-18 15:57:59: broadcast enx381428ecadc1 192.168.0.255
[1124312] 2025-02-18 15:57:59: broadcast docker0 172.17.255.255
[1124312] 2025-02-18 15:57:59: broadcast lxdbr0 10.163.218.1
[1124312] 2025-02-18 15:57:59: ignoring localhost lo for broadcast
[1124312] 2025-02-18 15:57:59: broadcast enx381428ecadc1 192.168.0.255
[1124312] 2025-02-18 15:57:59: broadcast docker0 172.17.255.255
[1124312] 2025-02-18 15:57:59: broadcast lxdbr0 10.163.218.1
[1124312] 2025-02-18 15:57:59: Received scheduler announcement from 192.168.0.146:54693 (version 42, netname ICECREAM)
[1124312] 2025-02-18 15:57:59: Received scheduler announcement from 172.17.0.1:54693 (version 42, netname ICECREAM)
[1124312] 2025-02-18 15:57:59: Received scheduler announcement from 10.163.218.1:54693 (version 42, netname ICECREAM)
[1124312] 2025-02-18 15:58:00: broadcast from 192.168.0.146:52443 (version 42)
[1124312] 2025-02-18 15:58:00: broadcast from 172.17.0.1:52443 (version 42)
[1124312] 2025-02-18 15:58:00: broadcast from 10.163.218.1:52443 (version 42)
[1124312] 2025-02-18 15:58:02: accepted 192.168.0.146
[1124312] 2025-02-18 15:58:32: broadcast from 172.17.0.2:49940 (version 42)
[1124312] 2025-02-18 15:58:32: broadcast from 192.168.0.42:57233 (version 42)
[1124312] 2025-02-18 15:58:35: accepted 172.17.0.2
[1124312] 2025-02-18 15:58:35: login 1a7f8dd9ab28 protocol version: 42 features: env_xz env_zstd []
[1124312] 2025-02-18 15:58:39: accepted 192.168.0.42
[1124312] 2025-02-18 15:58:39: login eric-Precision-5550 protocol version: 42 features: env_xz env_zstd []

[adriaandegroot]

Yes, it can. Well, maybe it depends on whether you mean "farm out from a docker container" or "send jobs into a docker container". At work-work we have 5 desktop machines, all beefy. They run one or more docker containers and we do our development inside the docker container. Then when running a build (inside the container) the jobs get farmed out to the desktop machines.

I can look up what exactly our setup is, there; it wasn't especially exciting to set up. But, if you mean the other way around, where jobs get sent into a container, I don't know. I think we put some non-zero effort into avoiding that, because our containers are supposed to be short-lived workspaces.

[goatyde]

We run build jobs inside docker, and I mount /var/run/icecc (SUSE Linux) into the container to make it work.

[ericriff]

farm out from a docker container

This is what I'm looking for. At the office we have a bunch of beefy PCs that we want to use for distributed builds. Those run Ubuntu, no docker.
Each dev locally develops within a docker container. But icecc is not sending the jobs to the beefy PCs.
Up until very recently, developers were not using docker on their laptops and on those scenarios ICECC was working as expected.

[ericriff]

I just noticed that if I use a different view on the monitor the nodes get listed as disabled but I couldn't find anything in the docs about this.
What flags a node as disabled?