CI: add docker build workflow

[zoedberg]

Let's add a Docker Build workflow that:

• builds android bindings dockerization
• builds node.js bindings dockerization

As said in #15:

Unfortunately the Docker Build workflow fails for No space left on device. As specified by github docs free runners have 14GB of SSD disk space. Android bindings docker image weights 15.1GB.

So I think our only option is to use a self-hosted runner

[dr-orlovsky]

What do we need to do in order to get self-hosted runner on our server?

[zoedberg]

@dieeasy use zoedberg/rgb-sdk@2aea1d5 (branch CI_docker_build) as a starting point

[nicbus]

some considerations

quickly looked into shrinking down the docker image size but looks like it's temporarily eating up way more than the final ~15GB, so I guess there's nothing we can do to stay in the 14GB limit and a self-hosted runner could be the only way to go

the first concern about a self-hosted runner is that the stock github actions runner is not meant to run unsafe workflows and needs to be isolated, as explained in a dedicated section of About self-hosted runners and stated again in Adding self-hosted runners

the runner will either need to execute on a dedicated host (but it might not be enough), a virtual machine or probably a docker container (provided the runner process is not executed as root)

[nicbus]

is there an available host that can be dedicated to CI?
if so, we can try it out and assess if that would be a viable option for dockerized workflows

minimum hardware requirements should be:

• 2-core CPU
• 8 GB of RAM memory
• 50+ GB of free SSD disk space (to be confirmed)

[dr-orlovsky]

We have host in our company which can be used, but it's 64GB 64 cores, so probably needs some virtualization + it is used for different other stuff (all dockerized). What would be the best way for me to slice a piece of it dedicated for CI? (docker, some VM). OS is the latest Ubuntu

[nicbus]

I may have found a way to avoid unauthorized workflow runs on a self-hosted runner, some compromises are required, though

overview

after trying some possible solutions, this is the way it could work:

• add a github application to the organization, in order to check at the github level (not runner level, where it's too late) and block workflows triggered by PRs that touch critical files
• deploy a self-hosted runner in docker on the mentioned host, knowing it's containerized and won't be told to run workflows executing potential malicious code (unless explicitly requested by an authorized user)

pros and cons

pros:

• the app allows automatic canceling of workflows that modify critical files (as per configuration), preventing possible attacks
• configuration is quite flexible and should allow definition of check and permission models that are secure enough but do not represent a day-to-day inconvenience
• the app can be enabled for each repository selectively and can be suspended if needed
• app configuration updates require a push to the default branch, which is protected and can only be pushed to by authorized members (inhibits circumventing the protection configuration itself)

cons:

• this github app needs to be added to the organization (it's the only one I found that addresses the issue), it will have write access to issues and actions and is unfortunately close source at the moment (issue here)
• there's no integrated (UI) way to run workflows that have been blocked once they've been reviewed (and thus authorized), but simple steps should do the trick (e.g. a commit from an authorized user on top of the blocked PR)

recap

adding a github app (to be discussed) and a dockerized self-hosted runner allows execution of workflows requiring more than 14GB of disk space while avoiding the big security issue arising from self-hosted runners on public projects

does this solution look promising or shall we look for other ways to overcome this runner limitation?

[eladchen]

Hello There 👋🏻

I'm the author of the mentioned app. If there is anything I can help with, let me know :)

[UkolovaOlga]

@dieeasy can you please elaborate more on this part? what steps can be taken?

there's no integrated (UI) way to run workflows that have been blocked once they've been reviewed (and thus authorized), but simple steps should do the trick (e.g. a commit from an authorized user on top of the blocked PR)

[nicbus]

@dieeasy can you please elaborate more on this part? what steps can be taken?

scenario:

1. the github application is configured in order to automatically cancel workflows triggered by commits that include changes to a list of protected files
2. somebody opens a PR that triggers a workflow
3. that workflow gets canceled by the github application (since it modified a forbidden file)
4. an authorized member reviews the PR and decides it's fine and the workflow should run
5. there's no way to "unblock" that workflow from the UI, the protection introduced by the github application needs to be bypassed via git

a possible solution is to configure the github application to always allow workflows triggered by a commit from an authorized member

with that in place, the "proper" way to "unblock" the workflow would be to have an authorized member review/edit and push the change to the forbidden file in a separate PR, then have the original PR rebased on that, thus removing the block as the forbidden file would no more be changed

another possible solution, e.g. in case the PR is fine and it just needs to be checked before merge, would be to have an authorized member push a new commit on top of the PR, which would (by)pass the protection and let workflow run

[dr-orlovsky]

Sorry for missing the line of thoughts, can we just set up some docker thing on our server doing the build and integrating with GitHub actions? Or all the above is required for this scenario?

[nicbus]

can we just set up some docker thing on our server doing the build and integrating with GitHub actions?

IMO no, hardening a runner to the point that it's safe to run any workload without first checking it is not the way we want to go

the main issue here is that github runners are not meant to run unsafe workflows

adding some details: workflows are triggered by github and then assigned to a runner (e.g. depending on labels), so a PR that triggers a workflow would be able to target a self-hosted runner and execute anything, anytime

[UkolovaOlga]

@dr-orlovsky the above described solution is concerning not only because of being pretty complicated, but also because the software is close-source AND demands total write-access to all activities of our GitHub organisation. We are currently talking to the author about auditing the code of the app and hosting the code ourselves (maybe with rewriting some parts of it if needed) + investigating other options/solutions to this CI problem. I'll tag you in the comments later when the results are there.