Skip to content

Commit 4a6b8a2

Browse files
npalmgertjanmaas
npalm
and
gertjanmaas
authored
Feature/config for lambda (#16)
* Merge * Add extra parameters to configure lambda * Experiment with varialbes * Experiment with varialbes * Experiment with varialbes * Experiment with varialbes * Experiment with varialbes * Experiment with varialbes * Experiment with varialbes * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Experiment with dist * Refactor to have predictive names * Refactor to have predictive names * Refactor to have predictive names * Refactor to have predictive names * Refactor to have predictive names * Refactor to have predictive names * Refactor to have predictive names * Refactor to have predictive names * Refactor to have predictive names * Add option to download lambd's * cleanup * Move sqs policy for webhook to root module * Fix release * Refactor to aws_iam_role_policy instead of policy attachment * trigger * cleanup, refactor * cleanup, refactor * Add tagging, and set concurrency for scale lambda to 1 * Update readme, explain offline runners * Update readme, fix typos and restructure some sentences Co-authored-by: Gertjan Maas <[email protected]>
1 parent 18493a0 commit 4a6b8a2

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+474
-192
lines changed

‎.github/workflows/lambda-runner-binaries-syncer.yml

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,14 @@
11
name: Lambda Runner Binaries Syncer
2+
env:
3+
lambda_name: runner-binaries-syncer
4+
lambda_path: modules/runner-binaries-syncer/lambdas/runner-binaries-syncer
25
on:
36
push:
47
branches:
58
- master
69
pull_request:
710
paths:
8-
- .github/workflows/lambda-agent-webhook.yml
11+
- .github/workflows/lambda-runner-binaries-syncer.yml
912
- "modules/runner-binaries-syncer/lambdas/runner-binaries-syncer/**"
1013

1114
jobs:
@@ -14,8 +17,7 @@ jobs:
1417
container: node:12
1518
defaults:
1619
run:
17-
working-directory: modules/runner-binaries-syncer/lambdas/runner-binaries-syncer
18-
20+
working-directory: ${{ env.lambda_path }}
1921
steps:
2022
- uses: actions/checkout@v2
2123
- name: Install dependencies

‎.github/workflows/lambda-scale-runners.yml renamed to ‎.github/workflows/lambda-runners.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,20 @@
1-
name: Lambda Scale Runners
1+
name: Lambda Runners
22
on:
33
push:
44
branches:
55
- master
66
pull_request:
77
paths:
8-
- .github/workflows/lambda-scale-runners.yml
9-
- "modules/runners/lambdas/scale-runners/**"
8+
- .github/workflows/lambda-runners.yml
9+
- "modules/runners/lambdas/runners/**"
1010

1111
jobs:
1212
build:
1313
runs-on: ubuntu-latest
1414
container: node:12
1515
defaults:
1616
run:
17-
working-directory: modules/runners/lambdas/scale-runners
17+
working-directory: modules/runners/lambdas/runners
1818

1919
steps:
2020
- uses: actions/checkout@v2

‎.github/workflows/release.yml

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
name: Release build
2+
3+
on:
4+
push:
5+
tags:
6+
- v*
7+
8+
jobs:
9+
release:
10+
name: Build runners distribution
11+
runs-on: ubuntu-latest
12+
container: node:12
13+
env:
14+
lambda_webhook_name: webhook
15+
lambda_runners_name: runners
16+
lambda_syncer_name: runner-binaries-syncer
17+
steps:
18+
- name: Create Release
19+
id: create_release
20+
uses: actions/create-release@latest
21+
env:
22+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
23+
with:
24+
tag_name: ${{ github.ref }}
25+
release_name: Release ${{ github.ref }}
26+
draft: true
27+
prerelease: true
28+
- name: Extract tag name
29+
id: tag_name
30+
run: echo ::set-output name=TAG::${GITHUB_REF##*/}
31+
- uses: actions/checkout@v2
32+
- name: Add zip
33+
run: apt update && apt install zip
34+
35+
- name: Lambda webhook - build
36+
working-directory: modules/webhook/lambdas/webhook
37+
run: yarn install && yarn dist
38+
- name: Lambda webhook - Upload asset
39+
uses: actions/upload-release-asset@v1
40+
env:
41+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
42+
with:
43+
upload_url: ${{ steps.create_release.outputs.upload_url }}
44+
asset_path: modules/${{ env.lambda_webhook_name }}/lambdas/${{ env.lambda_webhook_name }}/${{ env.lambda_webhook_name }}.zip
45+
asset_name: ${{ env.lambda_webhook_name }}-${{ steps.tag_name.outputs.TAG }}.zip
46+
asset_content_type: application/zip
47+
48+
- name: Lambda syncer - build
49+
working-directory: modules/runner-binaries-syncer/lambdas/runner-binaries-syncer
50+
run: yarn install && yarn dist
51+
- name: Lambda syncer - Upload asset
52+
uses: actions/upload-release-asset@v1
53+
env:
54+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
55+
with:
56+
upload_url: ${{ steps.create_release.outputs.upload_url }}
57+
asset_path: modules/${{ env.lambda_syncer_name }}/lambdas/${{ env.lambda_syncer_name }}/${{ env.lambda_syncer_name }}.zip
58+
asset_name: ${{ env.lambda_syncer_name }}-${{ steps.tag_name.outputs.TAG }}.zip
59+
asset_content_type: application/zip
60+
61+
- name: Lambda runners - build
62+
env:
63+
lambda_path: modules/runners/lambdas/runners
64+
working-directory: modules/runners/lambdas/runners
65+
run: yarn install && yarn dist
66+
- name: Lambda runners - Upload asset
67+
uses: actions/upload-release-asset@v1
68+
env:
69+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
70+
with:
71+
upload_url: ${{ steps.create_release.outputs.upload_url }}
72+
asset_path: modules/${{ env.lambda_runners_name }}/lambdas/${{ env.lambda_runners_name }}/${{ env.lambda_runners_name }}.zip
73+
asset_name: ${{ env.lambda_runners_name }}-${{ steps.tag_name.outputs.TAG }}.zip
74+
asset_content_type: application/zip

‎.github/workflows/terraform.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ jobs:
2121
- name: "Fake zip files" # Validate will fail if it cannot find the zip files
2222
run: |
2323
touch modules/webhook/lambdas/webhook/webhook.zip
24-
touch modules/runners/lambdas/scale-runners/scale-runners.zip
24+
touch modules/runners/lambdas/runners/runners.zip
2525
touch modules/runner-binaries-syncer/lambdas/runner-binaries-syncer/runner-binaries-syncer.zip
2626
- name: "Terraform Format"
2727
uses: hashicorp/terraform-github-actions@master

‎README.md

Lines changed: 79 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,10 @@
11
# Terraform module for scalable self hosted GitHub action runners
22

3-
> WIP: Module is in development
4-
53
This [Terraform](https://www.terraform.io/) modules create the required infra structure needed to host [GitHub Action](https://github.com/features/actions) self hosted runners on [AWS spot instances](https://aws.amazon.com/ec2/spot/). All logic required to handle the lifecycle for an action runners is implemented in AWS Lambda functions.
64

75
## Motivation
86

9-
GitHub Actions `self hosted` runners provides you with a flexible option to run your CI workloads on compute of your choice. Currently there is no option provided to automate the creation and scaling of action runners. This module takes care of creating the AWS infra structure to host action runners on spot instances. And provides lambda modules to orchestrate the lifecycle of the action runners.
7+
GitHub Actions `self hosted` runners provides you with a flexible option to run your CI workloads on compute of your choice. Currently there is no option provided to automate the creation and scaling of action runners. This module takes care of creating the AWS infra structure to host action runners on spot instances. And provides lambda modules to orchestrate the life cycle of the action runners.
108

119
Lambda is chosen as runtime for two major reasons. First it allows to create small components with minimal access to AWS and GitHub. Secondly it provides a scalable setup for minimal costs that works on repo level and scales to organization level. The lambdas will create Linux based EC2 instances with Docker to serve CI workloads that can run on Linux and/or Docker. The main goal is here to support Docker based workloads.
1210

@@ -37,12 +35,15 @@ Besides these permissions, the lambdas also need permission to CloudWatch (for l
3735
Examples are provided in [the example directory](examples/). Please ensure you have installed the following tools.
3836

3937
- Terraform, or [tfenv](https://github.com/tfutils/tfenv).
40-
- Bash shell or compatible.
41-
- TODO: building lambda ?
42-
- AWS cli
38+
- Bash shell or compatible
39+
- Docker (optional, to build lambdas without node).
40+
- AWS cli (optional)
41+
- Node and yarn (for lambda development).
4342

4443
The module support two main scenarios for creating runners. On repository level a runner will be dedicated to only one repository, no other repository can use the runner. On organization level you can use the runner(s) for all the repositories within the organization. See https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners for more information. Before starting the deployment you have to choose one option.
4544

45+
GitHub workflows will fail immediately if there is no action runner available for your builds. Since this module supports to scale from 0 and up, your builds will fail in case there is no active runner available. So we recommend to create an offline runner with matching labels to the configuration. Create this runner by following the GitHub instruction on your local machine. You can stop the process after the step of running the `config.sh`. This offline runner will ensure your builds will not fail immediately and stay queued until there is a runner to pick it up.
46+
4647
The setup consists of running Terraform to create all AWS resources and configure the GitHub App. The Terraform module requires configuration from the GitHub App and the GitHub app requires output from Terraform. Therefore you should first create the GitHub App, configure the basics. Then run Terraform and finalize the configuration of the GitHub App afterwards.
4748

4849
### Setup GitHub App (part 1)
@@ -62,11 +63,41 @@ Go to GitHub and create a new app. Beware you can create apps your organization
6263

6364
### Setup terraform module
6465

65-
1. Create a terraform workspace and initiate the module, see the examples for more details.
66+
First you need to download the lambda releases. The lambda code is available as a GitHub release asset. Downloading can be done with the provided terraform module for example. Note that this requires `curl` to be installed on your machine. Create an empty workspace with the following terraform code:
67+
68+
```terraform
69+
module "lambdas" {
70+
source = "../../../modules/download-lambda"
71+
lambdas = [
72+
{
73+
name = "webhook"
74+
tag = "v0.0.0-beta"
75+
},
76+
{
77+
name = "runners"
78+
tag = "v0.0.0-beta"
79+
},
80+
{
81+
name = "runner-binaries-syncer"
82+
tag = "v0.0.0-beta"
83+
}
84+
]
85+
}
86+
87+
output "files" {
88+
value = module.lambdas.files
89+
}
90+
```
91+
92+
Next run `terraform init && terraform apply` as result the lambdas will be download to the same directory. Alternatively you can download the zip artifacts with any other tool of you favour.
93+
94+
For local development you can build all the lambda's at once using `.ci/build.sh` or per lambda using `yarn dist`.
95+
96+
Next create a second terraform workspace and initiate the module, see the examples for more details.
6697

6798
```terraform
6899
module "runners" {
69-
source = "git::https://github.com/philips-labs/terraform-aws-github-runner/"
100+
source = "git::https://github.com/philips-labs/terraform-aws-github-runner.git?ref=master"
70101
71102
aws_region = "eu-west-1"
72103
vpc_id = "vpc-123"
@@ -78,10 +109,13 @@ module "runners" {
78109
key_base64 = "base64string"
79110
id = "1"
80111
client_id = "c-123"
81-
client_secret = "secret"
82-
webhook_secret = "secret"
112+
client_secret = "client_secret"
113+
webhook_secret = "webhook_secret"
83114
}
84115
116+
webhook_lambda_zip = "lambdas-download/webhook.zip"
117+
runner_binaries_syncer_lambda_zip = "lambdas-download/runner-binaries-syncer.zip.zip"
118+
runners_lambda_zip = "lambdas-download/runners.zip"
85119
enable_organization_runners = true
86120
}
87121
```
@@ -93,7 +127,7 @@ terraform init
93127
terrafrom apply
94128
```
95129

96-
3. Check the terraform output for the API gateway url, which you need in the next step.
130+
Check the terraform output for the API gateway url (endpoint), which you need in the next step. The lambda for syncing the GitHub distribution will be executed by a trigger via CloudWatch. To ensure the binary is cached, trigger the `runner-binaries-syncer` manually. The payload does not matter. (e.g. `aws lambda invoke --function-name <environment>-syncer response.json`)
97131

98132
### Setup GitHub App (part 2)
99133

@@ -104,8 +138,42 @@ Go back to the GitHub App and update the following settings.
104138
3. Provide the webhook secret.
105139
4. Enable the `Check run` event for the webhook.
106140

141+
You are now ready to run action workloads on self hosted runner, remember builds will fail if there is no (offline) runner available with matching labels.
142+
107143
## Examples
108144

145+
TODO
146+
147+
## Inputs
148+
149+
| Name | Description | Type | Default | Required |
150+
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | :------: |
151+
| aws_region | AWS region. | `string` | n/a | yes |
152+
| enable_organization_runners | n/a | `bool` | n/a | yes |
153+
| environment | A name that identifies the environment, used as prefix and for tagging. | `string` | n/a | yes |
154+
| github_app | GitHub app parameters, see your github aapp. Ensure the key is base64 encoded. | <pre>object({<br> key_base64 = string<br> id = string<br> client_id = string<br> client_secret = string<br> webhook_secret = string<br> })</pre> | n/a | yes |
155+
| subnet_ids | List of subnets in which the action runners will be launched, the subnets needs to be subnets in the `vpc_id`. | `list(string)` | n/a | yes |
156+
| vpc_id | The VPC for security groups of the action runners. | `string` | n/a | yes |
157+
| minimum_running_time_in_minutes | The time an ec2 action runner should be running at minium before terminated if non busy. | `number` | `5` | no |
158+
| runner_binaries_syncer_lambda_timeout | Time out of the binaries sync lambda in seconds. | `number` | `300` | no |
159+
| runner_binaries_syncer_lambda_zip | File location of the binaries sync lambda zip file. | `string` | `null` | no |
160+
| runner_extra_labels | Extra labels for the runners (GitHub). Separate each label by a comma | `string` | `""` | no |
161+
| runners_lambda_zip | File location of the lambda zip file for scaling runners. | `string` | `null` | no |
162+
| runners_scale_down_lambda_timeout | Time out for the scale up lambda in seconds. | `number` | `60` | no |
163+
| runners_scale_up_lambda_timeout | Time out for the scale down lambda in seconds. | `number` | `60` | no |
164+
| scale_down_schedule_expression | Scheduler expression to check every x for scale down. | `string` | `"cron(*/5 * * * ? *)"` | no |
165+
| tags | Map of tags that will be added to created resources. By default resources will be tagged with name and environment. | `map(string)` | `{}` | no |
166+
| webhook_lambda_timeout | Time out of the webhook lambda in seconds. | `number` | `10` | no |
167+
| webhook_lambda_zip | File location of the wehbook lambda zip file. | `string` | `null` | no |
168+
169+
## Outputs
170+
171+
| Name | Description |
172+
| --------------- | ----------- |
173+
| binaries_syncer | n/a |
174+
| runners | n/a |
175+
| webhook | n/a |
176+
109177
## Philips Forest
110178

111179
This module is part of the Philips Forest.

‎examples/default/lambdas-download/main.tf

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
module "lambdas" {
2+
source = "../../../modules/download-lambda"
3+
lambdas = [
4+
{
5+
name = "webhook"
6+
tag = "v0.0.0-beta"
7+
},
8+
{
9+
name = "runners"
10+
tag = "v0.0.0-beta"
11+
},
12+
{
13+
name = "runner-binaries-syncer"
14+
tag = "v0.0.0-beta"
15+
}
16+
]
17+
}
18+
19+
output "files" {
20+
value = module.lambdas.files
21+
}

‎examples/default/main.tf

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@ locals {
33
aws_region = "eu-west-1"
44
}
55

6-
76
resource "random_password" "random" {
87
length = 32
98
}
@@ -28,8 +27,11 @@ module "runners" {
2827
webhook_secret = random_password.random.result
2928
}
3029

31-
enable_organization_runners = false
32-
runner_extra_labels = "default,example"
30+
webhook_lambda_zip = "lambdas-download/webhook.zip"
31+
runner_binaries_syncer_lambda_zip = "lambdas-download/runner-binaries-syncer.zip"
32+
runners_lambda_zip = "lambdas-download/runners.zip"
33+
enable_organization_runners = false
34+
runner_extra_labels = "default,example"
3335
}
3436

3537

‎examples/default/outputs.tf

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
output "runners" {
22
value = {
3-
runners = module.runners.runners
3+
lambda_syncer_name = module.runners.binaries_syncer.lambda.function_name
44
}
55
}
66

77
output "webhook" {
88
value = {
9-
secret = random_password.random.result
10-
gateway = module.runners.webhook.gateway
9+
secret = random_password.random.result
10+
endpoint = module.runners.webhook.endpoint
1111
}
1212
}

0 commit comments

Comments
 (0)