Skip to content

Commit 376664c

Browse files
buenos-nachosbuenos-nachos
authored
chore: migrate all Coder modules to Registry repo (#4)
Addresses part of coder/internal#532 (but doesn't fully close it out). This is a huge PR, but chunking it up seemed pointless, since we're largely copying over existing files. I'm going to try commenting on the main areas I think are worth paying attention to ## Changes made - Migrated over all Coder modules from coder/modules, and put them in their correct user namespaces. - Added README.md files to all newly-created user namespaces - Updated all image paths for every image used, to make sure they don't break (I switched the paths programmatically, and then manually verified every README to guarantee this). - Added README.md files for each contributor who previously made a module - Updated our `tsconfig.json` file to use modern libraries when run from the server, and made a custom `tsconfig.json` just for the `windows-rdp` module (which has more restrictive browser concerns) - Added CI step to run all the module tests we currently have ## Notes - There were a lot of Bash script files that weren't carried over in this PR, partly because I don't know Bash well enough to know (1) whether they're still needed, or (2) modify them to account for the new file structure. Those can be brought over later. - We had a `lint.ts` file that provided some light validation of some of the modules. After going through it, there were so many bugs and issues with the code that I legitimately think that it barely provided a safety net at all. I got rid of it entirely, with the intention of adding the functionality that was originally intended to the current validation logic (to be handled in a separate PR). - I changed how we set up the `.images` directory, because it felt like it would be chaos if a bunch of users try to throw all their images in one giant directory, with no guidelines on how to do it. I instead made it so that images should be scoped by namespace, which felt a lot more manageable. The `.icons` directory is still at the top level, because realistically, there are only going to be so many types of icons referenced, so it's fine for those to be shared. - I don't think the `maintainer_github` and `contributor_github` fields make sense anymore, but those can be stripped out once we've updated the Registry site build step to use the new Registry repo - My gut instinct is to use the user namespace to determine the main owner of the module, and then add a `contributors` string list to indicate which other users have contributed meaningfully to it. We can then add validation to make sure that every value in that list exists as another namespace in the repo ## What still needs to be migrated (in separate PRs) These are the main files of interest that still probably need to be copied over from the `/modules` repo: - `new.sh` - `terraform_validate.sh` - `update-version.sh` They're probably going to require enough changes that it's worth handling them in a separate PR.
1 parent 7fd409f commit 376664c

File tree

200 files changed

+13132
-1
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

200 files changed

+13132
-1
lines changed

‎.github/dependabot.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
version: 2
2+
updates:
3+
- package-ecosystem: "github-actions"
4+
directory: "/"
5+
schedule:
6+
interval: "weekly"

‎.github/scripts/check_registry_site_health.sh‎

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
#!/usr/bin/env bash
2+
set -o pipefail
3+
set -u
4+
5+
VERBOSE="${VERBOSE:-0}"
6+
if [[ "${VERBOSE}" -ne "0" ]]; then
7+
set -x
8+
fi
9+
10+
# List of required environment variables
11+
required_vars=(
12+
"INSTATUS_API_KEY"
13+
"INSTATUS_PAGE_ID"
14+
"INSTATUS_COMPONENT_ID"
15+
"VERCEL_API_KEY"
16+
)
17+
18+
# Check if each required variable is set
19+
for var in "${required_vars[@]}"; do
20+
if [[ -z "${!var:-}" ]]; then
21+
echo "Error: Environment variable '$var' is not set."
22+
exit 1
23+
fi
24+
done
25+
26+
REGISTRY_BASE_URL="${REGISTRY_BASE_URL:-https://registry.coder.com}"
27+
28+
status=0
29+
declare -a modules=()
30+
declare -a failures=()
31+
32+
# Collect all module directories containing a main.tf file
33+
for path in $(find . -maxdepth 2 -not -path '*/.*' -type f -name main.tf | cut -d '/' -f 2 | sort -u); do
34+
modules+=("${path}")
35+
done
36+
37+
echo "Checking modules: ${modules[*]}"
38+
39+
# Function to update the component status on Instatus
40+
update_component_status() {
41+
local component_status=$1
42+
# see https://instatus.com/help/api/components
43+
(curl -X PUT "https://api.instatus.com/v1/$INSTATUS_PAGE_ID/components/$INSTATUS_COMPONENT_ID" \
44+
-H "Authorization: Bearer $INSTATUS_API_KEY" \
45+
-H "Content-Type: application/json" \
46+
-d "{\"status\": \"$component_status\"}")
47+
}
48+
49+
# Function to create an incident
50+
create_incident() {
51+
local incident_name="Degraded Service"
52+
local message="The following modules are experiencing issues:\n"
53+
for i in "${!failures[@]}"; do
54+
message+="$((i + 1)). ${failures[$i]}\n"
55+
done
56+
57+
component_status="PARTIALOUTAGE"
58+
if ((${#failures[@]} == ${#modules[@]})); then
59+
component_status="MAJOROUTAGE"
60+
fi
61+
# see https://instatus.com/help/api/incidents
62+
incident_id=$(curl -s -X POST "https://api.instatus.com/v1/$INSTATUS_PAGE_ID/incidents" \
63+
-H "Authorization: Bearer $INSTATUS_API_KEY" \
64+
-H "Content-Type: application/json" \
65+
-d "{
66+
\"name\": \"$incident_name\",
67+
\"message\": \"$message\",
68+
\"components\": [\"$INSTATUS_COMPONENT_ID\"],
69+
\"status\": \"INVESTIGATING\",
70+
\"notify\": true,
71+
\"statuses\": [
72+
{
73+
\"id\": \"$INSTATUS_COMPONENT_ID\",
74+
\"status\": \"PARTIALOUTAGE\"
75+
}
76+
]
77+
}" | jq -r '.id')
78+
79+
echo "Created incident with ID: $incident_id"
80+
}
81+
82+
# Function to check for existing unresolved incidents
83+
check_existing_incident() {
84+
# Fetch the latest incidents with status not equal to "RESOLVED"
85+
local unresolved_incidents=$(curl -s -X GET "https://api.instatus.com/v1/$INSTATUS_PAGE_ID/incidents" \
86+
-H "Authorization: Bearer $INSTATUS_API_KEY" \
87+
-H "Content-Type: application/json" | jq -r '.incidents[] | select(.status != "RESOLVED") | .id')
88+
89+
if [[ -n "$unresolved_incidents" ]]; then
90+
echo "Unresolved incidents found: $unresolved_incidents"
91+
return 0 # Indicate that there are unresolved incidents
92+
else
93+
echo "No unresolved incidents found."
94+
return 1 # Indicate that no unresolved incidents exist
95+
fi
96+
}
97+
98+
force_redeploy_registry() {
99+
# These are not secret values; safe to just expose directly in script
100+
local VERCEL_TEAM_SLUG="codercom"
101+
local VERCEL_TEAM_ID="team_tGkWfhEGGelkkqUUm9nXq17r"
102+
local VERCEL_APP="registry"
103+
104+
local latest_res
105+
latest_res=$(
106+
curl "https://api.vercel.com/v6/deployments?app=$VERCEL_APP&limit=1&slug=$VERCEL_TEAM_SLUG&teamId=$VERCEL_TEAM_ID&target=production&state=BUILDING,INITIALIZING,QUEUED,READY" \
107+
--fail \
108+
--silent \
109+
--header "Authorization: Bearer $VERCEL_API_KEY" \
110+
--header "Content-Type: application/json"
111+
)
112+
113+
# If we have zero deployments, something is VERY wrong. Make the whole
114+
# script exit with a non-zero status code
115+
local latest_id
116+
latest_id=$(echo "${latest_res}" | jq -r '.deployments[0].uid')
117+
if [[ "${latest_id}" = "null" ]]; then
118+
echo "Unable to pull any previous deployments for redeployment"
119+
echo "Please redeploy the latest deployment manually in Vercel."
120+
echo "https://vercel.com/codercom/registry/deployments"
121+
exit 1
122+
fi
123+
124+
local latest_date_ts_seconds
125+
latest_date_ts_seconds=$(echo "${latest_res}" | jq -r '.deployments[0].createdAt/1000|floor')
126+
local current_date_ts_seconds
127+
current_date_ts_seconds="$(date +%s)"
128+
local max_redeploy_interval_seconds=7200 # 2 hours
129+
if ((current_date_ts_seconds - latest_date_ts_seconds < max_redeploy_interval_seconds)); then
130+
echo "The registry was deployed less than 2 hours ago."
131+
echo "Not automatically re-deploying the regitstry."
132+
echo "A human reading this message should decide if a redeployment is necessary."
133+
echo "Please check the Vercel dashboard for more information."
134+
echo "https://vercel.com/codercom/registry/deployments"
135+
exit 1
136+
fi
137+
138+
local latest_deployment_state
139+
latest_deployment_state="$(echo "${latest_res}" | jq -r '.deployments[0].state')"
140+
if [[ "${latest_deployment_state}" != "READY" ]]; then
141+
echo "Last deployment was not in READY state. Skipping redeployment."
142+
echo "A human reading this message should decide if a redeployment is necessary."
143+
echo "Please check the Vercel dashboard for more information."
144+
echo "https://vercel.com/codercom/registry/deployments"
145+
exit 1
146+
fi
147+
148+
echo "============================================================="
149+
echo "!!! Redeploying registry with deployment ID: ${latest_id} !!!"
150+
echo "============================================================="
151+
152+
if ! curl -X POST "https://api.vercel.com/v13/deployments?forceNew=1&skipAutoDetectionConfirmation=1&slug=$VERCEL_TEAM_SLUG&teamId=$VERCEL_TEAM_ID" \
153+
--fail \
154+
--header "Authorization: Bearer $VERCEL_API_KEY" \
155+
--header "Content-Type: application/json" \
156+
--data-raw "{ \"deploymentId\": \"${latest_id}\", \"name\": \"${VERCEL_APP}\", \"target\": \"production\" }"; then
157+
echo "DEPLOYMENT FAILED! Please check the Vercel dashboard for more information."
158+
echo "https://vercel.com/codercom/registry/deployments"
159+
exit 1
160+
fi
161+
}
162+
163+
# Check each module's accessibility
164+
for module in "${modules[@]}"; do
165+
# Trim leading/trailing whitespace from module name
166+
module=$(echo "${module}" | xargs)
167+
url="${REGISTRY_BASE_URL}/modules/${module}"
168+
printf "=== Checking module %s at %s\n" "${module}" "${url}"
169+
status_code=$(curl --output /dev/null --head --silent --fail --location "${url}" --retry 3 --write-out "%{http_code}")
170+
if ((status_code != 200)); then
171+
printf "==> FAIL(%s)\n" "${status_code}"
172+
status=1
173+
failures+=("${module}")
174+
else
175+
printf "==> OK(%s)\n" "${status_code}"
176+
fi
177+
done
178+
179+
# Determine overall status and update Instatus component
180+
if ((status == 0)); then
181+
echo "All modules are operational."
182+
# set to
183+
update_component_status "OPERATIONAL"
184+
else
185+
echo "The following modules have issues: ${failures[*]}"
186+
# check if all modules are down
187+
if ((${#failures[@]} == ${#modules[@]})); then
188+
update_component_status "MAJOROUTAGE"
189+
else
190+
update_component_status "PARTIALOUTAGE"
191+
fi
192+
193+
# Check if there is an existing incident before creating a new one
194+
if ! check_existing_incident; then
195+
create_incident
196+
fi
197+
198+
# If a module is down, force a reployment to try getting things back online
199+
# ASAP
200+
# EDIT: registry.coder.com is no longer hosted on vercel
201+
#force_redeploy_registry
202+
fi
203+
204+
exit "${status}"

‎.github/workflows/check_registry_site_health.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Check modules health on registry.coder.com
2+
name: check-registry-site-health
3+
on:
4+
schedule:
5+
- cron: "0,15,30,45 * * * *" # Runs every 15 minutes
6+
workflow_dispatch: # Allows manual triggering of the workflow if needed
7+
8+
jobs:
9+
run-script:
10+
runs-on: ubuntu-latest
11+
12+
steps:
13+
- name: Checkout repository
14+
uses: actions/checkout@v4
15+
16+
- name: Run check.sh
17+
run: |
18+
./.github/scripts/check.sh
19+
env:
20+
INSTATUS_API_KEY: ${{ secrets.INSTATUS_API_KEY }}
21+
INSTATUS_PAGE_ID: ${{ secrets.INSTATUS_PAGE_ID }}
22+
INSTATUS_COMPONENT_ID: ${{ secrets.INSTATUS_COMPONENT_ID }}
23+
VERCEL_API_KEY: ${{ secrets.VERCEL_API_KEY }}

‎.github/workflows/ci.yaml‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ concurrency:
77
group: ${{ github.workflow }}-${{ github.ref }}
88
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
99
jobs:
10-
validate-contributors:
10+
validate-readme-files:
1111
runs-on: ubuntu-latest
1212
steps:
1313
- name: Check out code
@@ -20,3 +20,23 @@ jobs:
2020
run: go build ./scripts/contributors && ./contributors
2121
- name: Remove build file artifact
2222
run: rm ./contributors
23+
test-terraform:
24+
runs-on: ubuntu-latest
25+
steps:
26+
- name: Check out code
27+
uses: actions/checkout@v4
28+
- name: Set up Terraform
29+
uses: coder/coder/.github/actions/setup-tf@main
30+
- name: Set up Bun
31+
uses: oven-sh/setup-bun@v2
32+
with:
33+
# We're using the latest version of Bun for now, but it might be worth
34+
# reconsidering. They've pushed breaking changes in patch releases
35+
# that have broken our CI.
36+
# Our PR where issues started to pop up: https://github.com/coder/modules/pull/383
37+
# The Bun PR that broke things: https://github.com/oven-sh/bun/pull/16067
38+
bun-version: latest
39+
- name: Install dependencies
40+
run: bun install
41+
- name: Run tests
42+
run: bun test

‎.github/workflows/deploy_registry.yaml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
name: deploy-registry
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
8+
jobs:
9+
deploy:
10+
runs-on: ubuntu-latest
11+
12+
# Set id-token permission for gcloud
13+
# Adding a comment because retriggering the build manually hung? I am the lord of devops and you will bend?
14+
permissions:
15+
contents: read
16+
id-token: write
17+
18+
steps:
19+
- name: Checkout repository
20+
uses: actions/checkout@v4
21+
22+
- name: Authenticate to Google Cloud
23+
uses: google-github-actions/auth@71f986410dfbc7added4569d411d040a91dc6935
24+
with:
25+
workload_identity_provider: projects/309789351055/locations/global/workloadIdentityPools/github-actions/providers/github
26+
service_account: [email protected]
27+
28+
- name: Set up Google Cloud SDK
29+
uses: google-github-actions/setup-gcloud@77e7a554d41e2ee56fc945c52dfd3f33d12def9a
30+
31+
# For the time being, let's have the first couple merges to main in modules deploy a new version
32+
# to *dev*. Once we review and make sure everything's working, we can deploy a new version to *main*.
33+
# Maybe in the future we could automate this based on the result of E2E tests.
34+
- name: Deploy to dev.registry.coder.com
35+
run: |
36+
gcloud builds triggers run 29818181-126d-4f8a-a937-f228b27d3d34 --branch dev

‎.gitignore‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,3 +137,11 @@ dist
137137

138138
# Script output
139139
/contributors
140+
141+
# Terraform files generated during testing
142+
.terraform*
143+
*.tfstate
144+
*.tfstate.lock.info
145+
146+
# Generated credentials from google-github-actions/auth
147+
gha-creds-*.json

0 commit comments

Comments
 (0)