Skip to content

Commit 300762d

Browse files
girumbgirumb
committed
Merge branch 'main' into HEA-752/Dagster-GraphQL-API-is-intermittently-failing-with-a-ProtocolError-when-accessed-via-the-revproxy-Django-view
2 parents 2d11445 + 15bf8bb commit 300762d

File tree

6 files changed

+90
-95
lines changed

6 files changed

+90
-95
lines changed

‎.github/workflows/01-build-then-test.yml‎

Lines changed: 41 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ env:
2929

3030
jobs:
3131
lint:
32-
runs-on: [ data ]
32+
runs-on: [ fewsnet ]
3333
container: "python:3.11"
3434

3535
steps:
@@ -51,7 +51,7 @@ jobs:
5151
start-runner-test_branch:
5252
if: ${{ !(startsWith(github.ref, 'refs/tags/')) && !(github.ref == 'refs/heads/main') }}
5353
needs: lint
54-
runs-on: [ data ]
54+
runs-on: [ fewsnet ]
5555
outputs:
5656
label: ${{ steps.start-ec2-runner.outputs.label }}
5757
ec2-instance-id: ${{ steps.start-ec2-runner.outputs.ec2-instance-id }}
@@ -62,24 +62,14 @@ jobs:
6262
aws-region: ${{ vars.AWS_REGION }}
6363
- name: Start EC2 runner test_branch
6464
id: start-ec2-runner
65-
uses: FEWS-NET/ec2-github-runner@1f687d4b786b21bed4ad6f420daddc7665bff857
65+
uses: American-Institutes-for-Research/gitops-actions/fewsnet-start-runner@main
6666
with:
67-
mode: start
68-
github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
69-
ec2-image-id: ami-092d88ab9dba1ddc0
70-
ec2-instance-type: m5a.large
71-
subnet-id: subnet-0f50e5fa11397a87b
72-
security-group-id: sg-0c844cef0ffc13cde
73-
iam-role-name: data-prod-github-runner-role # optional, requires additional permissions
74-
aws-resource-tags: > # optional, requires additional permissions
75-
[
76-
{"Key": "Name", "Value": "HEA-github-autoscale-runner"},
77-
{"Key": "GitHubRepository", "Value": "${{ github.repository }}"}
78-
]
67+
gh_personal_access_token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
68+
project-key: "HEA"
7969

8070
start-runner-test_keepdb:
8171
needs: lint
82-
runs-on: [ data ]
72+
runs-on: [ fewsnet ]
8373
outputs:
8474
label: ${{ steps.start-ec2-runner.outputs.label }}
8575
ec2-instance-id: ${{ steps.start-ec2-runner.outputs.ec2-instance-id }}
@@ -90,25 +80,15 @@ jobs:
9080
aws-region: ${{ env.AWS_REGION }}
9181
- name: Start EC2 runner test_keepdb
9282
id: start-ec2-runner
93-
uses: FEWS-NET/ec2-github-runner@1f687d4b786b21bed4ad6f420daddc7665bff857
83+
uses: American-Institutes-for-Research/gitops-actions/fewsnet-start-runner@main
9484
with:
95-
mode: start
96-
github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
97-
ec2-image-id: ami-092d88ab9dba1ddc0
98-
ec2-instance-type: m5a.large
99-
subnet-id: subnet-0f50e5fa11397a87b
100-
security-group-id: sg-0c844cef0ffc13cde
101-
iam-role-name: data-prod-github-runner-role # optional, requires additional permissions
102-
aws-resource-tags: > # optional, requires additional permissions
103-
[
104-
{"Key": "Name", "Value": "HEA-github-autoscale-runner"},
105-
{"Key": "GitHubRepository", "Value": "${{ github.repository }}"}
106-
]
85+
gh_personal_access_token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
86+
project-key: "HEA"
10787

10888
start-runner-test_main:
10989
if: github.ref == 'refs/heads/main'
11090
needs: lint
111-
runs-on: [ data ]
91+
runs-on: [ fewsnet ]
11292
outputs:
11393
label: ${{ steps.start-ec2-runner.outputs.label }}
11494
ec2-instance-id: ${{ steps.start-ec2-runner.outputs.ec2-instance-id }}
@@ -119,25 +99,15 @@ jobs:
11999
aws-region: ${{ vars.AWS_REGION }}
120100
- name: Start EC2 runner test_main
121101
id: start-ec2-runner
122-
uses: FEWS-NET/ec2-github-runner@1f687d4b786b21bed4ad6f420daddc7665bff857
102+
uses: American-Institutes-for-Research/gitops-actions/fewsnet-start-runner@main
123103
with:
124-
mode: start
125-
github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
126-
ec2-image-id: ami-092d88ab9dba1ddc0
127-
ec2-instance-type: m5a.large
128-
subnet-id: subnet-0f50e5fa11397a87b
129-
security-group-id: sg-0c844cef0ffc13cde
130-
iam-role-name: data-prod-github-runner-role # optional, requires additional permissions
131-
aws-resource-tags: > # optional, requires additional permissions
132-
[
133-
{"Key": "Name", "Value": "HEA-github-autoscale-runner"},
134-
{"Key": "GitHubRepository", "Value": "${{ github.repository }}"}
135-
]
104+
gh_personal_access_token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
105+
project-key: "HEA"
136106

137107
start-runner-test_tag:
138108
if: startsWith(github.ref, 'refs/tags/')
139109
needs: lint
140-
runs-on: [ data ]
110+
runs-on: [ fewsnet ]
141111
outputs:
142112
label: ${{ steps.start-ec2-runner.outputs.label }}
143113
ec2-instance-id: ${{ steps.start-ec2-runner.outputs.ec2-instance-id }}
@@ -148,20 +118,10 @@ jobs:
148118
aws-region: ${{ env.AWS_REGION }}
149119
- name: Start EC2 runner test_tag
150120
id: start-ec2-runner
151-
uses: FEWS-NET/ec2-github-runner@1f687d4b786b21bed4ad6f420daddc7665bff857
121+
uses: American-Institutes-for-Research/gitops-actions/fewsnet-start-runner@main
152122
with:
153-
mode: start
154-
github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
155-
ec2-image-id: ami-092d88ab9dba1ddc0
156-
ec2-instance-type: m5a.large
157-
subnet-id: subnet-0f50e5fa11397a87b
158-
security-group-id: sg-0c844cef0ffc13cde
159-
iam-role-name: data-prod-github-runner-role # optional, requires additional permissions
160-
aws-resource-tags: > # optional, requires additional permissions
161-
[
162-
{"Key": "Name", "Value": "HEA-github-autoscale-runner"},
163-
{"Key": "GitHubRepository", "Value": "${{ github.repository }}"}
164-
]
123+
gh_personal_access_token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
124+
project-key: "HEA"
165125

166126
test_branch:
167127
if: ${{ !(startsWith(github.ref, 'refs/tags/')) && !(github.ref == 'refs/heads/main') }}
@@ -173,10 +133,10 @@ jobs:
173133
needs:
174134
- start-runner-test_branch
175135
container:
176-
image: ghcr.io/fews-net/inf/docker:latest
136+
image: airorg/fewsnet-inf:docker-latest
177137
credentials:
178-
username: ${{ github.repository_owner }}
179-
password: ${{ secrets.GHCR_PAT }}
138+
username: ${{ secrets.DOCKER_HUB_USERNAME }}
139+
password: ${{ secrets.DOCKER_HUB_TOKEN }}
180140
defaults:
181141
run:
182142
shell: ash --noprofile --norc -eo pipefail {0} # Fail on any non-zero exit code, even in piped commands
@@ -207,7 +167,7 @@ jobs:
207167
uses: docker/login-action@v3
208168
with:
209169
username: ${{ secrets.DOCKER_HUB_USERNAME }}
210-
password: ${{ secrets.DOCKER_HUB_PASSWORD }}
170+
password: ${{ secrets.DOCKER_HUB_TOKEN }}
211171
- name: "Pull previous images to speed up builds"
212172
run: |
213173
for SERVICE in ${BUILD_IMAGES}; do (docker pull ${CI_REGISTRY_IMAGE}/${SERVICE}:edge | grep -i -e 'Pulling from' -e Digest -e Status -e Error) || true; done
@@ -316,10 +276,10 @@ jobs:
316276
needs:
317277
- start-runner-test_keepdb
318278
container:
319-
image: ghcr.io/fews-net/inf/docker:latest
279+
image: airorg/fewsnet-inf:docker-latest
320280
credentials:
321-
username: ${{ github.repository_owner }}
322-
password: ${{ secrets.GHCR_PAT }}
281+
username: ${{ secrets.DOCKER_HUB_USERNAME }}
282+
password: ${{ secrets.DOCKER_HUB_TOKEN }}
323283
defaults:
324284
run:
325285
shell: ash --noprofile --norc -eo pipefail {0} # Fail on any non-zero exit code, even in piped commands
@@ -350,7 +310,7 @@ jobs:
350310
uses: docker/login-action@v3
351311
with:
352312
username: ${{ secrets.DOCKER_HUB_USERNAME }}
353-
password: ${{ secrets.DOCKER_HUB_PASSWORD }}
313+
password: ${{ secrets.DOCKER_HUB_TOKEN }}
354314
- name: "Pull previous images to speed up builds"
355315
run: |
356316
for SERVICE in ${BUILD_IMAGES}; do (docker pull ${CI_REGISTRY_IMAGE}/${SERVICE}:edge | grep -i -e 'Pulling from' -e Digest -e Status -e Error) || true; done
@@ -441,10 +401,10 @@ jobs:
441401
needs:
442402
- start-runner-test_main
443403
container:
444-
image: ghcr.io/fews-net/inf/docker:latest
404+
image: airorg/fewsnet-inf:docker-latest
445405
credentials:
446-
username: ${{ github.repository_owner }}
447-
password: ${{ secrets.GHCR_PAT }}
406+
username: ${{ secrets.DOCKER_HUB_USERNAME }}
407+
password: ${{ secrets.DOCKER_HUB_TOKEN }}
448408
defaults:
449409
run:
450410
shell: ash --noprofile --norc -eo pipefail {0} # Fail on any non-zero exit code, even in piped commands
@@ -475,7 +435,7 @@ jobs:
475435
uses: docker/login-action@v3
476436
with:
477437
username: ${{ secrets.DOCKER_HUB_USERNAME }}
478-
password: ${{ secrets.DOCKER_HUB_PASSWORD }}
438+
password: ${{ secrets.DOCKER_HUB_TOKEN }}
479439
- name: "Pull previous images to speed up builds"
480440
run: |
481441
echo Using registry image ${CI_REGISTRY_IMAGE}
@@ -569,10 +529,10 @@ jobs:
569529
needs:
570530
- start-runner-test_tag
571531
container:
572-
image: ghcr.io/fews-net/inf/docker:latest
532+
image: airorg/fewsnet-inf:docker-latest
573533
credentials:
574-
username: ${{ github.repository_owner }}
575-
password: ${{ secrets.GHCR_PAT }}
534+
username: ${{ secrets.DOCKER_HUB_USERNAME }}
535+
password: ${{ secrets.DOCKER_HUB_TOKEN }}
576536
defaults:
577537
run:
578538
shell: ash --noprofile --norc -eo pipefail {0} # Fail on any non-zero exit code, even in piped commands
@@ -603,7 +563,7 @@ jobs:
603563
uses: docker/login-action@v3
604564
with:
605565
username: ${{ secrets.DOCKER_HUB_USERNAME }}
606-
password: ${{ secrets.DOCKER_HUB_PASSWORD }}
566+
password: ${{ secrets.DOCKER_HUB_TOKEN }}
607567
- uses: "actions/checkout@v4"
608568
with:
609569
ssh-key: ${{ secrets.GIT_SSH_PRIVATE_KEY }}
@@ -694,15 +654,15 @@ jobs:
694654
needs:
695655
- start-runner-test_branch # required to get output from the start-runner-test_branch job
696656
- test_branch # required to wait until the main jobs are done
697-
runs-on: [ data ]
657+
runs-on: [ fewsnet ]
698658

699659
steps:
700660
- name: Configure AWS credentials
701661
uses: aws-actions/configure-aws-credentials@v4
702662
with:
703663
aws-region: ${{ env.AWS_REGION }}
704664
- name: Stop EC2 runner
705-
uses: FEWS-NET/ec2-github-runner@1f687d4b786b21bed4ad6f420daddc7665bff857
665+
uses: American-Institutes-for-Research/ec2-github-runner@598310b0225c539534b6ef47575f36404dedf3b1
706666
with:
707667
mode: stop
708668
github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
@@ -714,15 +674,15 @@ jobs:
714674
needs:
715675
- start-runner-test_keepdb # required to get output from the start-runner-test_keepdb job
716676
- test_keepdb
717-
runs-on: [ data ]
677+
runs-on: [ fewsnet ]
718678

719679
steps:
720680
- name: Configure AWS credentials
721681
uses: aws-actions/configure-aws-credentials@v4
722682
with:
723683
aws-region: ${{ env.AWS_REGION }}
724684
- name: Stop EC2 runner
725-
uses: FEWS-NET/ec2-github-runner@1f687d4b786b21bed4ad6f420daddc7665bff857
685+
uses: American-Institutes-for-Research/ec2-github-runner@598310b0225c539534b6ef47575f36404dedf3b1
726686
with:
727687
mode: stop
728688
github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
@@ -734,15 +694,15 @@ jobs:
734694
needs:
735695
- start-runner-test_main # required to get output from the start-runner-test_main job
736696
- test_main
737-
runs-on: [ data ]
697+
runs-on: [ fewsnet ]
738698

739699
steps:
740700
- name: Configure AWS credentials
741701
uses: aws-actions/configure-aws-credentials@v4
742702
with:
743703
aws-region: ${{ env.AWS_REGION }}
744704
- name: Stop EC2 runner
745-
uses: FEWS-NET/ec2-github-runner@1f687d4b786b21bed4ad6f420daddc7665bff857
705+
uses: American-Institutes-for-Research/ec2-github-runner@598310b0225c539534b6ef47575f36404dedf3b1
746706
with:
747707
mode: stop
748708
github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
@@ -754,15 +714,15 @@ jobs:
754714
needs:
755715
- start-runner-test_tag # required to get output from the start-runner-test_tag job
756716
- test_tag
757-
runs-on: [ data ]
717+
runs-on: [ fewsnet ]
758718

759719
steps:
760720
- name: Configure AWS credentials
761721
uses: aws-actions/configure-aws-credentials@v4
762722
with:
763723
aws-region: ${{ env.AWS_REGION }}
764724
- name: Stop EC2 runner
765-
uses: FEWS-NET/ec2-github-runner@1f687d4b786b21bed4ad6f420daddc7665bff857
725+
uses: American-Institutes-for-Research/ec2-github-runner@598310b0225c539534b6ef47575f36404dedf3b1
766726
with:
767727
mode: stop
768728
github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}

‎.github/workflows/02-deploy-env.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ jobs:
2424
deploy_env:
2525
# this will allow us to access secrets defined for this environment
2626
environment: ${{ inputs.environment }}
27-
runs-on: [ data ]
27+
runs-on: [ fewsnet ]
2828
container:
2929
image: alpine:3.17
3030
defaults:

‎.github/workflows/03-mirror-to-fdw.yml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
name: Mirror to FDW
2+
3+
permissions:
4+
contents: read
5+
6+
on:
7+
push:
8+
branches:
9+
- "main"
10+
tags:
11+
- "*"
12+
13+
jobs:
14+
mirror:
15+
runs-on: [ fewsnet ]
16+
# run in container to avoid file permission issues like EACCES: permission denied
17+
container: "python:3.12"
18+
steps:
19+
- uses: "actions/checkout@v4"
20+
with:
21+
# we don't want the current commit only,
22+
# but also previous commits in case those haven't been synced
23+
fetch-depth: 0
24+
25+
# since we are running this on the runner controller (like lint), action/checkout
26+
# will not create a new repo for subsequent runs. This lead to
27+
# error: remote mirror already exists.
28+
# We will just remove the remote here if it already exists, so mirror-action works
29+
- name: Remove existing remote
30+
run: |
31+
# seems to be necessary to run git commands
32+
git config --global --add safe.directory /__w/FEWSNET-HEA-Database-Development/FEWSNET-HEA-Database-Development
33+
git remote remove mirror || echo "No remote 'mirror' existed"
34+
35+
- uses: yesolutions/[email protected]
36+
with:
37+
REMOTE: "ssh://[email protected]/FEWS-NET/HEA-Database-Development.git"
38+
GIT_SSH_PRIVATE_KEY: ${{ secrets.FDW_GIT_SSH_PRIVATE_KEY }}
39+
# @TODO: change this to secrets.GIT_SSH_KNOWN_HOSTS
40+
GIT_SSH_NO_VERIFY_HOST: "true"
41+
PUSH_ALL_REFS: "false"

‎pipelines/assets/fixtures.py‎

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -382,12 +382,7 @@ def consolidated_fixture(
382382
consolidated_instances[model_name] = []
383383
consolidated_instances[model_name] += instances
384384

385-
fixture, metadata = get_fixture_from_instances(consolidated_instances)
386-
387-
return Output(
388-
fixture,
389-
metadata=metadata,
390-
)
385+
return get_fixture_from_instances(consolidated_instances)
391386

392387

393388
@asset(partitions_def=bss_files_partitions_def)
@@ -403,7 +398,8 @@ def uploaded_baselines(
403398
Downstream assets apply corrections to the original file and then process
404399
the contents to create Communities, Wealth Groups, Livelihood Strategies, etc.
405400
"""
406-
fixture, metadata = get_fixture_from_instances(baseline_instances)
401+
output = get_fixture_from_instances(baseline_instances)
402+
fixture = output.value
407403
metadata = import_fixture(fixture)
408404

409405
# Add the file objects `bss` and `profile_report` FileFields to the model instances
@@ -416,7 +412,7 @@ def uploaded_baselines(
416412

417413
return Output(
418414
None,
419-
metadata=metadata,
415+
metadata=metadata.metadata,
420416
)
421417

422418

‎pipelines/assets/livelihood_activity.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1337,6 +1337,8 @@ def get_annotated_instances_from_dataframe(
13371337
# Annotate the output metadata with completeness information
13381338
# Get the summary dataframe, grouped by strategy_type
13391339
summary_df = pd.DataFrame(reported_summary_output.value["LivelihoodActivity"])
1340+
for col in ["income", "expenditure", "kcals_consumed"]:
1341+
summary_df[col] = pd.to_numeric(summary_df[col], errors="coerce").fillna(0)
13401342
summary_df = (
13411343
summary_df[["strategy_type", "income", "expenditure", "kcals_consumed"]].groupby("strategy_type").sum()
13421344
)

0 commit comments

Comments
 (0)