Skip to content

Commit 9c563a0

Browse files
authored
Update pull_from_s3.yml
1 parent 63ccf58 commit 9c563a0

File tree

1 file changed

+11
-1
lines changed

1 file changed

+11
-1
lines changed

.github/workflows/pull_from_s3.yml

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,17 @@ jobs:
9595

9696
- name: Download DAGs from S3
9797
if: steps.check_dags.outputs.no_changes != 'true'
98-
run: aws s3 sync s3://airflow-dags-bc/ dags/ --exact-timestamps
98+
# run: aws s3 sync s3://airflow-dags-bc/ dags/ --exact-timestamps
99+
run: |
100+
files_to_download="${{ steps.check_dags.outputs.files_to_download }}"
101+
102+
for file in $files_to_download; do
103+
echo "Downloading $file"
104+
mkdir -p "$(dirname "dags/$file")"
105+
aws s3 cp "s3://airflow-dags-bc/$file" "dags/$file"
106+
done
107+
108+
echo "Downloaded $(echo $files_to_download | wc -w) files"
99109

100110
- name: Commit and push DAG changes
101111
if: steps.check_dags.outputs.no_changes != 'true'

0 commit comments

Comments
 (0)