Skip to content
Open
Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
d7812f7
create a dag to handle the daily scheduled nightlight NRT data. The d…
sanzog03 Jun 2, 2025
06a517a
update the nrt update check and nrt data update task
sanzog03 Jun 3, 2025
0426b14
refactor to use the date extracted from metadata, into the collection…
sanzog03 Jun 3, 2025
77d8fd6
replace space with underscore on id
sanzog03 Jun 3, 2025
157f85d
update code into more granular subgroups. also moved nrt update speci…
sanzog03 Jun 4, 2025
5a76e33
removed unnecessary import
sanzog03 Jun 4, 2025
dc8602f
fix proper scoping on the branch_task return, enabling followup task
sanzog03 Jun 5, 2025
9264cc3
refactor dag to use taskflow api completely. added docstring to the t…
sanzog03 Jun 5, 2025
a12768f
add stac_extension to include web map links used by wmts capabilities.
sanzog03 Jun 5, 2025
92b911b
generic def names
sanzog03 Jun 5, 2025
d68f9ea
task group taking whatever is needed via parameters instead of global
sanzog03 Jun 5, 2025
95a7d64
modified dag creation with the thought of re-useability
sanzog03 Jun 5, 2025
ad97e93
generate dag docs based on the reused dag
sanzog03 Jun 5, 2025
fcaba93
add veda_worldview_nrt_data_collection_update_dag_creator to generate…
sanzog03 Jun 5, 2025
358e1e1
changed variables names and dag id generation to be more semantic
sanzog03 Jun 5, 2025
86b258c
change the taskgroupid to be generic
sanzog03 Jun 5, 2025
5026a63
add relevant dag tags
sanzog03 Jun 5, 2025
c1a6d19
fix collection config
sanzog03 Jun 5, 2025
2b6bcf1
added some comments
sanzog03 Nov 13, 2025
73ee5e8
renamed the dags and configs for better clarity
sanzog03 Nov 13, 2025
9646c50
Merge branch 'dev' into GHGC-632/scheduled_worldview_nrt_collection_u…
sanzog03 Nov 13, 2025
0a72f8b
remove unnecessary code
sanzog03 Nov 13, 2025
dd30c6b
take schedule from generate dag config. better namings
sanzog03 Nov 14, 2025
260324f
add ability to ingest wmts without gibs
sanzog03 Nov 14, 2025
82987e9
remove unwanted extensions from wmts config json
sanzog03 Nov 14, 2025
7e09e8b
adhere to the return type
sanzog03 Nov 14, 2025
e5b16e3
single task instantiation for whatever is necessary
sanzog03 Nov 14, 2025
c5208a7
refactor for better task arrangement and readability
sanzog03 Nov 14, 2025
fa67e1b
import error
sanzog03 Nov 14, 2025
94ae24f
1. made gibs url and schedule optional.
sanzog03 Nov 17, 2025
0e16bb1
wmts and gibs branching fix
sanzog03 Nov 17, 2025
d58dc2f
rename to make the dag generic for all wmts and not just gibs wmts
sanzog03 Nov 17, 2025
6cf67d9
update doc string
sanzog03 Nov 17, 2025
2f0ffd9
1. add validation task which validates if the json config is as per t…
sanzog03 Nov 17, 2025
087b486
remove comment
sanzog03 Nov 17, 2025
f1fb195
corrected import for tests
sanzog03 Nov 17, 2025
c33b58c
update wmts with validator
kyle-lesinger Nov 25, 2025
a1f3d5a
update pytests
kyle-lesinger Nov 25, 2025
ed6d435
add slack fail alert
kyle-lesinger Nov 25, 2025
0d61a5f
add value error messages
kyle-lesinger Nov 25, 2025
b2249b9
reduce number of logging statements
kyle-lesinger Dec 1, 2025
4a240f1
update generate dags
kyle-lesinger Dec 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion dags/generate_dags.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,22 @@

from veda_data_pipeline.veda_discover_pipeline import get_discover_dag
from veda_data_pipeline.veda_vector_pipeline import get_ingest_vector_dag
from veda_data_pipeline.veda_wmts2stac_update_pipeline import get_ingest_wmts2stac_dag
from veda_data_pipeline.veda_pyarc2stac_pipeline import get_ingest_pyarc2stac_dag

dag_generators = {
"veda_discover": get_discover_dag,
"veda_ingest_vector": get_ingest_vector_dag,
"veda_pyarc2stac_ingest": get_ingest_pyarc2stac_dag,
"veda_wmts2stac_ingest": get_ingest_wmts2stac_dag
}

# preserve DAG history
dag_names = {
"veda_discover": "discover",
"veda_ingest_vector": "vector",
"veda_pyarc2stac_ingest": "pyarc2stac",
"veda_wmts2stac_ingest": "wmts2stac"
}

def generate_dags():
Expand Down Expand Up @@ -64,7 +67,6 @@ def generate_dags():
file_name = id # use id for DAG name if provided, otherwise default to file name
dag_generators[dag](id=f"{dag_names[dag]}-{file_name}", event=c)


generate_dags()
# create default DAGs (no config or schedule)
get_ingest_vector_dag(id="veda_ingest_vector", event={})
Expand Down
Empty file.
Loading