You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1. TCGPlayer pricing data and TCGDex card data are called and processed through a data pipeline orchestrated by Dagster and hosted on AWS.
14
+
15
+
2. When the pipeline starts, Pydantic validates the incoming API data against a pre-defined schema, ensuring the data types match the expected structure.
16
+
17
+
3. Polars is used to create DataFrames.
18
+
19
+
4. The data is loaded into a Supabase staging schema.
20
+
21
+
5. Soda data quality checks are performed.
22
+
23
+
6.`dbt` runs and builds the final tables in a Supabase production schema.
24
+
25
+
7. Users are then able to query the `pokeapi.co` or supabase APIs for either video game or trading card data, respectively.
0 commit comments