You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For the full `GraphQLResourceConfig` spec, see the [`source`](https://github.com/opensource-observer/oso/blob/05fe8b9192a08f6446225a89f4455c6b3723c5de/warehouse/oso_dagster/factories/graphql.py#L99)
75
+
For the full `GraphQLResourceConfig` spec, see the [`source`](https://github.com/opensource-observer/oso/blob/main/warehouse/oso_dagster/factories/graphql.py#L151)
75
76
:::
76
77
77
78
In this configuration, we define the following fields:
78
79
79
80
-**name**: A unique identifier for the dagster asset.
80
81
-**endpoint**: The URL of the GraphQL API.
82
+
-**target_type**: The GraphQL type containing the target query (usually
83
+
"Query").
84
+
-**target_query**: The name of the query to execute.
81
85
-**max_depth**: The maximum depth of the introspection query. This will
82
86
generate a query that explores all fields recursively up to this depth.
83
-
-**pagination**: A configuration object that defines how to handle pagination.
84
-
It includes the pagination type, page size, maximum number of pages to
85
-
fetch, rate limit in seconds, and the fields used for offset and limit.
86
87
-**parameters**: A dictionary of query parameters. The keys are the parameter
87
88
names, and the values are dictionaries with the parameter type and value.
89
+
-**pagination**: A configuration object that defines how to handle pagination.
90
+
It includes the pagination type, page size, maximum number of pages to fetch,
91
+
rate limit in seconds, and the fields used for offset and limit.
92
+
-**exclude**: A list of field names to exclude from the GraphQL schema
93
+
expansion.
88
94
-**transform_fn**: A function that processes the raw GraphQL response and
89
95
returns the desired data.
90
-
-**target_query**: The name of the query to execute.
91
-
-**target_type**: The name of the GraphQL type that contains the data of
92
-
interest.
93
-
94
-
The factory will create the following query automatically, recursively
95
-
introspecting all the fields up to the specified depth:
96
-
97
-
```graphql
98
-
query (
99
-
$type: TransactionType!
100
-
$dateFrom: DateTime!
101
-
$dateTo: DateTime!
102
-
$offset: Int
103
-
$limit: Int!
104
-
) {
105
-
transactions(
106
-
type: $type
107
-
dateFrom: $dateFrom
108
-
dateTo: $dateTo
109
-
offset: $offset
110
-
limit: $limit
111
-
) {
112
-
offset
113
-
limit
114
-
totalCount
115
-
nodes {
116
-
id
117
-
legacyId
118
-
uuid
119
-
group
120
-
type
121
-
kind
122
-
refundKind
123
-
amount {
124
-
value
125
-
currency
126
-
valueInCents
127
-
}
128
-
oppositeTransaction {
129
-
id
130
-
legacyId
131
-
uuid
132
-
# ... other generated fields ...
133
-
merchantId
134
-
invoiceTemplate
135
-
}
136
-
merchantId
137
-
balanceInHostCurrency {
138
-
value
139
-
currency
140
-
valueInCents
141
-
}
142
-
invoiceTemplate
143
-
}
144
-
kinds
145
-
paymentMethodTypes
146
-
}
147
-
}
148
-
```
149
96
150
-
### 2. Build the Factory
97
+
### 2. Build the Asset with Dependencies
151
98
152
-
:::tip
153
-
The GraphQL factory function takes a mandatory `config`
154
-
argument. The other arguments are directly passed to the underlying
155
-
`dlt_factory` function, allowing you to customize the behavior of the asset.
156
-
157
-
For the full reference of the allowed arguments, check out the Dagster
-**Chain GraphQL queries**: Use results from one query to feed parameters into
179
+
another
180
+
-**Transform data flow**: The main query results become intermediate data that
181
+
feeds dependencies
182
+
-**Return consistent shape**: Only the dependency results are yielded to the
183
+
final dataset
184
+
-**Rate limit dependencies**: Control the rate of dependent API calls
185
+
186
+
#### Tree-like Dependency Structure
187
+
188
+
Dependencies can be viewed as a **tree structure** where:
189
+
190
+
- The **root** is your main GraphQL query
191
+
-**Branches** are dependency functions that process each item from the parent
192
+
-**Leaves** are the final data shapes that get yielded to your dataset
193
+
174
194
```
195
+
Main Query (transactions)
196
+
├── Item 1 → fetch_account_details() → Account Data (leaf)
197
+
├── Item 2 → fetch_account_details() → Account Data (leaf)
198
+
└── Item 3 → fetch_account_details() → Account Data (leaf)
199
+
```
200
+
201
+
#### Data Shape Merging
202
+
203
+
When `deps` is provided, **DLT automatically handles different data shapes**:
204
+
205
+
1. The main query executes and fetches data
206
+
2. Each item from the main query is passed to dependency functions
207
+
3. Dependency functions can execute additional GraphQL queries with different schemas
208
+
4.**DLT merges different data shapes** from dependencies automatically
209
+
5.**Only leaf nodes** (final dependency results) are included in the final output
210
+
6. The main query data serves as intermediate processing data and is discarded
211
+
212
+
This means you can have dependencies that return completely different data structures (e.g., transactions → accounts → users), and DLT will intelligently combine them into a coherent dataset, only preserving the final leaf data from your dependency tree.
0 commit comments