You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/iot-operations/connect-to-cloud/concept-dataflow-language.md
+25-25Lines changed: 25 additions & 25 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -87,24 +87,24 @@ The example maps:
87
87
88
88
Field references show how to specify paths in the input and output, using dot notation like `Employee.DateOfBirth` or accessing data from a contextual dataset via `$context(position)`.
89
89
90
-
## Contextualization Dataset Selectors
90
+
## Contextualization dataset selectors
91
91
92
-
These selectors allow mappings to integrate additional data from external databases, referred to as *Contextualization Datasets*.
92
+
These selectors allow mappings to integrate additional data from external databases, referred to as *contextualization datasets*.
93
93
94
-
## Record Filtering
94
+
## Record filtering
95
95
96
96
Record filtering involves setting conditions to select which records should be processed or dropped.
97
97
98
98
## Dot-notation
99
99
100
-
Dot-notation is widely used in computer science to reference fields, even recursively. In programming, field names typically consist of letters and numbers, so a standard dot-notation might look like this:
100
+
Dot-notation is widely used in computer science to reference fields, even recursively. In programming, field names typically consist of letters and numbers. A standard dot-notation might look like this:
101
101
102
102
```yaml
103
103
- inputs:
104
104
- Person.Address.Street.Number
105
105
```
106
106
107
-
However, in dataflow a path described by dot-notation might include strings and some special characters without needing escaping:
107
+
However, in a dataflow a path described by dot-notation might include strings and some special characters without needing escaping:
108
108
109
109
```yaml
110
110
- inputs:
@@ -127,7 +127,7 @@ While dataflow parses a path, it treats only two characters as special:
127
127
128
128
Any other characters are treated as part of the field name. This flexibility is useful in formats like JSON, where field names can be arbitrary strings.
129
129
130
-
Note, however, that the path definition must also adhere to the rules of YAML: once a character with special meaning is included in the path, proper quoting is required in the configuration. Consult the YAML documentation for precise rules. Here are some examples that demonstrate the need for careful formatting:
130
+
Notethat the path definition must also adhere to the rules of YAML. Once a character with special meaning is included in the path, proper quoting is required in the configuration. Consult the YAML documentation for precise rules. Here are some examples that demonstrate the need for careful formatting:
131
131
132
132
```yaml
133
133
- inputs:
@@ -146,7 +146,7 @@ The primary function of escaping in a dot-notated path is to accommodate the use
146
146
147
147
In the previous example, the path consists of three segments: `Payload`, `Tag.10`, and `Value`. The outer single quotes (`'`) are necessary because of YAML syntax rules, allowing the inclusion of double quotes within the string.
148
148
149
-
**Escaping Rules in Dot-Notation:**
149
+
### Escaping rules in dot notation
150
150
151
151
* **Escape Each Segment Separately**: If multiple segments contain dots, those segments must be enclosed in quotes. Other segments can also be quoted, but it doesn't affect the path interpretation:
152
152
@@ -171,7 +171,7 @@ This example defines two fields in the dataDestination: `Payload` and `He said:
171
171
172
172
In this case, the path is split into the segments `Payload`, `He said: "No`, and `It's done"` (starting with a space).
173
173
174
-
**Segmentation Algorithm**:
174
+
### Segmentation algorithm
175
175
176
176
* If the first character of a segment is a quote, the parser searches for the next quote. The string enclosed between these quotes is considered a single segment.
177
177
* If the segment doesn't start with a quote, the parser identifies segments by searching for the next dot or the end of the path.
@@ -197,9 +197,9 @@ Here's how the asterisk (`*`) operates in this context:
197
197
198
198
This configuration demonstrates the most generic form of mapping, where every field in the input is directly mapped to a corresponding field in the output without modification.
199
199
200
-
Another example illustrates how wildcards can be used to match sub-sections and move them together. This example effectively flattens nested structures within a JSON object:
200
+
Another example illustrates how wildcards can be used to match sub-sections and move them together. This example effectively flattens nested structures within a JSON object.
201
201
202
-
*Original JSON:*
202
+
Original JSON:
203
203
204
204
```json
205
205
{
@@ -218,7 +218,7 @@ Another example illustrates how wildcards can be used to match sub-sections and
218
218
}
219
219
```
220
220
221
-
*Mapping Configuration Using Wildcards:*
221
+
Mapping Configuration Using Wildcards:
222
222
223
223
```yaml
224
224
- inputs:
@@ -230,7 +230,7 @@ Another example illustrates how wildcards can be used to match sub-sections and
230
230
output: *
231
231
```
232
232
233
-
*Resulting JSON:*
233
+
Resulting JSON:
234
234
235
235
```json
236
236
{
@@ -256,7 +256,7 @@ When placing a wildcard, the following rules must be followed:
256
256
* **In the Middle:** `path1.*.path3` - In this configuration, the asterisk matches any segment between `path1` and `path3`.
257
257
* **At the End:** `path1.path2.*` - The asterisk at the end matches any segment that follows after `path1.path2`.
258
258
259
-
### Multi-Input Wildcards
259
+
### Multi-input wildcards
260
260
261
261
*Original JSON:*
262
262
@@ -277,7 +277,7 @@ When placing a wildcard, the following rules must be followed:
277
277
}
278
278
```
279
279
280
-
*Mapping Configuration Using Wildcards:*
280
+
Mapping Configuration Using wildcards:
281
281
282
282
```yaml
283
283
- inputs:
@@ -287,7 +287,7 @@ When placing a wildcard, the following rules must be followed:
287
287
conversion: ($1 + $2) / 2
288
288
```
289
289
290
-
*Resulting JSON:*
290
+
Resulting JSON:
291
291
292
292
```json
293
293
{
@@ -303,7 +303,7 @@ If multi-input wildcards, the asterisk (`*`) must consistently represent the sam
303
303
304
304
Consider this detailed example:
305
305
306
-
**Original JSON:**
306
+
Original JSON:
307
307
308
308
```json
309
309
{
@@ -334,7 +334,7 @@ Consider this detailed example:
334
334
}
335
335
```
336
336
337
-
**Initial Mapping Configuration Using Wildcards:**
337
+
Initial mapping configuration using wildcards:
338
338
339
339
```yaml
340
340
- inputs:
@@ -357,7 +357,7 @@ This initial mapping tries to build an array (For example, for `Opacity`: `[0.88
357
357
358
358
Since `Avg` and `Mean` are nested within `Mid`, the asterisk in the initial mapping doesn't correctly capture these paths.
359
359
360
-
**Corrected Mapping Configuration:**
360
+
Corrected mapping configuration:
361
361
362
362
```yaml
363
363
- inputs:
@@ -371,9 +371,9 @@ Since `Avg` and `Mean` are nested within `Mid`, the asterisk in the initial mapp
371
371
372
372
This revised mapping accurately captures the necessary fields by correctly specifying the paths to include the nested `Mid` object, ensuring that the asterisks work effectively across different levels of the JSON structure.
373
373
374
-
### Second Rule vs. Specialization
374
+
### Second rule versus specialization
375
375
376
-
Using the example from Multi-Input Wildcards, consider the following mappings that generate two derived values for each property:
376
+
Using the previous example from multi-input wildcards, consider the following mappings that generate two derived values for each property:
377
377
378
378
```yaml
379
379
- inputs:
@@ -410,7 +410,7 @@ This mapping is intended to create two separate calculations (`Avg` and `Diff`)
410
410
}
411
411
```
412
412
413
-
Here, the second mapping definition on the same inputs acts as a `Second Rule` for mapping.
413
+
Here, the second mapping definition on the same inputs acts as a *second rule* for mapping.
414
414
415
415
Now, consider a scenario where a specific field needs a different calculation:
416
416
@@ -450,7 +450,7 @@ Consider a special case for the same fields to help deciding the right action:
450
450
451
451
An empty `output` field in the second definition implies not writing the fields in the output record (effectively removing `Opacity`). This setup is more of a `Specialization` than a `Second Rule`.
452
452
453
-
**Resolution of Overlapping Mappings by dataflow:**
453
+
Resolution of overlapping mappings by dataflow:
454
454
455
455
* The evaluation progresses from the top rule in the mapping definition.
456
456
* If a new mapping resolves to the same fields as a previous rule, the following applies:
@@ -543,7 +543,7 @@ Conversions use simple math formulas similar to those learned in middle school.
543
543
544
544
For more complex calculations, functions like `sqrt` (which finds the square root of a number) are also available.
545
545
546
-
**Available arithmetic, comparison, and boolean operators grouped by precedence:**
546
+
### Available arithmetic, comparison, and boolean operators grouped by precedence
547
547
548
548
| Operator | Description |
549
549
|----------|-------------|
@@ -654,7 +654,7 @@ The mapper is designed to be flexible, converting internal types into output typ
654
654
655
655
While the automatic conversions generally operate as one might expect, based on common implementation practices, there are instances where the right conversion cannot be determined automatically, resulting in an `unsupported` error. To address these situations, several conversion functions are available to explicitly define how data should be transformed. These functions provide more control over how data is converted and ensure that data integrity is maintained even when automatic methods fall short.
656
656
657
-
<!--**[FIXME - actually we don't have functions, need a list of what they usually want]**-->
657
+
<!-- We don't have functions, need a list of what they usually want-->
658
658
659
659
### Using conversion formula with types
660
660
@@ -778,7 +778,7 @@ There are numerous other functions available in different categories:
778
778
* string manipulation (for example, `uppercase()`)
779
779
* explicit conversion (for example, `ISO8601_datetime`)
780
780
* aggregation (for example, `avg()`)
781
-
* **[FIXME we actually don't have much methods, need a list about what to implement]**
781
+
<!--- minimal methods, need a list about what to implement -->
0 commit comments