Skip to content

Commit bd4add0

Browse files
committed
Add pagination data flow examples
1 parent a09e7af commit bd4add0

8 files changed

+54
-0
lines changed

articles/data-factory/connector-rest.md

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -882,6 +882,60 @@ The backend will automatically get the next URL based on the RFC 5988 style link
882882
>
883883
> :::image type="content" source="media/connector-rest/pagination-example-7-2.png" alt-text="Screenshot showing how to disable RFC 5988 setting for Example 7.":::
884884
885+
#### Example 8: The next request URL is from the response body when use pagination in mapping data flows
886+
887+
This example states how to set the pagination rule and the end condition rule in mapping data flows when the next request URL is from the response body.
888+
889+
The response schema is shown below:
890+
891+
:::image type="content" source="media/connector-rest/pagination-example-8-response-schema.png" alt-text="Screenshot showing the response schema of Example 8.":::
892+
893+
The pagination rules should be set as the following screenshot:
894+
895+
:::image type="content" source="media/connector-rest/pagination-example-8-pagination-rule.png" alt-text="Screenshot showing how to set the pagination rule for Example 8.":::
896+
897+
By default, the pagination will stop when body **.{@odata.nextLink}** is null or empty.
898+
899+
But if the value of **@odata.nextLink** in the last response body is equal to the last request URL, then it will lead to the endless loop. To avoid this condition, define end condition rules.
900+
901+
- If **Value** in the last response is **Empty**, then the end condition rule can be set as below:
902+
903+
:::image type="content" source="media/connector-rest/pagination-example-8-end-condition-1.png" alt-text="Screenshot showing setting the end condition rule when the last response is empty.":::
904+
905+
- If the value of the complete key in the response header equals to true indicates the end of pagination, then the end condition rule can be set as below:
906+
907+
:::image type="content" source="media/connector-rest/pagination-example-8-end-condition-2.png" alt-text="Screenshot showing setting the end condition rule when the complete key in the response header equals to true indicates the end of pagination.":::
908+
909+
#### Example 9: The response format is XML and the next request URL is from the response body when use pagination in mapping data flows
910+
911+
This example states how to set the pagination rule in mapping data flows when the response format is XML and the next request URL is from the response body. As shown in the following screenshot, the first URL is *https://\<user\>.dfs.core.windows.net/bugfix/test/movie_1.xml*
912+
913+
914+
:::image type="content" source="media/connector-rest/pagination-example-9-situation.png" alt-text="Screenshot showing the response format is XML and the next request URL is from the response body.":::
915+
916+
917+
The response schema is shown below:
918+
919+
:::image type="content" source="media/connector-rest/pagination-example-9-response-schema.png" alt-text="Screenshot showing the response schema of Example 9.":::
920+
921+
The pagination rules should be set as below:
922+
923+
:::image type="content" source="media/connector-rest/pagination-example-9-pagination-rule.png" alt-text="Screenshot showing setting the pagination rule for Example 9.":::
924+
925+
926+
>[!NOTE]
927+
> The pagination in mapping data flows is different from it in copy activities in the following aspects:
928+
>1. RANGE in Rest pagination is not supported in mapping data flows.
929+
>2. `['']` is not supported in mapping data flows and will use `{}` to escape special character. For example, `body.{@odata.nextLink}`, and its JSON node `@odata.nextLink` contains special character `.` .
930+
>3. The end condition is supported in mapping data flows, but the condition syntax is a little different from it in copy activities. `$` is not used to indicate the response body, instead, `body` is used. `headers` is not used to indicate the response header, instead, `header` is used. Here are two examples showing this difference:<br/>
931+
> - Example 1:<br/>
932+
> Copy activities: **"EndCondition:$.data": "Empty"**<br/>
933+
> Mapping data flows: **"EndCondition:body.data": "Empty"**<br/>
934+
> - Example 2:<br/>
935+
> Copy activities: **"EndCondition:headers.complete": "Exist"**<br/>
936+
> Mapping data flows: **"EndCondition:header.complete": "Exist"**<br/>
937+
938+
885939
## Use OAuth
886940
This section describes how to use a solution template to copy data from REST connector into Azure Data Lake Storage in JSON format using OAuth.
887941

13.2 KB
Loading
14.9 KB
Loading
8.42 KB
Loading
18.8 KB
Loading
11.1 KB
Loading
46.9 KB
Loading
177 KB
Loading

0 commit comments

Comments
 (0)