Skip to content

Commit 0a4cd06

Browse files
Clare-Zheng82Clare-Zheng82
committed
Update articles/data-factory/connector-rest.md
1 parent 7559262 commit 0a4cd06

File tree

1 file changed

+66
-150
lines changed

1 file changed

+66
-150
lines changed

‎articles/data-factory/connector-rest.md

Lines changed: 66 additions & 150 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ ms.service: data-factory
77
ms.subservice: data-movement
88
ms.custom: synapse
99
ms.topic: conceptual
10-
ms.date: 02/14/2022
10+
ms.date: 02/21/2022
1111
ms.author: makromer
1212
---
1313

@@ -508,7 +508,7 @@ This generic REST connector supports the following pagination patterns:
508508
| Headers.*request_header* OR Headers['request_header'] | "request_header" is user-defined, which references one header name in the next HTTP request. |
509509
| EndCondition:*end_condition* | "end_condition" is user-defined, which indicates the condition that will end the pagination loop in the next HTTP request. |
510510
| MaxRequestNumber | Indicates the maximum pagination request number. Leave it as empty means no limit. |
511-
| SupportRFC5988 | RFC 5988 is supported in the pagination rules. By default, this is set to true. It will only be honored if no other pagination rules are defined. If you don't want to enable this default pagination rule, you can set `supportRFC5988` to `false` or just delete it in script.
511+
| SupportRFC5988 | By default, this is set to true if no pagination rule is defined. You can disable this rule by setting `supportRFC5988` to false or remove this property from script. |
512512

513513
**Supported values** in pagination rules:
514514

@@ -517,102 +517,21 @@ This generic REST connector supports the following pagination patterns:
517517
| Headers.*response_header* OR Headers['response_header'] | "response_header" is user-defined, which references one header name in the current HTTP response, the value of which will be used to issue next request. |
518518
| A JSONPath expression starting with "$" (representing the root of the response body) | The response body should contain only one JSON object. The JSONPath expression should return a single primitive value, which will be used to issue next request. |
519519

520-
**Example:**
521-
522-
Facebook Graph API returns response in the following structure, in which case next page's URL is represented in ***paging.next***:
523-
524-
```json
525-
{
526-
"data": [
527-
{
528-
"created_time": "2017-12-12T14:12:20+0000",
529-
"name": "album1",
530-
"id": "1809938745705498_1809939942372045"
531-
},
532-
{
533-
"created_time": "2017-12-12T14:14:03+0000",
534-
"name": "album2",
535-
"id": "1809938745705498_1809941802371859"
536-
},
537-
{
538-
"created_time": "2017-12-12T14:14:11+0000",
539-
"name": "album3",
540-
"id": "1809938745705498_1809941879038518"
541-
}
542-
],
543-
"paging": {
544-
"cursors": {
545-
"after": "MTAxNTExOTQ1MjAwNzI5NDE=",
546-
"before": "NDMyNzQyODI3OTQw"
547-
},
548-
"previous": "https://graph.facebook.com/me/albums?limit=25&before=NDMyNzQyODI3OTQw",
549-
"next": "https://graph.facebook.com/me/albums?limit=25&after=MTAxNTExOTQ1MjAwNzI5NDE="
550-
}
551-
}
552-
```
553-
554-
The corresponding REST copy activity source configuration especially the `paginationRules` is as follows:
555-
556-
```json
557-
"typeProperties": {
558-
"source": {
559-
"type": "RestSource",
560-
"paginationRules": {
561-
"AbsoluteUrl": "$.paging.next"
562-
},
563-
...
564-
},
565-
"sink": {
566-
"type": "<sink type>"
567-
}
568-
}
569-
```
570-
571-
**Example: Pagination rules**
572-
573-
If you want to send multiple sequence requests with one variable in a range, you can define a variable such as `{offset}`, `{id}` in AbsoluteUrl, Headers, QueryParameters, and define the range rule in pagination rules. See the following examples of pagination rules:
574-
575-
- **Example 1**
576-
577-
You have multiple requests:
578-
579-
```
580-
baseUrl/api/now/table/incident?sysparm_limit=1000&sysparm_offset=0,
581-
582-
baseUrl/api/now/table/incident?sysparm_limit=1000&sysparm_offset=1000,
583-
584-
......
585-
586-
baseUrl/api/now/table/incident?sysparm_limit=1000&sysparm_offset=10000
587-
```
588-
You need to specify the range pagination:
589-
590-
`AbosoluteUrl = baseUrl/api/now/table/incident?sysparm_limit=1000&sysparm_offset={offset}`
591-
592-
The pagination rule is: `QueryParameter.{offset} = RANGE:0:10000:1000`
593-
594-
- **Example 2**
595-
596-
You have multiple requests:
597-
598-
```
599-
baseUrl/api/now/table/t1
600-
601-
baseUrl/api/now/table/t2
602-
603-
......
604-
605-
baseUrl/api/now/table/t100
606-
```
607-
You need to specify the range pagination:
608-
609-
`AbosoluteUrl = baseUrl/api/now/table/t{id}`
610-
611-
The pagination rule is: `AbsoluteUrl.{id} = RANGE:1:100:1`
520+
>[!NOTE]
521+
> The pagination rules in mapping data flows is different from it in copy activity in the following aspects:
522+
>1. Range is not supported in mapping data flows.
523+
>2. `['']` is not supported in mapping data flows. Instead, use `{}` to escape special character. For example, `body.{@odata.nextLink}`, whose JSON node `@odata.nextLink` contains special character `.` .
524+
>3. The end condition is supported in mapping data flows, but the condition syntax is different from it in copy activity. `body` is used to indicate the response body instead of `$`. `header` is used to indicate the response header instead of `headers`. Here are two examples showing this difference:
525+
> - Example 1:
526+
> Copy activity: **"EndCondition:$.data": "Empty"**
527+
> Mapping data flows: **"EndCondition:body.data": "Empty"**
528+
> - Example 2:
529+
> Copy activity: **"EndCondition:headers.complete": "Exist"**
530+
> Mapping data flows: **"EndCondition:header.complete": "Exist"**
612531
613-
### Pagination examples
532+
### Pagination rules examples
614533

615-
This section provides examples of sending different kinds of multiple requests.
534+
This section provides a list of examples for pagination rules settings.
616535

617536
#### Example 1: Variables in QueryParameters
618537

@@ -666,7 +585,6 @@ This example provides the configuration steps to send multiple requests whose va
666585
RequestUrl: *https://example/table*<br/>
667586
Request 1: `Header(id->0)`<br/>
668587
Request 2: `Header(id->10)`<br/>
669-
Request 2: `Header(id->20)`<br/>
670588
......<br/>
671589
Request 100: `Header(id->100)`<br/>
672590

@@ -693,101 +611,112 @@ Two responses encountered in this example:<br/>
693611

694612
Response 1：
695613

696-
```
614+
```json
697615
{
698-
Data: [
699-
{key1: val1, key2: val2},
700-
{key1: val3, key2: val4}
701-
]
616+
Data: [
617+
{key1: val1, key2: val2
618+
},
619+
{key1: val3, key2: val4
620+
}
621+
]
702622
}
703623
```
704624

705625
Response 2：
706-
```
626+
627+
```json
707628
{
708-
Data: [
709-
{key1: val5, key2: val6},
710-
{key1: val7, key2: val8}
711-
]
629+
Data: [
630+
{key1: val5, key2: val6
631+
},
632+
{key1: val7, key2: val8
633+
}
634+
]
712635
}
713636
```
714637

715-
*Step 1*: Set the range of **Pagination rules** as Example 1 and leave the end of range empty as **"AbsoluteUrl.{offset}": "RANGE:0::1000"**.
638+
*Step 1*: Set the range of **Pagination rules** as [Example 1](#example-1-variables-in-queryparameters) and leave the end of range empty as **"AbsoluteUrl.{offset}": "RANGE:0::1000"**.
716639

717-
*Step 2*: Set different end condition rules according to different last responses. Six examples are shown below:
640+
*Step 2*: Set different end condition rules according to different last responses. See below examples:
718641

719-
- **Example 4.1: The pagination will end when the value of the specific node in response is empty**
642+
- **Example 4.1: The pagination ends when the value of the specific node in response is empty**
720643

721644
The REST API returns the last response in the following structure:
722645

723-
```
646+
```json
724647
{
725-
Data: []
648+
Data: []
726649
}
727650
```
728651
Set the end condition rule as **"EndCondition:$.data": "Empty"** to end the pagination when the value of the specific node in response is empty.
729652

730653
:::image type="content" source="media/connector-rest/pagination-example-4-1.png" alt-text="Screenshot showing the EndCondition setting for Example 4.1.":::
731654

732-
- **Example 4.2: The pagination will end when the value of the specific node in response dose not exist**
655+
- **Example 4.2: The pagination ends when the value of the specific node in response dose not exist**
733656

734657
The REST API returns the last response in the following structure:
735658

736-
```
659+
```json
737660
{
738661
}
739662
```
740663
Set the end condition rule as **"EndCondition:$.data": "NonExist"** to end the pagination when the value of the specific node in response dose not exist.
741664

742665
:::image type="content" source="media/connector-rest/pagination-example-4-2.png" alt-text="Screenshot showing the EndCondition setting for Example 4.2.":::
743666

744-
- **Example 4.3: The pagination will end when the value of the specific node in response exists**
667+
- **Example 4.3: The pagination ends when the value of the specific node in response exists**
745668

746669
The REST API returns the last response in the following structure:
747670

748-
```
671+
```json
749672
{
750-
Data: [
751-
{key1: val991, key2: val992},
752-
{key1: val993, key2: val994}
753-
],
754-
Complete: true
673+
Data: [
674+
{key1: val991, key2: val992
675+
},
676+
{key1: val993, key2: val994
677+
}
678+
],
679+
Complete: true
755680
}
756681
```
757682
Set the end condition rule as **"EndCondition:$.Complete": "Exist"** to end the pagination when the value of the specific node in response exists.
758683

759684
:::image type="content" source="media/connector-rest/pagination-example-4-3.png" alt-text="Screenshot showing the EndCondition setting for Example 4.3.":::
760685

761-
- **Example 4.4: The pagination will end when the value of the specific node in response is a user-defined const value**
686+
- **Example 4.4: The pagination ends when the value of the specific node in response is a user-defined const value**
762687

763688
The REST API returns the response in the following structure:
764-
```
689+
```json
765690
{
766-
Data: [
767-
{key1: val1, key2: val2},
768-
{key1: val3, key2: val4}
769-
],
770-
Complete: false
691+
Data: [
692+
{key1: val1, key2: val2
693+
},
694+
{key1: val3, key2: val4
695+
}
696+
],
697+
Complete: false
771698
}
772699
```
773700
......
774701

775702
And the last response is in the following structure:
776703

777-
```
704+
```json
778705
{
779-
Data: [
780-
{key1: val991, key2: val992},
781-
{key1: val993, key2: val994}
782-
],
783-
Complete: true
706+
Data: [
707+
{key1: val991, key2: val992
708+
},
709+
{key1: val993, key2: val994
710+
}
711+
],
712+
Complete: true
784713
}
785714
```
786715
Set the end condition rule as **"EndCondition:$.Complete": "Const:true"** to end the pagination when the value of the specific node in response is a user-defined const value.
787716

788717
:::image type="content" source="media/connector-rest/pagination-example-4-4.png" alt-text="Screenshot showing the EndCondition setting for Example 4.4.":::
789718

790-
- **Example 4.5: The pagination will end when the value of the header key in response is equal to user-defined const value**
719+
- **Example 4.5: The pagination ends when the value of the header key in response equals to user-defined const value**
791720

792721
The header keys in REST API responses are shown in the structure below:
793722

@@ -799,7 +728,7 @@ Data: [
799728

800729
:::image type="content" source="media/connector-rest/pagination-example-4-5.png" alt-text="Screenshot showing the EndCondition setting for Example 4.5.":::
801730

802-
- **Example 4.6: The pagination will end when the key exists in the response header**
731+
- **Example 4.6: The pagination ends when the key exists in the response header**
803732

804733
The header keys in REST API responses are shown in the structure below:
805734

@@ -811,7 +740,7 @@ Data: [
811740

812741
:::image type="content" source="media/connector-rest/pagination-example-4-6.png" alt-text="Screenshot showing the EndCondition setting for Example 4.6.":::
813742

814-
#### Example 5：The range rule is not used but the end condition can be set to avoid endless requests
743+
#### Example 5：Set end condition to avoid endless requests when range rule is not defined
815744

816745
This example provides the configuration steps to send multiple requests when the range rule is not used. The end condition can be set refer to Example 4.1-4.6 to avoid endless requests. The REST API returns response in the following structure, in which case next page's URL is represented in ***paging.next***.
817746

@@ -923,19 +852,6 @@ The pagination rule sytax is the same as in Example 8 and should be set as below
923852
:::image type="content" source="media/connector-rest/pagination-example-9-pagination-rule.png" alt-text="Screenshot showing setting the pagination rule for Example 9.":::
924853

925854

926-
>[!NOTE]
927-
> The pagination in mapping data flows is different from it in copy activities in the following aspects:
928-
>1. RANGE in Rest pagination is not supported in mapping data flows.
929-
>2. `['']` is not supported in mapping data flows and will use `{}` to escape special character. For example, `body.{@odata.nextLink}`, and its JSON node `@odata.nextLink` contains special character `.` .
930-
>3. The end condition is supported in mapping data flows, but the condition syntax is a little different from it in copy activities. `$` is not used to indicate the response body, instead, `body` is used. `headers` is not used to indicate the response header, instead, `header` is used. Here are two examples showing this difference:<br/>
931-
> - Example 1:<br/>
932-
> Copy activities: **"EndCondition:$.data": "Empty"**<br/>
933-
> Mapping data flows: **"EndCondition:body.data": "Empty"**<br/>
934-
> - Example 2:<br/>
935-
> Copy activities: **"EndCondition:headers.complete": "Exist"**<br/>
936-
> Mapping data flows: **"EndCondition:header.complete": "Exist"**<br/>
937-
938-
939855
## Use OAuth
940856
This section describes how to use a solution template to copy data from REST connector into Azure Data Lake Storage in JSON format using OAuth.
941857

0 commit comments

Comments
 (0)