Skip to content

Commit 1aa220e

Browse files
committed
Change section position
1 parent 4028ed5 commit 1aa220e

File tree

1 file changed

+103
-103
lines changed

1 file changed

+103
-103
lines changed

articles/data-factory/data-flow-troubleshoot-guide.md

Lines changed: 103 additions & 103 deletions
Original file line numberDiff line numberDiff line change
@@ -22,109 +22,6 @@ This article explores common troubleshooting methods for mapping data flows in A
2222
2. Check the status of your file and table connections in the data flow designer. In debug mode, select **Data Preview** on your source transformations to ensure that you can access your data.
2323
3. If everything looks correct in data preview, go into the Pipeline designer and put your data flow in a Pipeline activity. Debug the pipeline for an end-to-end test.
2424

25-
### Improvement on CSV/CDM format in Data Flow
26-
27-
If you use the **Delimited Text or CDM formatting for mapping data flow in Azure Data Factory V2**, you may face the behavior changes to your existing pipelines because of the improvement for Delimited Text/CDM in data flow starting from **1 May 2021**.
28-
29-
You may encounter the following issues before the improvement, but after the improvement, the issues were fixed. Read the following content to determine whether this improvement affects you.
30-
31-
#### Scenario 1: Encounter the unexpected row delimiter issue
32-
33-
You are affected if you are in the following conditions:
34-
- Using the Delimited Text with the Multiline setting set to True or CDM as the source.
35-
- The first row has more than 128 characters.
36-
- The row delimiter in data files is not `\n`.
37-
38-
Before the improvement, the default row delimiter `\n` may be unexpectedly used to parse delimited text files, because when Multiline setting is set to True, it invalidates the row delimiter setting, and the row delimiter is automatically detected based on the first 128 characters. If you fail to detect the actual row delimiter, it would fall back to `\n`.
39-
40-
After the improvement, any one of the three-row delimiters: `\r`, `\n`, `\r\n` should have worked.
41-
42-
The following example shows you one pipeline behavior change after the improvement:
43-
44-
**Example**:<br/>
45-
For the following column:<br/>
46-
`C1, C2, {long first row}, C128\r\n `<br/>
47-
`V1, V2, {values………………….}, V128\r\n `<br/>
48-
49-
Before the improvement, `\r` is kept in the column value. The parsed column result is:<br/>
50-
`C1 C2 {long first row} C128`**`\r`**<br/>
51-
`V1 V2 {values………………….} V128`**`\r`**<br/> 
52-
53-
After the improvement, the parsed column result should be:<br/>
54-
`C1 C2 {long first row} C128`<br/>
55-
`V1 V2 {values………………….} V128`<br/>
56-
57-
#### Scenario 2: Encounter an issue of incorrectly reading column values containing '\r\n'
58-
59-
You are affected if you are in the following conditions:
60-
- Using the Delimited Text with the Multiline setting set to True or CDM as a source.
61-
- The row delimiter is `\r\n`.
62-
63-
Before the improvement, when reading the column value, the `\r\n` in it may be incorrectly replaced by `\n`.
64-
65-
After the improvement, `\r\n` in the column value will not be replaced by `\n`.
66-
67-
The following example shows you one pipeline behavior change after the improvement:
68-
69-
**Example**:<br/>
70-
71-
For the following column:<br/>
72-
**`"A\r\n"`**`, B, C\r\n`<br/>
73-
74-
Before the improvement, the parsed column result is:<br/>
75-
**`A\n`**` B C`<br/>
76-
77-
After the improvement, the parsed column result should be:<br/>
78-
**`A\r\n`**` B C`<br/>
79-
80-
#### Scenario 3: Encounter an issue of incorrectly writing column values containing '\n'
81-
82-
You are affected if you are in the following conditions:
83-
- Using the Delimited Text as a sink.
84-
- The column value contains `\n`.
85-
- The row delimiter is set to `\r\n`.
86-
87-
Before the improvement, when writing the column value, the `\n` in it may be incorrectly replaced by `\r\n`.
88-
89-
After the improvement, `\n` in the column value will not be replaced by `\r\n`.
90-
91-
The following example shows you one pipeline behavior change after the improvement:
92-
93-
**Example**:<br/>
94-
95-
For the following column:<br/>
96-
**`A\n`**` B C`<br/>
97-
98-
Before the improvement, the CSV sink is:<br/>
99-
**`"A\r\n"`**`, B, C\r\n` <br/>
100-
101-
After the improvement, the CSV sink should be:<br/>
102-
**`"A\n"`**`, B, C\r\n`<br/>
103-
104-
#### Scenario 4: Encounter an issue of incorrectly reading empty string as NULL
105-
106-
You are affected if you are in the following conditions:
107-
- Using the Delimited Text as a source.
108-
- NULL value is set to non-empty value.
109-
- The column value is empty string and is unquoted.
110-
111-
Before the improvement, the column value of unquoted empty string is read as NULL.
112-
113-
After the improvement, empty string will not be parsed as NULL value.
114-
115-
The following example shows you one pipeline behavior change after the improvement:
116-
117-
**Example**:<br/>
118-
119-
For the following column:<br/>
120-
`A, ,B, `<br/>
121-
122-
Before the improvement, the parsed column result is:<br/>
123-
`A null B null`<br/>
124-
125-
After the improvement, the parsed column result should be:<br/>
126-
`A "" (empty string) B "" (empty string)`<br/>
127-
12825
### Internal server errors
12926

13027
Specific scenarios that can cause internal server errors are shown as follows.
@@ -1043,6 +940,109 @@ This section lists common error codes and messages reported by mapping data flow
1043940
- **Cause**: In the Mapping data flow, multiline CSV source files don't currently work when \r\n is used as the row delimiter. Sometimes extra lines at carriage returns can cause errors.
1044941
- **Recommendation**: Generate the file at the source by using \n as the row delimiter rather than \r\n. Or use the Copy activity to convert the CSV file to use \n as a row delimiter.
1045942

943+
### Improvement on CSV/CDM format in Data Flow
944+
945+
If you use the **Delimited Text or CDM formatting for mapping data flow in Azure Data Factory V2**, you may face the behavior changes to your existing pipelines because of the improvement for Delimited Text/CDM in data flow starting from **1 May 2021**.
946+
947+
You may encounter the following issues before the improvement, but after the improvement, the issues were fixed. Read the following content to determine whether this improvement affects you.
948+
949+
#### Scenario 1: Encounter the unexpected row delimiter issue
950+
951+
You are affected if you are in the following conditions:
952+
- Using the Delimited Text with the Multiline setting set to True or CDM as the source.
953+
- The first row has more than 128 characters.
954+
- The row delimiter in data files is not `\n`.
955+
956+
Before the improvement, the default row delimiter `\n` may be unexpectedly used to parse delimited text files, because when Multiline setting is set to True, it invalidates the row delimiter setting, and the row delimiter is automatically detected based on the first 128 characters. If you fail to detect the actual row delimiter, it would fall back to `\n`.
957+
958+
After the improvement, any one of the three-row delimiters: `\r`, `\n`, `\r\n` should have worked.
959+
960+
The following example shows you one pipeline behavior change after the improvement:
961+
962+
**Example**:<br/>
963+
For the following column:<br/>
964+
`C1, C2, {long first row}, C128\r\n `<br/>
965+
`V1, V2, {values………………….}, V128\r\n `<br/>
966+
967+
Before the improvement, `\r` is kept in the column value. The parsed column result is:<br/>
968+
`C1 C2 {long first row} C128`**`\r`**<br/>
969+
`V1 V2 {values………………….} V128`**`\r`**<br/> 
970+
971+
After the improvement, the parsed column result should be:<br/>
972+
`C1 C2 {long first row} C128`<br/>
973+
`V1 V2 {values………………….} V128`<br/>
974+
975+
#### Scenario 2: Encounter an issue of incorrectly reading column values containing '\r\n'
976+
977+
You are affected if you are in the following conditions:
978+
- Using the Delimited Text with the Multiline setting set to True or CDM as a source.
979+
- The row delimiter is `\r\n`.
980+
981+
Before the improvement, when reading the column value, the `\r\n` in it may be incorrectly replaced by `\n`.
982+
983+
After the improvement, `\r\n` in the column value will not be replaced by `\n`.
984+
985+
The following example shows you one pipeline behavior change after the improvement:
986+
987+
**Example**:<br/>
988+
989+
For the following column:<br/>
990+
**`"A\r\n"`**`, B, C\r\n`<br/>
991+
992+
Before the improvement, the parsed column result is:<br/>
993+
**`A\n`**` B C`<br/>
994+
995+
After the improvement, the parsed column result should be:<br/>
996+
**`A\r\n`**` B C`<br/>
997+
998+
#### Scenario 3: Encounter an issue of incorrectly writing column values containing '\n'
999+
1000+
You are affected if you are in the following conditions:
1001+
- Using the Delimited Text as a sink.
1002+
- The column value contains `\n`.
1003+
- The row delimiter is set to `\r\n`.
1004+
1005+
Before the improvement, when writing the column value, the `\n` in it may be incorrectly replaced by `\r\n`.
1006+
1007+
After the improvement, `\n` in the column value will not be replaced by `\r\n`.
1008+
1009+
The following example shows you one pipeline behavior change after the improvement:
1010+
1011+
**Example**:<br/>
1012+
1013+
For the following column:<br/>
1014+
**`A\n`**` B C`<br/>
1015+
1016+
Before the improvement, the CSV sink is:<br/>
1017+
**`"A\r\n"`**`, B, C\r\n` <br/>
1018+
1019+
After the improvement, the CSV sink should be:<br/>
1020+
**`"A\n"`**`, B, C\r\n`<br/>
1021+
1022+
#### Scenario 4: Encounter an issue of incorrectly reading empty string as NULL
1023+
1024+
You are affected if you are in the following conditions:
1025+
- Using the Delimited Text as a source.
1026+
- NULL value is set to non-empty value.
1027+
- The column value is empty string and is unquoted.
1028+
1029+
Before the improvement, the column value of unquoted empty string is read as NULL.
1030+
1031+
After the improvement, empty string will not be parsed as NULL value.
1032+
1033+
The following example shows you one pipeline behavior change after the improvement:
1034+
1035+
**Example**:<br/>
1036+
1037+
For the following column:<br/>
1038+
`A, ,B, `<br/>
1039+
1040+
Before the improvement, the parsed column result is:<br/>
1041+
`A null B null`<br/>
1042+
1043+
After the improvement, the parsed column result should be:<br/>
1044+
`A "" (empty string) B "" (empty string)`<br/>
1045+
10461046
## Next steps
10471047

10481048
For more help with troubleshooting, see these resources:

0 commit comments

Comments
 (0)