You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/hdinsight-hadoop-script-actions-linux.md
+9-9Lines changed: 9 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ ms.author: hrasheed
6
6
ms.reviewer: jasonh
7
7
ms.service: hdinsight
8
8
ms.topic: conceptual
9
-
ms.date: 04/22/2019
9
+
ms.date: 11/28/2019
10
10
---
11
11
12
12
# Script action development with HDInsight
@@ -70,7 +70,7 @@ fi
70
70
71
71
### <aname="bps10"></a> Target the operating system version
72
72
73
-
Linux-based HDInsight is based on the Ubuntu Linux distribution. Different versions of HDInsight rely on different versions of Ubuntu, which may change how your script behaves. For example, HDInsight 3.4 and earlier are based on Ubuntu versions that use Upstart. Versions 3.5 and greater are based on Ubuntu 16.04, which uses Systemd. Systemd and Upstart rely on different commands, so your script should be written to work with both.
73
+
HDInsight is based on the Ubuntu Linux distribution. Different versions of HDInsight rely on different versions of Ubuntu, which may change how your script behaves. For example, HDInsight 3.4 and earlier are based on Ubuntu versions that use Upstart. Versions 3.5 and greater are based on Ubuntu 16.04, which uses Systemd. Systemd and Upstart rely on different commands, so your script should be written to work with both.
74
74
75
75
Another important difference between HDInsight 3.4 and 3.5 is that `JAVA_HOME` now points to Java 8. The following code demonstrates how to determine if the script is running on Ubuntu 14 or 16:
76
76
@@ -128,11 +128,11 @@ To reduce the time it takes to run the script, avoid operations that compile res
128
128
129
129
Scripts must be idempotent. If the script runs multiple times, it should return the cluster to the same state every time.
130
130
131
-
For example, a script that modifies configuration files should not add duplicate entries if ran multiple times.
131
+
For example, a script that modifies configuration files shouldn't add duplicate entries if ran multiple times.
132
132
133
133
### <aname="bPS5"></a>Ensure high availability of the cluster architecture
134
134
135
-
Linux-based HDInsight clusters provide two head nodes that are active within the cluster, and script actions run on both nodes. If the components you install expect only one head node, do not install the components on both head nodes.
135
+
Linux-based HDInsight clusters provide two head nodes that are active within the cluster, and script actions run on both nodes. If the components you install expect only one head node, don't install the components on both head nodes.
136
136
137
137
> [!IMPORTANT]
138
138
> Services provided as part of HDInsight are designed to fail over between the two head nodes as needed. This functionality is not extended to custom components installed through script actions. If you need high availability for custom components, you must implement your own failover mechanism.
@@ -141,7 +141,7 @@ Linux-based HDInsight clusters provide two head nodes that are active within the
141
141
142
142
Components that you install on the cluster might have a default configuration that uses Apache Hadoop Distributed File System (HDFS) storage. HDInsight uses either Azure Storage or Data Lake Storage as the default storage. Both provide an HDFS compatible file system that persists data even if the cluster is deleted. You may need to configure components you install to use WASB or ADL instead of HDFS.
143
143
144
-
For most operations, you do not need to specify the file system. For example, the following copies the hadoop-common.jar file from the local file system to cluster storage:
144
+
For most operations, you don't need to specify the file system. For example, the following copies the hadoop-common.jar file from the local file system to cluster storage:
@@ -183,7 +183,7 @@ line 1: #!/usr/bin/env: No such file or directory
183
183
184
184
### <aname="bps9"></a> Use retry logic to recover from transient errors
185
185
186
-
When downloading files, installing packages using apt-get, or other actions that transmit data over the internet, the action may fail due to transient networking errors. For example, the remote resource you are communicating with may be in the process of failing over to a backup node.
186
+
When downloading files, installing packages using apt-get, or other actions that transmit data over the internet, the action may fail because of transient networking errors. For example, the remote resource you're communicating with may be in the process of failing over to a backup node.
187
187
188
188
To make your script resilient to transient errors, you can implement retry logic. The following function demonstrates how to implement retry logic. It retries the operation three times before failing.
189
189
@@ -232,7 +232,7 @@ The following helpers available for use in your script:
232
232
233
233
| Helper usage | Description |
234
234
| --- | --- |
235
-
|`download_file SOURCEURL DESTFILEPATH [OVERWRITE]`|Downloads a file from the source URI to the specified file path. By default, it does not overwrite an existing file. |
235
+
|`download_file SOURCEURL DESTFILEPATH [OVERWRITE]`|Downloads a file from the source URI to the specified file path. By default, it doesn't overwrite an existing file. |
236
236
|`untar_file TARFILE DESTDIR`|Extracts a tar file (using `-xf`) to the destination directory. |
237
237
|`test_is_headnode`|If ran on a cluster head node, return 1; otherwise, 0. |
238
238
|`test_is_datanode`|If the current node is a data (worker) node, return a 1; otherwise, 0. |
@@ -324,15 +324,15 @@ Microsoft provides sample scripts to install components on an HDInsight cluster.
324
324
325
325
## Troubleshooting
326
326
327
-
The following are errors you may encounter when using scripts you have developed:
327
+
The following are errors you may come across when using scripts you've developed:
328
328
329
329
**Error**: `$'\r': command not found`. Sometimes followed by `syntax error: unexpected end of file`.
330
330
331
331
*Cause*: This error is caused when the lines in a script end with CRLF. Unix systems expect only LF as the line ending.
332
332
333
333
This problem most often occurs when the script is authored on a Windows environment, as CRLF is a common line ending for many text editors on Windows.
334
334
335
-
*Resolution*: If it is an option in your text editor, select Unix format or LF for the line ending. You may also use the following commands on a Unix system to change the CRLF to an LF:
335
+
*Resolution*: If it's an option in your text editor, select Unix format or LF for the line ending. You may also use the following commands on a Unix system to change the CRLF to an LF:
336
336
337
337
> [!NOTE]
338
338
> The following commands are roughly equivalent in that they should change the CRLF line endings to LF. Select one based on the utilities available on your system.
0 commit comments