You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Learn how to use Script Actions to configure an [Apache Spark](https://spark.apache.org/) cluster on HDInsight to use external, community-contributed **python** packages that aren't included out-of-the-box in the cluster.
19
-
20
-
> [!NOTE]
21
-
> You can also configure a Jupyter notebook by using `%%configure` magic to use external packages. For instructions, see [Use external packages with Jupyter notebooks in Apache Spark clusters on HDInsight](apache-spark-jupyter-notebook-use-external-packages.md).
22
-
23
-
You can search the [package index](https://pypi.python.org/pypi) for the complete list of packages that are available. You can also get a list of available packages from other sources. For example, you can install packages made available through [conda-forge](https://conda-forge.org/feedstocks/).
24
-
25
-
In this article, you learn how to install the [TensorFlow](https://www.tensorflow.org/) package using Script Action on your cluster and use it via the Jupyter notebook as an example.
18
+
HDInsight has two built-in Python installations in the Spark cluster, Anaconda Python 2.7 and Python 3.5. In some cases, customers need to customize the Python environment, like installing external Python packages or another Python version. In this article, we show the best practice of safely managing Python environments for an [Apache Spark](https://spark.apache.org/) cluster on HDInsight.
26
19
27
20
## Prerequisites
28
21
@@ -39,61 +32,94 @@ The Microsoft Azure HDInsight service uses an ecosystem of open-source technolog
39
32
40
33
There are two types of open-source components that are available in the HDInsight service:
41
34
42
-
***Built-in components** - These components are pre-installed on HDInsight clusters and provide core functionality of the cluster. For example, Apache Hadoop YARN ResourceManager, the Apache Hive query language (HiveQL), and the Mahout library belong to this category. A full list of cluster components is available in [What's new in the Apache Hadoop cluster versions provided by HDInsight](https://docs.microsoft.com/azure/hdinsight/hdinsight-component-versioning).
35
+
***Built-in components** - These components are pre-installed on HDInsight clusters and provide core functionality of the cluster. For example, Apache Hadoop YARN Resource Manager, the Apache Hive query language (HiveQL), and the Mahout library belong to this category. A full list of cluster components is available in [What's new in the Apache Hadoop cluster versions provided by HDInsight](https://docs.microsoft.com/azure/hdinsight/hdinsight-component-versioning).
43
36
***Custom components** - You, as a user of the cluster, can install or use in your workload any component available in the community or created by you.
44
37
45
38
> [!IMPORTANT]
46
39
> Components provided with the HDInsight cluster are fully supported. Microsoft Support helps to isolate and resolve issues related to these components.
47
40
>
48
41
> Custom components receive commercially reasonable support to help you to further troubleshoot the issue. Microsoft support may be able to resolve the issue OR they may ask you to engage available channels for the open source technologies where deep expertise for that technology is found. For example, there are many community sites that can be used, like: [MSDN forum for HDInsight](https://social.msdn.microsoft.com/Forums/azure/home?forum=hdinsight), [https://stackoverflow.com](https://stackoverflow.com). Also Apache projects have project sites on [https://apache.org](https://apache.org), for example: [Hadoop](https://hadoop.apache.org/).
49
42
50
-
## Use external packages with Jupyter notebooks
43
+
## Understand default Python installation
44
+
45
+
HDInsight Spark cluster is created with Anaconda installation. There are two Python installations in the cluster, Anaconda Python 2.7 and Python 3.5. The table below shows the default Python settings for Spark, Livy, and Jupyter.
51
46
52
-
1. From the [Azure portal](https://portal.azure.com/), navigate to your cluster.
2. With your cluster selected, from the left pane, under **Settings**, select **Script actions**.
54
+
## Safely install external Python packages
55
55
56
-
3. Select **+ Submit new**.
56
+
HDInsight cluster depends on the built-in Python environment, both Python 2.7 and Python 3.5. Directly installing custom packages in those default built-in environments may cause unexpected library version changes, and break the cluster further. In order to safely install custom external Python packages for your Spark applications, follow below steps.
57
+
58
+
1. Create Python virtual environment using conda. A virtual environment provides an isolated space for your projects without breaking others. When creating the Python virtual environment, you can specify python version that you want to use. Note that you still need to create virtual environment even though you would like to use Python 2.7 and 3.5. This is to make sure the cluster’s default environment not getting broke. Run script actions on your cluster for all nodes with below script to create a Python virtual environment.
59
+
60
+
-`--prefix` specifies a path where a conda virtual environment lives. There are several configs that need to be changed further based on the path specified here. In this example, we use the py35new, as the cluster has an existing virtual environment called py35 already.
61
+
-`python=` specifies the Python version for the virtual environment. In this example, we use version 3.5, the same version as the cluster built in one. You can also use other Python versions to create the virtual environment.
62
+
-`anaconda` specifies the package_spec as anaconda to install Anaconda packages in the virtual environment.
4. Enter the following values for the **Submit script action** window:
68
+
2. Install external Python packages inthe created virtual environment if needed. Run script actions on your cluster forall nodes with below script to install external Python packages. You need to have sudo privilege herein order to write files to the virtual environment folder.
59
69
60
-
|Parameter | Value |
61
-
|---|---|
62
-
|Script type | Select **- Custom** from the drop-down list.|
63
-
|Name |Enter `tensorflow` in the text box.|
64
-
|Bash script URI |Enter `https://hdiconfigactions.blob.core.windows.net/linuxtensorflow/tensorflowinstall.sh` in the text box. |
65
-
|Node type(s) | Select the **Head**, and **Worker** check boxes. |
70
+
You can search the [package index](https://pypi.python.org/pypi) for the complete list of packages that are available. You can also get a list of available packages from other sources. For example, you can install packages made available through [conda-forge](https://conda-forge.org/feedstocks/).
66
71
67
-
`tensorflowinstall.sh` contains the following commands:
72
+
- `seaborn` is the package name that you would like to install.
73
+
- `-n py35new` specify the virtual environment name that just gets created. Make sure to change the name correspondingly based on your virtual environment creation.
5. Select **Create**. Visit the documentation on [how to use custom script actions](../hdinsight-hadoop-customize-cluster-linux.md).
79
+
if you don't know the virtual environment name, you can SSH to the header node of the cluster and run `/usr/bin/anaconda/bin/conda info -e` to show all virtual environments.
75
80
76
-
6. Wait for the script to complete. The **Script actions** pane will state **New script actions can be submitted after the current cluster operation finishes**while the script is executing. A progress bar can be viewed from the Ambari UI **Background Operations** window.
81
+
3. Change Spark and Livy configs and point to the created virtual environment.
77
82
78
-
7. Open a PySpark Jupyter notebook. See [Create a Jupyter notebook on Spark HDInsight](./apache-spark-jupyter-notebook-kernels.md#create-a-jupyter-notebook-on-spark-hdinsight) for steps.
83
+
1. Open Ambari UI, go to Spark2 page, Configs tab.
84
+
85
+
86
+
87
+
2. Expand Advanced livy2-env, add below statements at bottom. If you installed the virtual environment with a different prefix, change the path correspondingly.
79
88
80
-
8. You will now `import tensorflow` and run a hello world example. Enter the following code:
94
+
95
+
96
+
3. Expand Advanced spark2-env, replace the existing export PYSPARK_PYTHON statement at bottom. If you installed the virtual environment with a different prefix, change the path correspondingly.
103
+
104
+
4. Save the changes and restart affected services. These changes need a restart of Spark2 service. Ambari UI will prompt a required restart reminder, click Restart to restart all affected services.
105
+
106
+
107
+
108
+
4. If you would like to use the new created virtual environment on Jupyter. You need to change Jupyter configs and restart Jupyter. Run script actions on all header nodes with below statement to point Jupyter to the new created virtual environment. Make sure to modify the path to the prefix you specified for your virtual environment. After running this script action, restart Jupyter service through Ambari UI to make this change available.
83
109
84
110
```
85
-
import tensorflow as tf
86
-
hello = tf.constant('Hello, TensorFlow!')
87
-
sess = tf.Session()
88
-
print(sess.run(hello))
111
+
sudo sed -i '/python3_executable_path/c\ \"python3_executable_path\":\"/usr/bin/anaconda/envs/py35new/bin/python3\"' /home/spark/.sparkmagic/config.json
You could double confirm the Python environment in Jupyter Notebook by running below code:
115
+
116
+
117
+
118
+
## Known issue
119
+
120
+
There is a known bug for Anaconda version 4.7.11 and 4.7.12. If you see your script actions hanging at `"Collecting package metadata (repodata.json): ...working..."` and failing with `"Python script has been killed due to timeout after waiting 3600 secs"`. You can download [this script](https://gregorysfixes.blob.core.windows.net/public/fix-conda.sh) and run it as script actions on all nodes to fix the issue.
94
121
95
-
> [!NOTE]
96
-
> There are two python installations in the cluster. Spark will use the Anaconda python installation located at `/usr/bin/anaconda/bin` and will default to the Python 2.7 environment. To use Python 3.x and install packages in the PySpark3 kernel, use the path to the `conda` executable for that environment and use the `-n` parameter to specify the environment. For example, the command`/usr/bin/anaconda/envs/py35/bin/conda install -c conda-forge ggplot -n py35`, installs the `ggplot` package to the Python 3.5 environment using the `conda-forge` channel.
122
+
To check your Anaconda version, you can SSH to the cluster header node and run `/usr/bin/anaconda/bin/conda --v`.
0 commit comments