You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/synapse-analytics/spark/apache-spark-manage-session-packages.md
+13-10Lines changed: 13 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -38,7 +38,15 @@ To specify session-scoped Python packages:
38
38
39
39
You can use the popular *%pip* and *%conda* commands to install additional third party libraries or your custom libraries during your Apache Spark notebook session. In this section, we use *%pip* commands to demonstrate several common scenarios.
40
40
41
-
### Install a third party package
41
+
> [!NOTE]
42
+
>
43
+
> - We recommend to put the *%pip* and *%conda* commands at the first cell of your notebook if you want to install new libraries. The Python interpreter will be restarted after the session-level library being managed to bring the changes effective.
44
+
> - These commands of managing Python libraries will be disabled when running pipeline jobs. If you want to install a package within a pipeline, you must leverage the library management capabilities at the pool level.
45
+
> - Session-scoped Python libraries are automatically installed across both the driver and worker nodes.
46
+
> - The following *%conda* commands are not supported: *create*, *clean*, *compare*, *activate*, *deactivate*, *run*, *package*.
47
+
> - You can refer to [%pip commands](https://pip.pypa.io/en/stable/cli/) and [%conda commands](https://docs.conda.io/projects/conda/en/latest/commands.html) for the full list of commands.
48
+
49
+
#### Install a third party package
42
50
43
51
You can easily install a Python library from [PyPI](https://pypi.org/).
In order to install library from storage, you need to mount to your storage account by running following commands.
70
78
@@ -84,7 +92,7 @@ And then, you can use the *%pip install* command to install the required wheel p
84
92
%pip install /<<path to wheel file>>/<<wheel package name>>.whl
85
93
```
86
94
87
-
### Install another version of built-in library
95
+
####Install another version of built-in library
88
96
89
97
You can use the following command to see what's the built-in version of certain package. We use *pandas* as an example
90
98
@@ -108,25 +116,20 @@ You can use the following command to switch *pandas* to another version, let's s
108
116
%pip install pandas==1.2.4
109
117
```
110
118
111
-
### Uninstall a session-scoped library
119
+
####Uninstall a session-scoped library
112
120
113
121
If you want to uninstall a package, which installed on this notebook session, you may refer to following commands. However, you cannot uninstall the built-in packages.
114
122
115
123
```python
116
124
%pip uninstall altair vega_datasets --yes
117
125
```
118
126
119
-
### Using *%pip* command to install libraries from a *requirement.txt* file
127
+
####Using *%pip* command to install libraries from a *requirement.txt* file
120
128
121
129
```python
122
130
%pip install -r /<<path to requirement file>>/requirements.txt
123
131
```
124
132
125
-
> [!NOTE]
126
-
>
127
-
> - We recommend you to put the *%pip* and *%conda* commands at the beginning of your notebook if you want to install new libraries. The python interpreter will be restarted after the session-level library being managed to bring the changes effective.
128
-
> - You can refer to this [%pip commands](https://pip.pypa.io/en/stable/cli/) and [%conda commands](https://docs.conda.io/projects/conda/en/latest/commands.html) for the full list of available commands.
129
-
130
133
## Session-scoped Java or Scala packages
131
134
132
135
To specify session-scoped Java or Scala packages, you can use the ```%%configure``` option:
0 commit comments