You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/synapse-analytics/spark/apache-spark-manage-session-packages.md
+17-17Lines changed: 17 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,12 +14,12 @@ ms.subservice: spark
14
14
15
15
In addition to pool level packages, you can also specify session-scoped libraries at the beginning of a notebook session. Session-scoped libraries let you specify and use Python, jar, and R packages within a notebook session.
16
16
17
-
When using session-scoped libraries, it is important to keep the following points in mind:
17
+
When using session-scoped libraries, it's important to keep the following points in mind:
18
18
19
19
- When you install session-scoped libraries, only the current notebook has access to the specified libraries.
20
-
- These libraries will not impact other sessions or jobs using the same Spark pool.
21
-
- These libraries are installed on top of the base runtime and pool level libraries, and take thr highest precedence.
22
-
- Session-scoped libraries do not persist across sessions.
20
+
- These libraries have no impact on other sessions or jobs using the same Spark pool.
21
+
- These libraries install on top of the base runtime and pool level libraries, and take the highest precedence.
22
+
- Session-scoped libraries don't persist across sessions.
23
23
24
24
## Session-scoped Python packages
25
25
@@ -29,18 +29,18 @@ To specify session-scoped Python packages:
29
29
30
30
1. Navigate to the selected Spark pool and ensure that you have enabled session-level libraries. You can enable this setting by navigating to the **Manage** > **Apache Spark pool** > **Packages** tab.
31
31
:::image type="content" source="./media/apache-spark-azure-portal-add-libraries/enable-session-packages.png" alt-text="Screenshot of enabling session packages." lightbox="./media/apache-spark-azure-portal-add-libraries/enable-session-packages.png":::
32
-
2. Once the setting has been applied, you can open a notebook and select **Configure Session**> **Packages**.
32
+
2. Once the setting applies, you can open a notebook and select **Configure Session**> **Packages**.
33
33

34
34

35
-
3. Here, you can upload a Conda *environment.yml* file to install or upgrade packages within a session. Once you start your session, the specified libraries will be installed. Once your session ends, these libraries will no longer be available as they are specific to your session.
35
+
3. Here, you can upload a Conda *environment.yml* file to install or upgrade packages within a session.The specified libraries are present once the session starts. These libraries will no longer be available after session ends.
36
36
37
37
### Manage session-scoped Python packages through *%pip* and *%conda* commands
38
38
39
-
You can leverage the popular *%pip* and *%conda* commands to install additional 3rd party libraries or your custom libraries during your Apache Spark notebook session. In this section, we will use *%pip* commands to demonstrate several common scenarios.
39
+
You can use the popular *%pip* and *%conda* commands to install additional third party libraries or your custom libraries during your Apache Spark notebook session. In this section, we use *%pip* commands to demonstrate several common scenarios.
40
40
41
-
### Install a third-party package
41
+
### Install a thirdparty package
42
42
43
-
You can easily install an Python library from [PyPI](https://pypi.org/).
43
+
You can easily install a Python library from [PyPI](https://pypi.org/).
44
44
45
45
```python
46
46
# Install vega_datasets
@@ -92,7 +92,7 @@ You can use the following command to see what's the built-in version of certain
92
92
%pip show pandas
93
93
```
94
94
95
-
The result will be as following:
95
+
The result is as following log:
96
96
97
97
```markdown
98
98
Name: pandas
@@ -110,7 +110,7 @@ You can use the following command to switch *pandas* to another version, let's s
110
110
111
111
### Uninstall a session-scoped library
112
112
113
-
If you want to uninstall a package which was installed on this notebook session, you may refer to following commands. However, you cannot uninstall the built-in packages.
113
+
If you want to uninstall a package, which installed on this notebook session, you may refer to following commands. However, you cannot uninstall the built-in packages.
114
114
115
115
```python
116
116
%pip uninstall altair vega_datasets --yes
@@ -146,7 +146,7 @@ To specify session-scoped Java or Scala packages, you can use the ```%%configure
146
146
147
147
## Session-scoped R packages (Preview)
148
148
149
-
Azure Synapse Analytics pools include many popular R libraries out-of-the-box. You can also install additional 3rd party libraries during your Apache Spark notebook session.
149
+
Azure Synapse Analytics pools include many popular R libraries out-of-the-box. You can also install extra third party libraries during your Apache Spark notebook session.
150
150
151
151
> [!NOTE]
152
152
>
@@ -162,7 +162,7 @@ You can easily install an R library from [CRAN](https://cran.r-project.org/).
162
162
install.packages(c("nycflights13", "Lahman"))
163
163
```
164
164
165
-
You can also leverage CRAN snapshots as the repository to ensure that the same package version is downloaded each time.
165
+
You can also use CRAN snapshots as the repository to ensure to download the same package version each time.
You can use the ```detach``` function to remove a library from the namespace. These libraries will stay on disk until they are loaded again.
218
+
You can use the ```detach``` function to remove a library from the namespace. These libraries stay on disk until they're loaded again.
219
219
220
220
```r
221
221
# detach a library
222
222
223
223
detach("package: caesar")
224
224
```
225
225
226
-
To remove a session-scoped package from a notebook, use the ```remove.packages()``` command. This will not impact other sessions on the same cluster. Users cannot uninstall or remove libraries that are installed as part of the default Azure Synapse Analytics runtime.
226
+
To remove a session-scoped package from a notebook, use the ```remove.packages()``` command. This library change has no impact on other sessions on the same cluster. Users can't uninstall or remove built-in libraries of the default Azure Synapse Analytics runtime.
227
227
228
228
```r
229
229
remove.packages("caesar")
230
230
```
231
231
232
232
> [!NOTE]
233
-
> You cannot remove core packages like SparkR, SparklyR, or R.
233
+
> You can't remove core packages like SparkR, SparklyR, or R.
With spark_apply() in SparklyR, you can use any R package inside Spark. By default, in sparklyr::spark_apply(), the packages argument is set to FALSE. This copies libraries in the current libPaths to the workers, allowing you to import and use them on workers. For example, you can run the following to generate a caesar-encrypted message with sparklyr::spark_apply():
259
+
With spark_apply() in SparklyR, you can use any R package inside Spark. By default, in sparklyr::spark_apply(), the packages argument sets to FALSE. This copies libraries in the current libPaths to the workers, allowing you to import and use them on workers. For example, you can run the following to generate a caesar-encrypted message with sparklyr::spark_apply():
0 commit comments