Merge pull request #212410 from midesa/main

core r docs

Author: v-dirichards

articles/synapse-analytics/spark/apache-spark-3-runtime.md

@@ -38,7 +38,7 @@ When a Spark instance starts, these libraries are included automatically. More p
 
 ## Workspace packages
 
-When your team develops custom applications or models, you might develop various code artifacts like *.whl* or *.jar* files to package your code.
+When your team develops custom applications or models, you might develop various code artifacts like *.whl*, *.jar*, or *tar.gz* files to package your code.
 
 In Synapse, workspace packages can be custom or private *.whl* or *.jar* files. You can upload these packages to your workspace and later assign them to a specific serverless Apache Spark pool. Once assigned, these workspace packages are installed automatically on all Spark pool sessions.
 
@@ -70,8 +70,11 @@ Session-scoped packages allow users to define package dependencies at the start
 
 To learn more about how to manage session-scoped packages, see the following articles:
 
-- [Python session packages: ](./apache-spark-manage-session-packages.md#session-scoped-python-packages) At the start of a session, provide a Conda *environment.yml* to install more Python packages from popular repositories. 
-- [Scala/Java session packages: ](./apache-spark-manage-session-packages.md#session-scoped-java-or-scala-packages) At the start of your session, provide a list of *.jar* files to install using `%%configure`.
+- [Python session packages:](./apache-spark-manage-session-packages.md#session-scoped-python-packages) At the start of a session, provide a Conda *environment.yml* to install more Python packages from popular repositories.
+
+- [Scala/Java session packages:](./apache-spark-manage-session-packages.md#session-scoped-java-or-scala-packages) At the start of your session, provide a list of *.jar* files to install using `%%configure`.
+
+- [R session packages:](./apache-spark-manage-session-packages.md#session-scoped-r-packages-preview) Within your session, you can install packages across all nodes within your Spark pool using `install.packages` or `devtools`.
 
 ## Manage your packages outside Synapse Analytics UI
 
@@ -83,5 +86,6 @@ To learn more about Azure PowerShell cmdlets and package management REST APIs, s
 - Package management REST APIs: [Manage your Spark pool libraries through REST APIs](apache-spark-manage-packages-outside-ui.md#manage-packages-through-rest-apis)
 
 ## Next steps
+
 - View the default libraries: [Apache Spark version support](apache-spark-version-support.md)
 - Troubleshoot library installation errors: [Troubleshoot library errors](apache-spark-troubleshoot-library-errors.md)

articles/synapse-analytics/spark/apache-spark-32-runtime.md

@@ -1,6 +1,6 @@
 ---
-title: Visualizations
-description: Use Azure Synapse notebooks to visualize your data
+title: Python Visualizations
+description: Use Python and Azure Synapse notebooks to visualize your data
 author: midesa
 ms.author: midesa
 ms.reviewer: euang
@@ -11,43 +11,44 @@ ms.subservice: spark
 ms.date: 09/13/2020
 ---
 # Visualize data
-Azure Synapse is an integrated analytics service that accelerates time to insight, across data warehouses and big data analytics systems. Data visualization is a key component in being able to gain insight into your data. It helps make big and small data easier for humans to understand. It also makes it easier to detect patterns, trends, and outliers in groups of data. 
+
+Azure Synapse is an integrated analytics service that accelerates time to insight, across data warehouses and big data analytics systems. Data visualization is a key component in being able to gain insight into your data. It helps make big and small data easier for humans to understand. It also makes it easier to detect patterns, trends, and outliers in groups of data.
 
 When using Apache Spark in Azure Synapse Analytics, there are various built-in options to help you visualize your data, including Synapse notebook chart options, access to popular open-source libraries, and integration with Synapse SQL and Power BI.
 
 ## Notebook chart options
-When using an Azure Synapse notebook, you can turn your tabular results view into a customized chart using chart options. Here, you can visualize your data without having to write any code. 
+
+When using an Azure Synapse notebook, you can turn your tabular results view into a customized chart using chart options. Here, you can visualize your data without having to write any code.
 
 ### display(df) function
-The ```display``` function allows you to turn SQL queries and Apache Spark dataframes and RDDs into rich data visualizations.The ```display``` function can be used on dataframes or RDDs created in PySpark, Scala, Java, and .NET.
+
+The ```display``` function allows you to turn SQL queries and Apache Spark dataframes and RDDs into rich data visualizations. The ```display``` function can be used on dataframes or RDDs created in PySpark, Scala, Java, R, and .NET.
 
 To access the chart options:
-1. The output of ```%%sql``` magic commands appear in the rendered table view by default. You can also call ```display(df)``` on Spark DataFrames or Resilient Distributed Datasets (RDD) function to produce the rendered table view. 
-   
+
+1. The output of ```%%sql``` magic commands appear in the rendered table view by default. You can also call ```display(df)``` on Spark DataFrames or Resilient Distributed Datasets (RDD) function to produce the rendered table view.
 2. Once you have a rendered table view, switch to the Chart View.
    ![built-in-charts](./media/apache-spark-development-using-notebooks/synapse-built-in-charts.png#lightbox)
-
 3. You can now customize your visualization by specifying the following values:
 
    | Configuration | Description |
-   |--|--| 
+   |--|--|
    | Chart Type | The ```display``` function supports a wide range of chart types, including bar charts, scatter plots, line graphs, and more |
    | Key | Specify the range of values for the x-axis|
    | Value | Specify the range of values for the y-axis values |
-   | Series Group | Used to determine the groups for the aggregation | 
-   | Aggregation | Method to aggregate data in your visualization| 
-   
-   
+   | Series Group | Used to determine the groups for the aggregation |
+   | Aggregation | Method to aggregate data in your visualization|
     > [!NOTE]
     > By default the ```display(df)``` function will only take the first 1000 rows of the data to render the charts. Check the **Aggregation over all results** and click the **Apply** button, you will apply the chart generation from the whole dataset. A Spark job will be triggered when the chart setting changes. Please note that it may take several minutes to complete the calculation and render the chart.
-   
 4. Once done, you can view and interact with your final visualization!
 
 ### display(df) statistic details
+
 You can use <code>display(df, summary = true)</code> to check the statistics summary of a given Apache Spark DataFrame that include the column name, column type, unique values, and missing values for each column. You can also select on specific column to see its minimum value, maximum value, mean value and standard deviation.
     ![built-in-charts-summary](./media/apache-spark-development-using-notebooks/synapse-built-in-charts-summary.png#lightbox)
-   
+
 ### displayHTML() option
+
 Azure Synapse Analytics notebooks support HTML graphics using the ```displayHTML``` function.
 
 The following image is an example of creating visualizations using [D3.js](https://d3js.org/).
@@ -140,10 +141,12 @@ svg
 
 ```
 
-## Popular Libraries
-When it comes to data visualization, Python offers multiple graphing libraries that come packed with many different features. By default, every Apache Spark Pool in Azure Synapse Analytics contains a set of curated and popular open-source libraries. You can also add or manage additional libraries & versions by using the Azure Synapse Analytics library management capabilities. 
+## Python Libraries
+
+When it comes to data visualization, Python offers multiple graphing libraries that come packed with many different features. By default, every Apache Spark Pool in Azure Synapse Analytics contains a set of curated and popular open-source libraries. You can also add or manage additional libraries & versions by using the Azure Synapse Analytics library management capabilities.
 
 ### Matplotlib
+
 You can render standard plotting libraries, like Matplotlib, using the built-in rendering functions for each library.
 
 The following image is an example of creating a bar chart using **Matplotlib**.
@@ -173,9 +176,9 @@ plt.legend()
 plt.show()
 ```
 
-
 ### Bokeh
-You can render HTML or interactive libraries, like **bokeh**, using the ```displayHTML(df)```. 
+
+You can render HTML or interactive libraries, like **bokeh**, using the ```displayHTML(df)```.
 
 The following image is an example of plotting glyphs over a map using **bokeh**.
 
@@ -212,15 +215,14 @@ html = file_html(p, CDN, "my plot1")
 displayHTML(html)
 ```
 
-
 ### Plotly
+
 You can render HTML or interactive libraries like **Plotly**, using the **displayHTML()**.
 
 Run the following sample code to draw the image below.
 
    ![plotly-example](./media/apache-spark-development-using-notebooks/synapse-plotly-image.png#lightbox)
 
-
 ```python
 from urllib.request import urlopen
 import json
@@ -248,9 +250,10 @@ h = plotly.offline.plot(fig, output_type='div')
 # display this html
 displayHTML(h)
 ```
+
 ### Pandas
 
-You can view html output of pandas dataframe as the default output, notebook will automatically show the styled html content. 
+You can view html output of pandas dataframe as the default output, notebook will automatically show the styled html content.
 
    ![Panda graph example.](./media/apache-spark-data-viz/support-panda.png#lightbox)
 
@@ -267,20 +270,116 @@ df = pd.DataFrame([[38.0, 2.0, 18.0, 22.0, 21, np.nan],[19, 439, 6, 452, 226,232
 df 
 ```
 
+### Additional libraries
 
-### Additional libraries 
 Beyond these libraries, the Azure Synapse Analytics Runtime also includes the following set of libraries that are often used for data visualization:
 
-- [Seaborn](https://seaborn.pydata.org/) 
+- [Seaborn](https://seaborn.pydata.org/)
 
 You can visit the Azure Synapse Analytics Runtime [documentation](./spark/../apache-spark-version-support.md) for the most up to date information about the available libraries and versions.
 
+## R Libraries (Preview)
+
+The R ecosystem offers multiple graphing libraries that come packed with many different features. By default, every Apache Spark Pool in Azure Synapse Analytics contains a set of curated and popular open-source libraries. You can also add or manage additional libraries & versions by using the Azure Synapse Analytics library management capabilities.
+
+### ggplot2
+
+The [ggplot2](https://ggplot2.tidyverse.org/) library is popular for data visualization and exploratory data analysis.
+
+   ![Screenshot of a ggplot2 graph example.](./media/apache-spark-data-viz/ggplot2.png#lightbox)
+
+```r
+library(ggplot2)
+data(mpg, package="ggplot2") 
+theme_set(theme_bw()) 
+
+g <- ggplot(mpg, aes(cty, hwy))
+
+# Scatterplot
+g + geom_point() + 
+  geom_smooth(method="lm", se=F) +
+  labs(subtitle="mpg: city vs highway mileage", 
+       y="hwy", 
+       x="cty", 
+       title="Scatterplot with overlapping points", 
+       caption="Source: midwest")
+```
+
+### rBokeh
+
+[rBokeh](https://hafen.github.io/rbokeh/) is a native R plotting library for creating interactive graphics which are backed by the Bokeh visualization library.
+
+To install rBokeh, you can use the following command:
+
+```r
+install.packages("rbokeh")
+```
+
+Once installed, you can leverage rBokeh to create interactive visualizations.
+
+   ![Screenshot of a rBokeh graph example.](./media/apache-spark-data-viz/bokeh-plot.png#lightbox)
+
+```r
+library(rbokeh)
+p <- figure() %>%
+  ly_points(Sepal.Length, Sepal.Width, data = iris,
+    color = Species, glyph = Species,
+    hover = list(Sepal.Length, Sepal.Width))
+```
+
+### R Plotly
+
+[Plotly's](https://plotly.com/r/) R graphing library makes interactive, publication-quality graphs.
+
+To install Plotly, you can use the following command:
+
+```r
+install.packages("plotly")
+```
+
+Once installed, you can leverage Plotly to create interactive visualizations.
+
+   ![Screenshot of a Plotly graph example.](./media/apache-spark-data-viz/plotly-r.png#lightbox)
+
+```r
+library(plotly) 
+
+fig <- plot_ly() %>% 
+  add_lines(x = c("a","b","c"), y = c(1,3,2))%>% 
+  layout(title="sample figure", xaxis = list(title = 'x'), yaxis = list(title = 'y'), plot_bgcolor = "#c7daec") 
+
+fig
+```
+
+### Highcharter
+
+[Highcharter](https://jkunst.com/highcharter/) is a R wrapper for Highcharts Javascript library and its modules.
+
+To install Highcharter, you can use the following command:
+
+```r
+install.packages("highcharter")
+```
+
+Once installed, you can leverage Highcharter to create interactive visualizations.
+
+   ![Screenshot of a Highcharter graph example.](./media/apache-spark-data-viz/highcharter.png#lightbox)
+
+```r
+library(magrittr)
+library(highcharter)
+hchart(mtcars, "scatter", hcaes(wt, mpg, z = drat, color = hp)) %>%
+  hc_title(text = "Scatter chart with size and color")
+
+```
+
 ## Connect to Power BI using Apache Spark & SQL On-Demand
+
 Azure Synapse Analytics integrates deeply with Power BI allowing data engineers to build analytics solutions.
 
 Azure Synapse Analytics allows the different workspace computational engines to share databases and tables between its Spark pools and serverless SQL pool. Using the [shared metadata model](../metadata/overview.md),you can query your Apache Spark tables using SQL on-demand. Once done, you can connect your SQL on-demand endpoint to Power BI to easily query your synced Spark tables.
 
-
 ## Next steps
+
 - For more information on how to set up the Spark SQL DW Connector: [Synapse SQL connector](./spark/../synapse-spark-sql-pool-import-export.md)
-- View the default libraries: [Azure Synapse Analytics runtime](../spark/apache-spark-version-support.md)
+- View the default libraries: [Azure Synapse Analytics runtime](../spark/apache-spark-version-support.md)

articles/synapse-analytics/spark/apache-spark-azure-portal-add-libraries.md

@@ -9,7 +9,7 @@ ms.topic: overview
 ms.subservice: spark
 ms.date: 02/15/2022
 ms.custom: devx-track-csharp
-zone_pivot_groups: programming-languages-spark-all-minus-sql
+zone_pivot_groups: programming-languages-spark-all-minus-sql-r
 ---
 
 # Linux Foundation Delta Lake overview
@@ -160,7 +160,7 @@ The order of the results is different from above as there was no order explicitl
 
 ## Update table data
 
-Delta Lake supports several operations to modify tables using standard DataFrame APIs, this is one of the big enhancements that delta format adds. The following example runs a batch job to overwrite the data in the table.
+Delta Lake supports several operations to modify tables using standard DataFrame APIs. These operations are one of the enhancements that delta format adds. The following example runs a batch job to overwrite the data in the table.
 
 :::zone pivot = "programming-language-python"
 
@@ -344,7 +344,7 @@ Results in:
 
 ## Conditional update without overwrite
 
-Delta Lake provides programmatic APIs to conditional update, delete, and merge (this is commonly referred to as an upsert) data into tables.
+Delta Lake provides programmatic APIs to conditional update, delete, and merge (this command is commonly referred to as an upsert) data into tables.
 
 :::zone pivot = "programming-language-python"
 
@@ -530,7 +530,7 @@ Here you have a combination of the existing data. The existing data has been ass
 
 ### History
 
-Delta Lake's has the ability to allow looking into history of a table. That is, the changes that were made to the underlying Delta Table. The cell below shows how simple it is to inspect the history.
+Delta Lake's has the ability to allow looking into history of a table. That is, the changes that were made to the underlying Delta Table. The cell below shows how simple it's to inspect the history.
 
 :::zone pivot = "programming-language-python"
 
@@ -572,7 +572,7 @@ Here you can see all of the modifications made over the above code snippets.
 
 It's possible to query previous snapshots of your Delta Lake table by using a feature called Time Travel. If you want to access the data that you overwrote, you can query a snapshot of the table before you overwrote the first set of data using the versionAsOf option.
 
-Once you run the cell below, you should see the first set of data from before you overwrote it. Time Travel is an extremely powerful feature that takes advantage of the power of the Delta Lake transaction log to access data that is no longer in the table. Removing the version 0 option (or specifying version 1) would let you see the newer data again. For more information, see [Query an older snapshot of a table](https://docs.delta.io/latest/delta-batch.html#deltatimetravel).
+Once you run the cell below, you should see the first set of data from before you overwrote it. Time Travel is a powerful feature that takes advantage of the power of the Delta Lake transaction log to access data that is no longer in the table. Removing the version 0 option (or specifying version 1) would let you see the newer data again. For more information, see [Query an older snapshot of a table](https://docs.delta.io/latest/delta-batch.html#deltatimetravel).
 
 :::zone pivot = "programming-language-python"
 
@@ -611,7 +611,7 @@ Results in:
 |  3|
 |  2|
 
-Here you can see you have gone back to the earliest version of the data.
+Here you can see you've gone back to the earliest version of the data.
 
 ## Write a stream of data to a table
 
@@ -626,7 +626,7 @@ In the cells below, here's what we are doing:
 * Cell 32 Stop the structured streaming job
 * Cell 33 Inspect history <--You'll notice appends have stopped
 
-First you are going to set up a simple Spark Streaming job to generate a sequence and make the job write to your Delta Table.
+First you're going to set up a simple Spark Streaming job to generate a sequence and make the job write to your Delta Table.
 
 :::zone pivot = "programming-language-python"
 
@@ -748,7 +748,7 @@ Results in:
 |      1|2020-04-25 00:35:05|           WRITE|                                               [mode -> Overwrite, partitionBy -> []]|          0|
 |      0|2020-04-25 00:34:34|           WRITE|                                           [mode -> ErrorIfExists, partitionBy -> []]|       null|
 
-Here you are dropping some of the less interesting columns to simplify the viewing experience of the history view.
+Here you're dropping some of the less interesting columns to simplify the viewing experience of the history view.
 
 :::zone pivot = "programming-language-python"
 
@@ -792,7 +792,7 @@ Results in:
 
 You can do an in-place conversion from the Parquet format to Delta.
 
-Here you are going to test if the existing table is in delta format or not.
+Here you're going to test if the existing table is in delta format or not.
 :::zone pivot = "programming-language-python"
 
 ```python
@@ -830,7 +830,7 @@ Results in:
 
 False
 
-Now you are going to convert the data to delta format and verify it worked.
+Now you're going to convert the data to delta format and verify it worked.
 
 :::zone pivot = "programming-language-python"
 
@@ -936,7 +936,7 @@ Results in:
 |--------------------|
 |abfss://data@arca...|
 
-Now you are going to verify that a table is not a delta format table, then convert it to delta format using Spark SQL and confirm it was converted correctly.
+Now, you're going to verify that a table is not a delta format table. Then, you will convert the table to delta format using Spark SQL and confirm that it was converted correctly.
 
 :::zone pivot = "programming-language-python"