You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/synapse-analytics/spark/spark-dotnet.md
+19-7Lines changed: 19 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,25 +10,35 @@ ms.author: mamccrea
10
10
ms.reviewer: jrasnick
11
11
---
12
12
13
-
# Use .NET for Apache Spark with Azure Synapse Analytics
13
+
<!--# Use .NET for Apache Spark with Azure Synapse Analytics
14
14
15
15
Azure Synapse Analytics uses Spark pools (preview) for data processing. Apache Spark is a general-purpose distributed processing engine for analytics over large data sets - typically terabytes or petabytes of data. You can use Apache Spark for several popular big data scenarios, including:
16
16
17
17
* Batch processing
18
18
* Machine Learning
19
-
* Impromptu querying
19
+
* Impromptu querying-->
20
20
21
-
##What is .NET for Apache Spark
21
+
# What is .NET for Apache Spark?
22
22
23
-
.NET for Apache Spark is free, open-source, and cross-platform .NET support for Spark. .NET for Apache Spark provides .NET bindings for Spark that allow you to access Spark APIs through C# and F#.
23
+
[.NET for Apache Spark](https://dot.net/spark) provides free, open-source, and cross-platform .NET support for Spark. .NET for Apache Spark provides .NET bindings for Spark that allow you to access Spark APIs through C# and F# and gives you the ability to write and execute user-defined functions for Spark using .NET.
24
24
25
25
The .NET APIs for Spark enable you to access all aspects of Spark that help you analyze your data, including Spark SQL and Structured Streaming.
26
26
27
27
## .NET for Apache Spark in Azure Synapse Analytics
28
28
29
-
You can analyze your data using .NET for Apache Spark through interactive Azure Synapse Analytics notebooks. When creating a new Azure Synapse Analytics notebook, you choose a language kernel that you wish to express your business logic. There is kernel support for several languages, including C#.
29
+
You can analyze your data using .NET for Apache Spark through either Spark batch job definitions or with interactive Azure Synapse Analytics notebooks.
30
30
31
-
To use .NET for Apache Spark in your Azure Synapse Analytics notebook, select **Spark.NET C#** as your kernel and attach the notebook to an existing Spark pool.
31
+
<!--
32
+
### .NET for Apache Spark in Azure Synapse batch job definitions
33
+
Jenny or someone please add details on the batch mode submission -->
34
+
35
+
### .NET for Apache Spark in Azure Synapse Analytics notebooks
36
+
37
+
When creating a new notebook, you choose a language kernel that you wish to express your business logic. There is kernel support for several languages, including C#.
38
+
39
+
To use .NET for Apache Spark in your Azure Synapse Analytics notebook, select **.NET Spark (C#)** as your kernel and attach the notebook to an existing Spark pool.
40
+
41
+
The .NET Spark notebook is based on the .NET interactive experiences and provides interactive C# experiences with the ability to use .NET for Spark out of the box (with the Spark session variable `spark` already predefined). For more details on the available notebook capabilities [see below](#sparknet-c-kernel-features).
32
42
33
43
## .NET for Apache Spark scenarios
34
44
@@ -49,9 +59,11 @@ The following features are available when you use .NET for Apache Spark in the A
49
59
* Support for [C# 8.0 language features](https://docs.microsoft.com/dotnet/csharp/whats-new/csharp-8).
50
60
* 'spark' as a pre-defined variable to give you access to your Apache Spark session.
51
61
* Support for defining [.NET user-defined functions that can run within Apache Spark](https://github.com/dotnet/spark/blob/master/examples/Microsoft.Spark.CSharp.Examples/Sql).
52
-
* Support for visualizing output from your Spark jobs using different charts (such as line, bar, or histogram) and layouts (such as single, overlaid, and so on).
62
+
* Support for visualizing output from your Spark jobs using different charts (such as line, bar, or histogram) and layouts (such as single, overlaid, and so on) using the `XPlot.Plotly` library.
63
+
* Ability to include NuGet packages into your C# notebook.
53
64
54
65
## Next steps
55
66
56
67
*[.NET for Apache Spark documentation](https://docs.microsoft.com/dotnet/spark)
0 commit comments