Merge pull request #90681 from cutecycle/patch-1

Clarity regarding dotnet Spark Job definition dependencies for Synapse

Author: ShannonLeavitt

articles/synapse-analytics/spark/media/apache-spark-job-definitions/net-spark-workspace-compatibility.png

@@ -23,18 +23,43 @@ You can analyze data with .NET for Apache Spark through Spark batch job definiti
 
 Visit the tutorial to learn how to use Azure Synapse Analytics to [create Apache Spark job definitions for Synapse Spark pools](apache-spark-job-definitions.md). If you haven't packaged your app to submit to Azure Synapse, complete the following steps.
 
-1. Run the following commands to publish your app. Be sure to replace *mySparkApp* with the path to your app.
+1. Configure your `dotnet` application dependencies for compatibility with Synapse Spark.
+The required .NET Spark version will be noted in the Synapse Studio interface under your Apache Spark Pool configuration, under the Manage toolbox.
+
+   :::image type="content" source="./media/apache-spark-job-definitions/net-spark-workspace-compatibility.png" alt-text="Screenshot that shows properties, including the .NET Spark version.":::
+
+   Create your project as a .NET console application that outputs an Ubuntu x86 executable.
+  
+   ```
+   <Project Sdk="Microsoft.NET.Sdk">
+ 
+     <PropertyGroup>
+       <OutputType>Exe</OutputType>
+       <TargetFramework>netcoreapp3.1</TargetFramework>
+     </PropertyGroup>
+ 
+     <ItemGroup>
+       <PackageReference Include="Microsoft.Spark" Version="2.1.0" />
+     </ItemGroup>
+ 
+   </Project>
+   ```
+
+2. Run the following commands to publish your app. Be sure to replace *mySparkApp* with the path to your app.
 
    ```dotnetcli
    cd mySparkApp
    dotnet publish -c Release -f netcoreapp3.1 -r ubuntu.18.04-x64
    ```
 
-2. Zip the contents of the publish folder, `publish.zip` for example, that was created as a result of Step 1. All the assemblies should be in the first layer of the ZIP file and there should be no intermediate folder layer. This means when you unzip `publish.zip`, all assemblies are extracted into your current working directory.
+3. Zip the contents of the publish folder, `publish.zip` for example, that was created as a result of Step 1. All the assemblies should be in the root of the ZIP file and there should be no intermediate folder layer. This means when you unzip `publish.zip`, all assemblies are extracted into your current working directory. 
 
     **On Windows:**
 
-    Use an extraction program, like [7-Zip](https://www.7-zip.org/) or [WinZip](https://www.winzip.com/), to extract the file into the bin directory with all the published binaries.
+    Using Windows PowerShell or PowerShell 7, create a .zip from the contents of your publish directory.
+    ```PowerShell
+    Compress-Archive publish/* publish.zip -Update
+    ```
 
     **On Linux:**
 
@@ -48,7 +73,7 @@ Visit the tutorial to learn how to use Azure Synapse Analytics to [create Apache
 
 Notebooks are a great option for prototyping your .NET for Apache Spark pipelines and scenarios. You can start working with, understanding, filtering, displaying, and visualizing your data quickly and efficiently. 
 
-Data engineers, data scientists, business analysts, and machine learning engineers are all able to collaborate over a shared,  interactive document. You see immediate results from data exploration, and can visualize your data in the same notebook.
+Data engineers, data scientists, business analysts, and machine learning engineers are all able to collaborate over a shared, interactive document. You see immediate results from data exploration, and can visualize your data in the same notebook.
 
 ### How to use .NET for Apache Spark notebooks
 
@@ -79,6 +104,12 @@ The following features are available when you use .NET for Apache Spark in the A
 * Support for defining [.NET user-defined functions that can run within Apache Spark](/dotnet/spark/how-to-guides/udf-guide). We recommend [Write and call UDFs in .NET for Apache Spark Interactive environments](/dotnet/spark/how-to-guides/dotnet-interactive-udf-issue) for learning how to use UDFs in .NET for Apache Spark Interactive experiences.
 * Support for visualizing output from your Spark jobs using different charts (such as line, bar, or histogram) and layouts (such as single, overlaid, and so on) using the `XPlot.Plotly` library.
 * Ability to include NuGet packages into your C# notebook.
+## Troubleshooting
+
+### `DotNetRunner: null` / `Futures timeout` in Synapse Spark Job Definition Run
+Synapse Spark Job Definitions on Spark Pools using Spark 2.4 require `Microsoft.Spark` 1.0.0. Clear your `bin` and `obj` directories, and publish the project using 1.0.0.
+### OutOfMemoryError: java heap space at org.apache.spark...
+Dotnet Spark 1.0.0 uses a different debug architecture than 1.1.1+. You will have to use 1.0.0 for your published version and 1.1.1+ for local debugging.
 
 ## Next steps