You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Use Spark to manage Spark created databases. For example, delete it through a Spark pool job, and create tables in it from Spark.
35
33
36
34
If you create objects in a Spark created database using SQL on-demand, or try to drop the database, the operation will succeed. But, the original Spark database won't be changed.
@@ -39,8 +37,6 @@ If you try to drop the synchronized schema in a SQL pool, or try to create a tab
If the name of a Spark database conflicts with the name of an existing SQL on-demand database, a suffix is appended in SQL on-demand to the Spark database. The suffix in SQL on-demand is `_<workspace name>-ondemand-DefaultSparkConnector`.
45
41
46
42
For example, if a Spark database called `mydb` gets created in the Azure Synapse workspace `myws` and a SQL on-demand database with that name already exists, then the Spark database in SQL on-demand will have to be referenced using the name `mydb_myws-ondemand-DefaultSparkConnector`.
@@ -50,8 +46,6 @@ For example, if a Spark database called `mydb` gets created in the Azure Synapse
First create a new Spark database named `mytestdb` using a Spark cluster you have already created in your workspace. You can achieve that, for example, using a Spark C# Notebook with the following .NET for Spark statement:
72
64
73
65
```csharp
@@ -84,8 +76,6 @@ Verify that `mytestdb` is included in the results.
Spark allows you to create databases, external tables, managed tables, and views. Since Spark views require a Spark engine to process the defining Spark SQL statement, and cannot be processed by a SQL engine, only databases and their contained external and managed tables that use the Parquet storage format are shared with the workspace SQL engines. Spark views are only shared among the Spark pool instances.
The Spark databases and tables, along with their synchronized representations in the SQL engines, are secured at the underlying storage level. When the table is queried by any of the engines that the query submitter has the right to use, the query submitter's security principal is being passed through to the underlying files. Permissions are checked at the file system level.
53
49
54
50
For more information, see [Azure Synapse Analytics shared database](database.md).
Use Spark to manage Spark created databases. For example, delete it through a Spark pool job, and create tables in it from Spark.
33
31
34
32
If you create objects in such a database from SQL on-demand or try to drop the database, the operation will succeed, but the original Spark database will not be changed.
@@ -37,8 +35,6 @@ If you try to drop the synchronized schema in a SQL pool, or try to create a tab
The Spark databases and tables, as well as their synchronized representations in the SQL engines will be secured at the underlying storage level. Since they do not currently have permissions on the objects themselves, the objects can be seen in the object explorer.
93
87
94
88
The security principal who creates a managed table is considered the owner of that table and has all the rights to the table as well as the underlying folders and files. In addition, the owner of the database will automatically become co-owner of the table.
@@ -101,8 +95,6 @@ For more information on how to set permissions on the folders and files, see [Az
101
95
102
96
### Create a managed table backed by Parquet in Spark and query from SQL on-demand
In this scenario, you have a Spark database named `mytestdb`. See [Create & connect to Spark database - SQL on-demand](database.md#create--connect-to-spark-database---sql-on-demand).
107
99
108
100
Create a managed Spark table with SparkSQL by running the following command:
@@ -162,8 +154,6 @@ id | name | birthdate
162
154
163
155
### Creating an external table backed by Parquet in Spark and querying it from SQL on-demand
With the tables created in the previous examples, now create a SQL pool in your workspace named `mysqlpool` that enables metadata synchronization (or use the already created pool from [Exposing a Spark database in a SQL pool](database.md#exposing-a-spark-database-in-a-sql-pool).
207
195
208
196
Run the following statement against the `mysqlpool` SQL pool:
Check to see if the service is available. In the Azure portal, go to the Synapse SQL pool you're trying to connect. In the left TOC panel, click on **Diagnose and solve problems**.
> Azure Synapse Analytics (workspaces) is currently in public preview.
15
+
> This preview version is provided without a service level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.
16
+
> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
0 commit comments