You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Azure Synapse Analytics allows the different workspace computational engines to share databases and Parquet-backed tables between its Apache Spark pools (preview), SQL on-demand (preview) engine, and SQL pools.
18
+
Azure Synapse Analytics allows the different workspace computational engines to share databases and Parquet-backed tables between its Apache Spark pools (preview) and SQL on-demand (preview) engine.
19
19
20
20
Once a database has been created by a Spark job, you can create tables in it with Spark that use Parquet as the storage format. These tables will immediately become available for querying by any of the Azure Synapse workspace Spark pools. They can also be used from any of the Spark jobs subject to permissions.
21
21
22
-
The Spark created, managed, and external tables are also made available as external tables with the same name in the corresponding synchronized database in SQL on-demand and in the corresponding `$`-prefixed schemas in the SQL pools that have their metadata synchronization enabled. [Exposing a Spark table in SQL](#exposing-a-spark-table-in-sql) provides more detail on the table synchronization.
22
+
The Spark created, managed, and external tables are also made available as external tables with the same name in the corresponding synchronized database in SQL on-demand. [Exposing a Spark table in SQL](#exposing-a-spark-table-in-sql) provides more detail on the table synchronization.
23
23
24
-
Since the tables are synchronized to SQL on-demand and the SQL pools asynchronously, there will be a delay until they appear.
25
-
26
-
Mapping of tables to external tables, data sources and file formats.
24
+
Since the tables are synchronized to SQL on-demand asynchronously, there will be a delay until they appear.
27
25
28
26
## Manage a Spark created table
29
27
30
28
Use Spark to manage Spark created databases. For example, delete it through a Spark pool job, and create tables in it from Spark.
31
29
32
30
If you create objects in such a database from SQL on-demand or try to drop the database, the operation will succeed, but the original Spark database will not be changed.
33
31
34
-
If you try to drop the synchronized schema in a SQL pool, or try to create a table in it, Azure returns an error.
35
-
36
32
## Exposing a Spark table in SQL
37
33
38
34
### Which Spark tables are shared
@@ -51,7 +47,7 @@ Azure Synapse currently only shares managed and external Spark tables that store
51
47
52
48
### How are Spark tables shared
53
49
54
-
The shareable managed and external Spark tables exposed in the SQL engines as external tables with the following properties:
50
+
The shareable managed and external Spark tables exposed in the SQL engine as external tables with the following properties:
55
51
56
52
- The SQL external table's data source is the data source representing the Spark table's location folder.
57
53
- The SQL external table's file format is Parquet.
@@ -83,7 +79,7 @@ Spark tables provide different data types than the Synapse SQL engines. The foll
83
79
84
80
## Security model
85
81
86
-
The Spark databases and tables, as well as their synchronized representations in the SQL engines will be secured at the underlying storage level. Since they do not currently have permissions on the objects themselves, the objects can be seen in the object explorer.
82
+
The Spark databases and tables, as well as their synchronized representations in the SQL engine will be secured at the underlying storage level. Since they do not currently have permissions on the objects themselves, the objects can be seen in the object explorer.
87
83
88
84
The security principal who creates a managed table is considered the owner of that table and has all the rights to the table as well as the underlying folders and files. In addition, the owner of the database will automatically become co-owner of the table.
89
85
@@ -189,27 +185,6 @@ id | name | birthdate
189
185
1 | Alice | 2010-01-01
190
186
```
191
187
192
-
### Querying Spark tables in a SQL pool
193
-
194
-
With the tables created in the previous examples, now create a SQL pool in your workspace named `mysqlpool` that enables metadata synchronization (or use the already created pool from [Exposing a Spark database in a SQL pool](database.md#exposing-a-spark-database-in-a-sql-pool).
195
-
196
-
Run the following statement against the `mysqlpool` SQL pool:
197
-
198
-
```sql
199
-
SELECT*FROMsys.tables;
200
-
```
201
-
202
-
Verify that the tables `myParquetTable` and `myExternalParquetTable` are visible in the schema `$mytestdb`.
203
-
204
-
Now you can read the data from SQL on-demand as follows:
205
-
206
-
```sql
207
-
SELECT*FROM [$mytestdb].myParquetTable WHERE name ='Alice';
208
-
SELECT*FROM [$mytestdb].myExternalParquetTable WHERE name ='Alice';
209
-
```
210
-
211
-
You should get the same results as with SQL on-demand above.
212
-
213
188
## Next steps
214
189
215
190
-[Learn more about Azure Synapse Analytics' shared metadata](overview.md)
0 commit comments