Skip to content

Commit 4660733

Browse files
committed
Add Spark Table Interoperability tutorial.
1 parent b36b0bc commit 4660733

File tree

1 file changed

+72
-0
lines changed

1 file changed

+72
-0
lines changed
Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"[![AWS Data Wrangler](_static/logo.png \"AWS Data Wrangler\")](https://github.com/awslabs/aws-data-wrangler)\n",
8+
"\n",
9+
"# 20 - Spark Table Interoperability\n",
10+
"\n",
11+
"[Wrangler](https://github.com/awslabs/aws-data-wrangler) has no difficults to insert, overwrite or do any other kind of interaction with a Table created by Apache Spark.\n",
12+
"\n",
13+
"But if you want to do the oposite (Spark interacting with a table created by Wrangler) you should be aware that Wrangler follows the Hive's format and you must be explicit when using the Spark's `saveAsTable` method:"
14+
]
15+
},
16+
{
17+
"cell_type": "code",
18+
"execution_count": null,
19+
"metadata": {},
20+
"outputs": [],
21+
"source": [
22+
"spark_df.write.format(\"hive\").saveAsTable(\"database.table\")"
23+
]
24+
},
25+
{
26+
"cell_type": "markdown",
27+
"metadata": {},
28+
"source": [
29+
"Or just move forward using the `insertInto` alternative:"
30+
]
31+
},
32+
{
33+
"cell_type": "code",
34+
"execution_count": null,
35+
"metadata": {},
36+
"outputs": [],
37+
"source": [
38+
"spark_df.write.insertInto(\"database.table\")"
39+
]
40+
}
41+
],
42+
"metadata": {
43+
"kernelspec": {
44+
"display_name": "Python 3",
45+
"language": "python",
46+
"name": "python3"
47+
},
48+
"language_info": {
49+
"codemirror_mode": {
50+
"name": "ipython",
51+
"version": 3
52+
},
53+
"file_extension": ".py",
54+
"mimetype": "text/x-python",
55+
"name": "python",
56+
"nbconvert_exporter": "python",
57+
"pygments_lexer": "ipython3",
58+
"version": "3.6.10"
59+
},
60+
"pycharm": {
61+
"stem_cell": {
62+
"cell_type": "raw",
63+
"metadata": {
64+
"collapsed": false
65+
},
66+
"source": []
67+
}
68+
}
69+
},
70+
"nbformat": 4,
71+
"nbformat_minor": 4
72+
}

0 commit comments

Comments
 (0)