Skip to content

Commit 727c6c4

Browse files
author
Varun Rao
committed
Add support for EMR 6.1 and PrestoSQL plugin
1 parent d3084b7 commit 727c6c4

File tree

8 files changed

+232
-100
lines changed

8 files changed

+232
-100
lines changed

‎README.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,9 @@ It uses agents to sync policies and users, and plugins that run within the same
77

88
The repo contains code tied to [AWS Big Data Blog](https://aws.amazon.com/blogs/big-data/implementing-authorization-and-auditing-using-apache-ranger-on-amazon-emr/).
99

10-
> **Development Status** the code has gone through unit and functional test against a **few recent versions** of Amazon EMR.
10+
> **NOTE:** the code has gone through unit and functional test against a **few recent versions** of Amazon EMR.
1111
> It is likely that it may not work with **all** EMR versions.
12-
> Plugins marked as **beta** has not been tested in production.
12+
> Code/plugin marked as **beta** has not been suitable for production use.
1313
>> Please submit Pull Request or to create an [Issue](https://github.com/aws-samples/aws-emr-apache-ranger/issues/new)
1414
>
1515
### Deployment options:
@@ -21,9 +21,13 @@ The repo contains code tied to [AWS Big Data Blog](https://aws.amazon.com/blogs/
2121
### Compatibility/Supported plugins:
2222
| Module| Tag | Cloudformation stack | Apache Ranger Version | EMR Version | Supported Plugins|
2323
| -------| --- | --- | --- | --- |-------------------------------------------------------- |
24-
| V1 | [1.0](https://github.com/aws-samples/aws-emr-apache-ranger/tree/1.0) | [![Foo](images/launch_stack.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=EMRSecurityWithRangerBlogV1&templateURL=https://s3.amazonaws.com/aws-bigdata-blog/artifacts/aws-blog-emr-ranger/1.0/cloudformation/nestedstack.template) | Apache Ranger 1.0, 2.1 | emr-5.28.1, emr-5.29.0, emr-5.30.1| Hive 2.x, Hadoop 2.x, PrestoDB 0.227/0.232 (Presto plugin needs Ranger 2.0) |
24+
| V1 | [1.0](https://github.com/aws-samples/aws-emr-apache-ranger/tree/v1.0) | [![Foo](images/launch_stack.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=EMRSecurityWithRangerBlogV1&templateURL=https://s3.amazonaws.com/aws-bigdata-blog/artifacts/aws-blog-emr-ranger/1.0/cloudformation/nestedstack.template) | Apache Ranger 1.0, 2.1 | emr-5.28.1, emr-5.29.0, emr-5.30.1| Hive 2.x, Hadoop 2.x, PrestoDB 0.227/0.232 (Presto plugin needs Ranger 2.0) |
2525
| V1 | (work in progress) | [![Foo](images/launch_stack.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=EMRSecurityWithRangerBlogV1&templateURL=https://s3.amazonaws.com/aws-bigdata-blog/artifacts/aws-blog-emr-ranger/1.1/cloudformation/nestedstack.template) | Apache Ranger 2.2 | emr-6.1.0 | Hive 3.x, Hadoop 3.x, PrestoSQL 338 OR PrestoDB 0.232 |
2626

27+
> WARNING: The current V1 setup does not enable strong cluster level Auth (Kerberos) for EMR. Only LDAP enabled Hue UI. V2 will support Kerberos - refer to the [roadmap](https://github.com/aws-samples/aws-emr-apache-ranger/projects/1) for details.
28+
### PrestoSQL Ranger plugin (EMR 6.1 & Ranger 2.2)
29+
Shows how the plugin can be used to enable column level access controls, column masking and row filter. Demo uses the [Presto Redshift connector](https://prestosql.io/docs/current/connector/redshift.html). The same functionality should work with other [Presto connectors](https://prestosql.io/docs/current/connector.html).
30+
![](images/prestosql_ranger_plugin.gif)
2731
Please open Git Issues if you would like to see updates/other plugin integrations.
2832
### References:
2933

‎aws_emr_blog_v1/cloudformation/emr-template.template

Lines changed: 27 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -59,27 +59,40 @@ Parameters:
5959
- emr-5.28.0
6060
- emr-5.28.1
6161
- emr-5.29.0
62-
- emr-5.30.1
62+
- emr-5.30.0
63+
- emr-5.31.0
64+
- emr-5.32.0
65+
- emr-6.1.0
6366
Description: Release label for the EMR cluster
6467
Type: String
6568
rangerVersion:
6669
Default: '2.0'
6770
Description: Version of the Ranger Server.
6871
Type: String
6972
AllowedValues:
70-
- '0.6'
71-
- '0.7'
7273
- '1.0'
7374
- '2.0'
7475
s3artifactsRepo:
75-
Default: s3://aws-bigdata-blog/artifacts/aws-blog-emr-ranger/1.0
76+
Default: aws-bigdata-blog/artifacts/aws-blog-emr-ranger
7677
Description: Git Repo URL for this blog.
7778
Type: String
79+
s3artifactsRepoVersion:
80+
Default: 1.0
81+
Description: Current version of the code
82+
Type: String
83+
AllowedValues:
84+
- 1.0
85+
- 1.1
7886
InstallPrestoRangerPlugin:
7987
Description: Install the Ranger Presto Plugin. NOTE - This needs Ranger 2.0 and has not been tested with all versions
8088
Default: false
8189
Type: String
8290
AllowedValues: [true, false]
91+
PrestoEngine:
92+
Description: Presto Engine. PrestoSQL is only available with EMR 6.x
93+
Default: Presto
94+
Type: String
95+
AllowedValues: [Presto, PrestoSQL]
8396
# InstallSparkRangerPlugin:
8497
# Description: Install the Ranger Spark Plugin. NOTE - This needs Ranger 2.0 and has not been tested with all versions
8598
# Default: false
@@ -106,7 +119,7 @@ Conditions:
106119
- emr-5.29.0
107120
emr-5.30: !Equals
108121
- !Ref 'emrReleaseLabel'
109-
- emr-5.30.1
122+
- emr-5.30.0
110123
emr-6.0: !Equals
111124
- !Ref 'emrReleaseLabel'
112125
- emr-6.0.0
@@ -124,18 +137,15 @@ Resources:
124137
- Name: Hive
125138
- Name: Hadoop
126139
- Name: Hue
127-
- Name: Presto
140+
- Name: !Ref PrestoEngine
128141
- Name: Spark
129142
- Name: Livy
130143
BootstrapActions:
131144
- Name: Download scripts
132145
ScriptBootstrapAction:
133-
Path: !Join
134-
- ''
135-
- - !Ref 's3artifactsRepo'
136-
- /scripts/download-scripts.sh
146+
Path: !Join ['', ['s3://', !Ref s3artifactsRepo, '/', !Ref s3artifactsRepoVersion, '/scripts/download-scripts.sh']]
137147
Args:
138-
- !Ref 's3artifactsRepo'
148+
- !Join ['', ['s3://', !Ref s3artifactsRepo]]
139149
Configurations:
140150
- Classification: hue-ini
141151
Configurations:
@@ -281,7 +291,8 @@ Resources:
281291
- /mnt/tmp/aws-blog-emr-ranger/scripts/emr-steps/install-hive-hdfs-ranger-plugin.sh
282292
- !Ref 'RangerHostname'
283293
- !Ref 'rangerVersion'
284-
- !Ref 's3artifactsRepo'
294+
- !Join ['', ['s3://', !Ref s3artifactsRepo]]
295+
- !Ref 'emrReleaseLabel'
285296
Jar: s3://elasticmapreduce/libs/script-runner/script-runner.jar
286297
MainClass: ''
287298
JobFlowId: !Ref 'EMRSampleCluster'
@@ -294,10 +305,7 @@ Resources:
294305
Args:
295306
- /mnt/tmp/aws-blog-emr-ranger/scripts/emr-steps/install-hive-hdfs-ranger-policies.sh
296307
- !Ref 'RangerHostname'
297-
- !Join
298-
- ''
299-
- - !Ref 's3artifactsRepo'
300-
- /inputdata
308+
- !Join ['', ['s3://', !Ref s3artifactsRepo, '/', !Ref s3artifactsRepoVersion, '/inputdata']]
301309
Jar: s3://elasticmapreduce/libs/script-runner/script-runner.jar
302310
MainClass: ''
303311
JobFlowId: !Ref 'EMRSampleCluster'
@@ -311,8 +319,9 @@ Resources:
311319
- /mnt/tmp/aws-blog-emr-ranger/scripts/emr-steps/install-presto-ranger-plugin.sh
312320
- !Ref 'RangerHostname'
313321
- !Ref 'rangerVersion'
314-
- !Ref 's3artifactsRepo'
322+
- !Join ['', ['s3://', !Ref s3artifactsRepo]]
315323
- !Ref 'emrReleaseLabel'
324+
- !Ref 'PrestoEngine'
316325
Jar: s3://elasticmapreduce/libs/script-runner/script-runner.jar
317326
MainClass: ''
318327
JobFlowId: !Ref 'EMRSampleCluster'
@@ -326,10 +335,7 @@ Resources:
326335
Args:
327336
- /mnt/tmp/aws-blog-emr-ranger/scripts/emr-steps/install-presto-ranger-policies.sh
328337
- !Ref 'RangerHostname'
329-
- !Join
330-
- ''
331-
- - !Ref 's3artifactsRepo'
332-
- /inputdata
338+
- !Join ['', ['s3://', !Ref s3artifactsRepo, '/', !Ref s3artifactsRepoVersion, '/inputdata']]
333339
Jar: s3://elasticmapreduce/libs/script-runner/script-runner.jar
334340
MainClass: ''
335341
JobFlowId: !Ref 'EMRSampleCluster'

‎aws_emr_blog_v1/cloudformation/nestedstack.template

Lines changed: 29 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -80,17 +80,14 @@ Parameters:
8080
Type: String
8181
NoEcho: true
8282
rangerVersion:
83-
Description: 'RangerVersion. Expected values are : 0.6,0.7,1.0,2.0. NOTE: Use Ranger 0.6 if
84-
EMR version is 5.0'
83+
Description: 'RangerVersion. Expected values are : 1.0,2.0'
8584
AllowedValues:
86-
- '0.6'
87-
- '0.7'
8885
- '1.0'
8986
- '2.0'
9087
Type: String
9188
Default: '2.0'
9289
emrReleaseLabel:
93-
Description: EMR Version. Pick Ranger 0.6 if EMR version is 5.0 or higher
90+
Default: emr-5.29.0
9491
AllowedValues:
9592
- emr-5.0.0
9693
- emr-5.4.0
@@ -101,17 +98,28 @@ Parameters:
10198
- emr-5.28.0
10299
- emr-5.28.1
103100
- emr-5.29.0
104-
- emr-5.30.1
101+
- emr-5.30.0
102+
- emr-5.31.0
103+
- emr-5.32.0
104+
- emr-6.1.0
105+
Description: Release label for the EMR cluster
105106
Type: String
106-
Default: emr-5.29.0
107-
s3artifactsRepoHttp:
108-
Default: https://s3.amazonaws.com/aws-bigdata-blog/artifacts/aws-blog-emr-ranger/1.0
109-
Description: HTTP location of the repo.
107+
PrestoEngine:
108+
Description: Presto Engine. PrestoSQL is only available with EMR 6.x
109+
Default: Presto
110110
Type: String
111+
AllowedValues: [Presto, PrestoSQL]
111112
s3artifactsRepo:
112-
Default: s3://aws-bigdata-blog/artifacts/aws-blog-emr-ranger/1.0
113+
Default: aws-bigdata-blog/artifacts/aws-blog-emr-ranger
113114
Description: S3 location of the repo.
114115
Type: String
116+
s3artifactsRepoVersion:
117+
Default: 1.0
118+
Description: Project version
119+
Type: String
120+
AllowedValues:
121+
- 1.0
122+
- 1.1
115123
InstallPrestoRangerPlugin:
116124
Description: Install the Ranger Presto Plugin. NOTE - This needs Ranger 2.0 and has not been tested with all versions
117125
Default: false
@@ -123,8 +131,8 @@ Metadata:
123131
- Label:
124132
default: Artifacts repo
125133
Parameters:
126-
- s3artifactsRepoHttp
127134
- s3artifactsRepo
135+
- s3artifactsRepoVersion
128136
- Label:
129137
default: Network Configuration
130138
Parameters:
@@ -161,10 +169,7 @@ Resources:
161169
SimpleAD:
162170
Type: AWS::CloudFormation::Stack
163171
Properties:
164-
TemplateURL: !Join
165-
- ''
166-
- - !Ref 's3artifactsRepoHttp'
167-
- /cloudformation/simple-ad-template.template
172+
TemplateURL: !Join ['', ['https://s3.amazonaws.com/', !Ref 's3artifactsRepo', '/', !Ref 's3artifactsRepoVersion', '/cloudformation/', 'simple-ad-template.template']]
168173
Parameters:
169174
VPC: !Ref 'VPC'
170175
Subnet1SimpleAD: !Ref 'Subnet1SimpleAD'
@@ -176,10 +181,7 @@ Resources:
176181
DependsOn:
177182
- SimpleAD
178183
Properties:
179-
TemplateURL: !Join
180-
- ''
181-
- - !Ref 's3artifactsRepoHttp'
182-
- /cloudformation/ranger-server.template
184+
TemplateURL: !Join ['', ['https://s3.amazonaws.com/', !Ref 's3artifactsRepo', '/', !Ref 's3artifactsRepoVersion', '/cloudformation/', 'ranger-server.template']]
183185
Parameters:
184186
VPC: !Ref 'VPC'
185187
VPCCidrBlock: !Ref 'VPCCidrBlock'
@@ -190,20 +192,19 @@ Resources:
190192
myDirectoryBaseDN: !Ref 'myDirectoryBaseDN'
191193
myDirectoryBindUser: !Ref 'myDirectoryBindUser'
192194
rangerVersion: !Ref 'rangerVersion'
193-
s3artifactsRepoHttp: !Ref 's3artifactsRepoHttp'
195+
s3artifactsRepo: !Ref s3artifactsRepo
196+
s3artifactsRepoVersion: !Ref s3artifactsRepoVersion
194197
myDirectoryAdminPassword: !Ref 'myDirectoryAdminPassword'
195198
myDirectoryBindPassword: !Ref 'myDirectoryBindPassword'
196199
myDirectoryDefaultUserPassword: !Ref 'myDirectoryDefaultUserPassword'
200+
emrReleaseLabel: !Ref 'emrReleaseLabel'
197201
TimeoutInMinutes: '60'
198202
EMRCluster:
199203
Type: AWS::CloudFormation::Stack
200204
DependsOn:
201205
- RangerServer
202206
Properties:
203-
TemplateURL: !Join
204-
- ''
205-
- - !Ref 's3artifactsRepoHttp'
206-
- /cloudformation/emr-template.template
207+
TemplateURL: !Join ['', ['https://s3.amazonaws.com/', !Ref 's3artifactsRepo', '/', !Ref 's3artifactsRepoVersion', '/cloudformation/', 'emr-template.template']]
207208
Parameters:
208209
myDirectoryBindUser: !Ref 'myDirectoryBindUser'
209210
myDirectoryBindPassword: !Ref 'myDirectoryBindPassword'
@@ -219,8 +220,10 @@ Resources:
219220
VPC: !Ref 'VPC'
220221
KeyName: !Ref 'KeyName'
221222
Subnet: !Ref 'DefaultSubnet'
222-
s3artifactsRepo: !Ref 's3artifactsRepo'
223+
s3artifactsRepo: !Ref s3artifactsRepo
224+
s3artifactsRepoVersion: !Ref s3artifactsRepoVersion
223225
InstallPrestoRangerPlugin: !Ref 'InstallPrestoRangerPlugin'
226+
PrestoEngine: !Ref 'PrestoEngine'
224227
TimeoutInMinutes: '60'
225228
Outputs:
226229
RangerServerIP:

0 commit comments

Comments
 (0)