Skip to content

Commit 0795f72

Browse files
owlleg6owlleg6
authored
Merge pull request #3 from data-platform-hq/optional-folders-creation-with-terraform
feat: readme update; gen2_path resources
2 parents ac51acd + 2d4ccbd commit 0795f72

File tree

3 files changed

+150
-18
lines changed

3 files changed

+150
-18
lines changed

‎README.md‎

Lines changed: 62 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,52 @@
22
Terraform module for creation Azure Data Lake Storage Gen2 file system
33

44
## Usage
5+
By default, this module will provision Data Lake Gen2 Path in a File System within an Azure Storage Account with custom Bash script, which does not require any Storage specific role. (E.x., Storage Blob Data Contributor/Owner).
56

7+
If your Service Principal for Terraform is assigned to **"Storage Blob Data Owner"** role it is possible to create File System folders with Azurerm resource(azurerm_storage_data_lake_gen2_path). In this case, please set _**"storage_role_assigned"**_ variable to 'true'.
8+
9+
Below is an example of module usage, here you are providing users in "_adls_ro_users_" and "_adls_rw_users_" variables. After, passing this map to ADLS module.
10+
```
11+
locals {
12+
13+
# Read only users
14+
adls_ro_users = [
15+
"[email protected]",
16+
"[email protected]"
17+
]
18+
19+
# Read and Write users
20+
adls_rw_users = [
21+
"[email protected]"
22+
]
23+
24+
# Map with user permissions
25+
adls2fs = {
26+
"my-data-lake" = concat([
27+
for u in locals.adls_ro_users : { "user" = var.user_object_ids[u], "permissions" = "r-x", "scope" = "access", "type" = "user" }
28+
], [
29+
for u in locals.adls_ro_users : { "user" = var.user_object_ids[u], "permissions" = "r-x", "scope" = "default", "type" = "user" }
30+
], [
31+
for u in locals.adls_rw_users : { "user" = var.user_object_ids[u], "permissions" = "rwx", "scope" = "access", "type" = "user" }
32+
], [
33+
for u in locals.adls_rw_users : { "user" = var.user_object_ids[u], "permissions" = "rwx", "scope" = "default", "type" = "user" }
34+
]
35+
)
36+
}
37+
}
38+
39+
module "adls" {
40+
source = "data-platform-hq/adls-v2/azurerm"
41+
for_each = local.adls2fs
42+
43+
name = each.key
44+
permissions = each.value
45+
storage_role_assigned = true
46+
folders = ["raw", "refined", "test", "data-product", "e2e"]
47+
storage_account_id = module.storage_account.id
48+
storage_account_name = module.storage_account.name
49+
}
50+
```
651
<!-- BEGIN_TF_DOCS -->
752
## Requirements
853

@@ -25,24 +70,27 @@ No modules.
2570

2671
## Resources
2772

28-
| Name | Type |
29-
|------|------|
73+
| Name | Type |
74+
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|------|
3075
| [azurerm_storage_data_lake_gen2_filesystem.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_data_lake_gen2_filesystem) | resource |
31-
| [null_resource.create_folders](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource |
32-
| [null_resource.create_root_folder](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource |
76+
| [null_resource.create_folders](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource |
77+
| [null_resource.create_root_folder](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource |
78+
| [azurerm_storage_data_lake_gen2_path.root](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_data_lake_gen2_path) | resource |
79+
| [azurerm_storage_data_lake_gen2_path.other](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_data_lake_gen2_path) | resource |
3380

3481
## Inputs
3582

36-
| Name | Description | Type | Default | Required |
37-
|------|-------------|------|---------|:--------:|
38-
| <a name="input_ace_default"></a> [ace\_default](#input\_ace\_default) | Default ACE values | `list(map(string))` | <pre>[<br> {<br> "permissions": "---",<br> "scope": "access",<br> "type": "other"<br> },<br> {<br> "permissions": "---",<br> "scope": "default",<br> "type": "other"<br> },<br> {<br> "permissions": "rwx",<br> "scope": "access",<br> "type": "group"<br> },<br> {<br> "permissions": "rwx",<br> "scope": "access",<br> "type": "mask"<br> },<br> {<br> "permissions": "rwx",<br> "scope": "access",<br> "type": "user"<br> },<br> {<br> "permissions": "rwx",<br> "scope": "default",<br> "type": "group"<br> },<br> {<br> "permissions": "rwx",<br> "scope": "default",<br> "type": "mask"<br> },<br> {<br> "permissions": "rwx",<br> "scope": "default",<br> "type": "user"<br> }<br>]</pre> | no |
39-
| <a name="input_ad_groups"></a> [ad\_groups](#input\_ad\_groups) | Data which is contain mapping AD group name and GUID | `map(string)` | `{}` | no |
40-
| <a name="input_folders"></a> [folders](#input\_folders) | Name of ADLS folders to create in root directory | `list(any)` | `[]` | no |
41-
| <a name="input_name"></a> [name](#input\_name) | Name of ADLS FS to create | `string` | n/a | yes |
42-
| <a name="input_permissions"></a> [permissions](#input\_permissions) | List of ADLS FS permissions | `list(map(string))` | <pre>[<br> {}<br>]</pre> | no |
43-
| <a name="input_root_dir"></a> [root\_dir](#input\_root\_dir) | Name of ADLS root directory | `string` | `"data"` | no |
44-
| <a name="input_storage_account_id"></a> [storage\_account\_id](#input\_storage\_account\_id) | ID of storage account to create ADLS in | `string` | n/a | yes |
45-
| <a name="input_storage_account_name"></a> [storage\_account\_name](#input\_storage\_account\_name) | Name of storage account to create ADLS in | `string` | n/a | yes |
83+
| Name | Description | Type | Default | Required |
84+
|--------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
85+
| <a name="input_ace_default"></a> [ace\_default](#input\_ace\_default) | Default ACE values | `list(map(string))` | <pre>[<br> {<br> "permissions": "---",<br> "scope": "access",<br> "type": "other"<br> },<br> {<br> "permissions": "---",<br> "scope": "default",<br> "type": "other"<br> },<br> {<br> "permissions": "rwx",<br> "scope": "access",<br> "type": "group"<br> },<br> {<br> "permissions": "rwx",<br> "scope": "access",<br> "type": "mask"<br> },<br> {<br> "permissions": "rwx",<br> "scope": "access",<br> "type": "user"<br> },<br> {<br> "permissions": "rwx",<br> "scope": "default",<br> "type": "group"<br> },<br> {<br> "permissions": "rwx",<br> "scope": "default",<br> "type": "mask"<br> },<br> {<br> "permissions": "rwx",<br> "scope": "default",<br> "type": "user"<br> }<br>]</pre> | no |
86+
| <a name="input_ad_groups"></a> [ad\_groups](#input\_ad\_groups) | Data which is contain mapping AD group name and GUID | `map(string)` | `{}` | no |
87+
| <a name="input_folders"></a> [folders](#input\_folders) | Name of ADLS folders to create in root directory | `list(any)` | `[]` | no |
88+
| <a name="input_storage_role_assigned"></a> [storage\_role\_assigned](#input\_storage\_role\_assigned) | Is Storage Blob Data Owner Role assigned to Terraform Service Principal? Provides an ability to create File System with bash script(false) or azurerm resources(true). | `bool` | false | no |
89+
| <a name="input_name"></a> [name](#input\_name) | Name of ADLS FS to create | `string` | n/a | yes |
90+
| <a name="input_permissions"></a> [permissions](#input\_permissions) | List of ADLS FS permissions | `list(map(string))` | <pre>[<br> {}<br>]</pre> | no |
91+
| <a name="input_root_dir"></a> [root\_dir](#input\_root\_dir) | Name of ADLS root directory | `string` | `"data"` | no |
92+
| <a name="input_storage_account_id"></a> [storage\_account\_id](#input\_storage\_account\_id) | ID of storage account to create ADLS in | `string` | n/a | yes |
93+
| <a name="input_storage_account_name"></a> [storage\_account\_name](#input\_storage\_account\_name) | Name of storage account to create ADLS in | `string` | n/a | yes |
4694

4795
## Outputs
4896

0 commit comments

Comments
 (0)