feat: added support to install and start the Elastic's Natural Language Processing model using new input enable_elser_model<br>- the default value of the member_host_flavor in the DA has changed from multitenant to b3c.4x16.encrypted.<br>- the default value of the member_cpu_count in the DA has changed from 0 to 3<br>- the default value of plan in the DA has updated from enterprise to platinum. (#223)

Author: Aayush-Abhyarthi

README.md

@@ -61,6 +61,7 @@ You need the following permissions to run this module.
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3.0 |
 | <a name="requirement_ibm"></a> [ibm](#requirement\_ibm) | >= 1.65.0, <2.0.0 |
+| <a name="requirement_null"></a> [null](#requirement\_null) | >= 3.2.1, < 4.0.0 |
 | <a name="requirement_time"></a> [time](#requirement\_time) | >= 0.9.1 |
 
 ### Modules
@@ -77,6 +78,8 @@ You need the following permissions to run this module.
 | [ibm_iam_authorization_policy.policy](https://registry.terraform.io/providers/ibm-cloud/ibm/latest/docs/resources/iam_authorization_policy) | resource |
 | [ibm_resource_key.service_credentials](https://registry.terraform.io/providers/ibm-cloud/ibm/latest/docs/resources/resource_key) | resource |
 | [ibm_resource_tag.elasticsearch_tag](https://registry.terraform.io/providers/ibm-cloud/ibm/latest/docs/resources/resource_tag) | resource |
+| [null_resource.put_vectordb_model](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource |
+| [null_resource.start_vectordb_model](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource |
 | [time_sleep.wait_for_authorization_policy](https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep) | resource |
 | [ibm_database_connection.database_connection](https://registry.terraform.io/providers/ibm-cloud/ibm/latest/docs/data-sources/database_connection) | data source |
 
@@ -91,6 +94,7 @@ You need the following permissions to run this module.
 | <a name="input_backup_encryption_key_crn"></a> [backup\_encryption\_key\_crn](#input\_backup\_encryption\_key\_crn) | The CRN of a KMS (Key Protect or Hyper Protect Crypto Service) key to use for encrypting the disk that holds deployment backups. Applies only if `kms_encryption_enabled` is true. Limitations exist for regions. For more information, see [Key Protect integration](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-key-protect&interface=ui#key-byok) or [Hyper Protect Crypto Services integration](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-hpcs#use-hpcs-backups). | `string` | `null` | no |
 | <a name="input_cbr_rules"></a> [cbr\_rules](#input\_cbr\_rules) | The list of context-based restriction rules to create. | <pre>list(object({<br>    description = string<br>    account_id  = string<br>    rule_contexts = list(object({<br>      attributes = optional(list(object({<br>        name  = string<br>        value = string<br>    }))) }))<br>    enforcement_mode = string<br>  }))</pre> | `[]` | no |
 | <a name="input_elasticsearch_version"></a> [elasticsearch\_version](#input\_elasticsearch\_version) | The version of Databases for Elasticsearch to deploy. Possible values: `8.7`, `8.10`, `8.12`, which requires an Enterprise Platinum pricing plan. If no value is specified, the current preferred version for IBM Cloud Databases is used. | `string` | `null` | no |
+| <a name="input_enable_elser_model"></a> [enable\_elser\_model](#input\_enable\_elser\_model) | Set it to true to install and start the Elastic's Natural Language Processing model. [Learn more](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-elser-embeddings-elasticsearch) | `bool` | `false` | no |
 | <a name="input_existing_kms_instance_guid"></a> [existing\_kms\_instance\_guid](#input\_existing\_kms\_instance\_guid) | The GUID of a Hyper Protect Crypto Services or Key Protect instance for the CRN specified in `kms_key_crn` and `backup_encryption_key_crn`. Applies only if `kms_encryption_enabled` is true, `skip_iam_authorization_policy` is false, and you specify values for `kms_key_crn` or `backup_encryption_key_crn`. | `string` | `null` | no |
 | <a name="input_kms_encryption_enabled"></a> [kms\_encryption\_enabled](#input\_kms\_encryption\_enabled) | Whether to specify the keys used to encrypt data in the database. Specify `true` to identify the encryption keys. If set to `false`, the data is encrypted with randomly generated keys. [Learn more about Key Protect integration](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-key-protect). [Learn more about HPCS integration](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-hpcs). | `bool` | `false` | no |
 | <a name="input_kms_key_crn"></a> [kms\_key\_crn](#input\_kms\_key\_crn) | The root key CRN of the Key Protect or Hyper Protect Crypto Services instance to use for disk encryption. Applies only if `kms_encryption_enabled` is true. | `string` | `null` | no |

examples/fscloud/main.tf

@@ -68,6 +68,7 @@ module "elasticsearch" {
   member_host_flavor         = "b3c.4x16.encrypted"
   backup_encryption_key_crn  = var.backup_encryption_key_crn
   backup_crn                 = var.backup_crn
+  enable_elser_model         = var.enable_elser_model
   cbr_rules = [
     {
       description      = "${var.prefix}-elasticsearch access only from vpc"

examples/fscloud/variables.tf

@@ -108,3 +108,9 @@ variable "backup_encryption_key_crn" {
   default     = null
   # Validation happens in the root module
 }
+
+variable "enable_elser_model" {
+  type        = bool
+  description = "Set it to true to install and start the Elastic's Natural Language Processing model. [Learn more](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-elser-embeddings-elasticsearch)"
+  default     = false
+}

main.tf

@@ -8,6 +8,10 @@ locals {
   validate_auth_policy = var.kms_encryption_enabled && var.skip_iam_authorization_policy == false && var.existing_kms_instance_guid == null ? tobool("When var.skip_iam_authorization_policy is set to false, and var.kms_encryption_enabled to true, a value must be passed for var.existing_kms_instance_guid in order to create the auth policy.") : true
   # tflint-ignore: terraform_unused_declarations
   validate_backup_key = var.backup_encryption_key_crn != null && var.use_default_backup_encryption_key == true ? tobool("When passing a value for 'backup_encryption_key_crn' you cannot set 'use_default_backup_encryption_key' to 'true'") : true
+  # tflint-ignore: terraform_unused_declarations
+  validate_plan = var.enable_elser_model && var.plan != "platinum" ? tobool("When var.enable_elser_model is set to true, a value for var.plan must be 'platinum' in order to enable ELSER model.") : true
+  # tflint-ignore: terraform_unused_declarations
+  validate_es_user = var.enable_elser_model && !((length(var.service_credential_names) > 0 && length([for k, v in var.service_credential_names : k if v == "Administrator"]) > 0) || var.admin_pass != null) ? tobool("When var.enable_elser_model is set to true, a value must be passed for var.service_credential_names or var.admin_pass. var.service_credential_names must contain at least one credential name with Administrator role.") : true
 
   # If no value passed for 'backup_encryption_key_crn' use the value of 'kms_key_crn' and perform validation of 'kms_key_crn' to check if region is supported by backup encryption key.
 
@@ -262,3 +266,46 @@ data "ibm_database_connection" "database_connection" {
   user_id       = ibm_database.elasticsearch.adminuser
   user_type     = "database"
 }
+
+##############################################################################
+# ELSER support
+##############################################################################
+
+# Enable Elastic's Natural Language Processing model (ELSER) support by calling ES REST API directly using shell script. Learn more https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-elser-embeddings-elasticsearch
+# Firstly, ELSER model is installed using 'put_vectordb_model' null_resource. Secondly, ELSER model is started with `start_vectordb_model` null_resource.
+#
+# To authenticate ES rest API, the credentials are extracted from 'service_credential_names' or ES 'adminpassword' using the following logic:
+# if elser_model is enabled, then
+#   if service_credential_names are used, then get the key name of a credential where role is 'Administrator'
+#       use the key name to obtain username and password from service_credentials_object
+#   else if admin_pass is used, then use 'admin' for username and password from ES password
+locals {
+  es_admin_users = var.enable_elser_model && var.service_credential_names != null && length(var.service_credential_names) > 0 ? [for k, v in var.service_credential_names : k if v == "Administrator"] : []
+  es_admin_user  = length(local.es_admin_users) > 0 ? local.es_admin_users[0] : null
+  es_username    = local.es_admin_user != null ? local.service_credentials_object["credentials"][local.es_admin_user]["username"] : var.admin_pass != null ? "admin" : null
+  es_password    = local.es_admin_user != null ? local.service_credentials_object["credentials"][local.es_admin_user]["password"] : var.admin_pass != null ? ibm_database.elasticsearch.adminpassword : null
+  es_url         = local.es_username != null && local.es_password != null ? "https://${local.es_username}:${local.es_password}@${data.ibm_database_connection.database_connection.https[0].hosts[0].hostname}:${data.ibm_database_connection.database_connection.https[0].hosts[0].port}" : null
+}
+
+resource "null_resource" "put_vectordb_model" {
+  count = var.enable_elser_model ? 1 : 0
+  provisioner "local-exec" {
+    command     = "${path.module}/scripts/put_vectordb_model.sh"
+    interpreter = ["/bin/bash", "-c"]
+    environment = {
+      ES = local.es_url
+    }
+  }
+}
+
+resource "null_resource" "start_vectordb_model" {
+  depends_on = [null_resource.put_vectordb_model]
+  count      = var.enable_elser_model ? 1 : 0
+  provisioner "local-exec" {
+    command     = "${path.module}/scripts/start_vectordb_model.sh"
+    interpreter = ["/bin/bash", "-c"]
+    environment = {
+      ES = local.es_url
+    }
+  }
+}

modules/fscloud/README.md

@@ -37,6 +37,7 @@ No resources.
 | <a name="input_backup_encryption_key_crn"></a> [backup\_encryption\_key\_crn](#input\_backup\_encryption\_key\_crn) | The CRN of a Hyper Protect Crypto Service use for encrypting the disk that holds deployment backups. Only used if var.kms\_encryption\_enabled is set to true. There are limitation per region on the Hyper Protect Crypto Services and region for those services. See https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-hpcs#use-hpcs-backups | `string` | `null` | no |
 | <a name="input_cbr_rules"></a> [cbr\_rules](#input\_cbr\_rules) | (Optional, list) List of CBR rules to create | <pre>list(object({<br>    description = string<br>    account_id  = string<br>    rule_contexts = list(object({<br>      attributes = optional(list(object({<br>        name  = string<br>        value = string<br>    }))) }))<br>    enforcement_mode = string<br>  }))</pre> | `[]` | no |
 | <a name="input_elasticsearch_version"></a> [elasticsearch\_version](#input\_elasticsearch\_version) | Version of the Elasticsearch instance. If no value is passed, the current preferred version of IBM Cloud Databases is used. | `string` | `null` | no |
+| <a name="input_enable_elser_model"></a> [enable\_elser\_model](#input\_enable\_elser\_model) | Set it to true to install and start the Elastic's Natural Language Processing model.[Learn more](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-elser-embeddings-elasticsearch) | `bool` | `false` | no |
 | <a name="input_existing_kms_instance_guid"></a> [existing\_kms\_instance\_guid](#input\_existing\_kms\_instance\_guid) | The GUID of the Hyper Protect Crypto Services instance. It is only required while creating authorization policy. | `string` | `null` | no |
 | <a name="input_kms_key_crn"></a> [kms\_key\_crn](#input\_kms\_key\_crn) | The root key CRN of the Hyper Protect Crypto Service (HPCS) to use for disk encryption. | `string` | n/a | yes |
 | <a name="input_member_cpu_count"></a> [member\_cpu\_count](#input\_member\_cpu\_count) | Allocated dedicated CPU per member. For shared CPU, set to 0. For more information, see https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-resources-scaling | `number` | `0` | no |

modules/fscloud/main.tf

@@ -24,4 +24,5 @@ module "elasticsearch" {
   member_host_flavor            = var.member_host_flavor
   auto_scaling                  = var.auto_scaling
   service_credential_names      = var.service_credential_names
+  enable_elser_model            = var.enable_elser_model
 }

modules/fscloud/variables.tf

@@ -190,3 +190,9 @@ variable "cbr_rules" {
   default     = []
   # Validation happens in the rule module
 }
+
+variable "enable_elser_model" {
+  type        = bool
+  description = "Set it to true to install and start the Elastic's Natural Language Processing model.[Learn more](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-elser-embeddings-elasticsearch)"
+  default     = false
+}

scripts/put_vectordb_model.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+set -e
+
+#Function to install the vectorDB model
+Put_model() {
+
+sleep=2
+for i in $(seq 1 4); do
+  sleep=$((sleep*2))
+
+  sleep $sleep
+  # learn more https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-ml-put-trained-model#operation-ml-put-trained-model-wait_for_completion
+  response=$(curl -s -w "%{http_code}" -kX PUT "$ES/_ml/trained_models/.elser_model_1?wait_for_completion=true&pretty" -H 'Content-Type: application/json' -d'
+  {
+    "input": {
+    "field_names": ["text_field"]
+    }
+  }
+  ')
+
+  http_code=$(tail -n1 <<< "$response")
+  content=$(sed '$ d' <<< "$response")
+
+  # Check the HTTP status code
+  if [ "$http_code" -eq 200 ] || [ "$http_code" -eq 201 ]; then
+    echo "Request sent successfully."
+    break
+  else
+    echo "Failed to install the vectorDB model. HTTP status code: $http_code"
+    echo "Reponse: $content"
+    if [ "$i" -eq 4 ]; then
+      exit 1
+    fi
+  fi
+done
+
+}
+
+Put_model

scripts/start_vectordb_model.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+set -e
+
+#Function to start the vectorDB model
+Start_model() {
+
+# It takes few minute for the model to finish installing before we can start trained model deployment, therefore we sleep for 180 seconds (3m).
+sleep 180
+# Learn more https://www.elastic.co/guide/en/elasticsearch/reference/current/start-trained-model-deployment.html
+response=$(curl -s -w "%{http_code}" -kX POST "$ES/_ml/trained_models/.elser_model_1/deployment/_start?wait_for=started&timeout=3m&deployment_id=for_search&pretty")
+
+http_code=$(tail -n1 <<< "$response")
+content=$(sed '$ d' <<< "$response")
+
+# Check the HTTP status code
+if [ "$http_code" -eq 200 ] || [ "$http_code" -eq 201 ]; then
+  echo "Request sent successfully."
+else
+  echo "Failed to start the vectorDB model. HTTP status code: $http_code"
+  echo "Reponse: $content"
+  exit 1
+fi
+
+}
+
+Start_model

solutions/standard/main.tf

@@ -83,4 +83,5 @@ module "elasticsearch" {
   member_cpu_count              = var.member_cpu_count
   auto_scaling                  = var.auto_scaling
   service_credential_names      = var.service_credential_names
+  enable_elser_model            = var.enable_elser_model
 }