[CYB-190] user can specify names of CDP cloud artifacts in example create_datahub_config.sh setup script (#76)

Author: carolynduby

flink-cyber/cyber-jobs/src/main/resources/examples/README.md

@@ -43,7 +43,7 @@ Install the following services on the CDP Base Cluster.
 
 #### CDP Public Cloud
 
-Provision the resources below in the same CDP Environment and use the same prefix at the begining of each resource name:
+Provision the resources below in the same CDP Environment:
 
 | Resource Type | Configuration | Basic | Full |
 | -----------------| ------| ---- | ------|
@@ -66,26 +66,26 @@ Provision the resources below in the same CDP Environment and use the same prefi
 CYBERSEC-2.3.1-1.16.1-csadh1.10.0.0-cdh7.2.17.0-334-2308141830/meta/
 ...
 ```
-4. Create a link from CYBERSEC to the parcel directory.
+4\. Create a link from CYBERSEC to the parcel directory.
 ```shell script
 [cduby@cduby-csa-081423-master0 ~]$ ln -s CYBERSEC-2.3.1-1.16.1-csadh1.10.0.0-cdh7.2.17.0-334-2308141830 CYBERSEC
 [cduby@cduby-csa-081423-master0 ~]$ ls -ld CYBERSEC
 lrwxrwxrwx. 1 cduby cduby 62 Aug 14 20:41 CYBERSEC -> CYBERSEC-2.3.1-1.16.1-csadh1.10.0.0-cdh7.2.17.0-334-2308141830
 [cduby@cduby-csa-081423-master0 ~]$ ls CYBERSEC
 bin  etc  jobs  lib  meta  tools
 ```
-5. Edit the shell configuration defining the PATH variable.  For example, edit .bash_profile.  
+5\. Edit the shell configuration defining the PATH variable.  For example, edit .bash_profile.  
 ```shell script
 ## The shell config file, maybe different.  Locate the definition of PATH in your configs.
 [cduby@cduby-csa-081423-master0 ~]$ vi .bash_profile
 ```
-6. Add $HOME/CYBERSEC/bin to the PATH
+6\. Add $HOME/CYBERSEC/bin to the PATH
 ```shell script
 ### this is an example, use your path here
 PATH=$PATH:$HOME/CYBERSEC/bin:$HOME/.local/bin:$HOME/bin
 export PATH
 ```
-7. Source the shell config or log out and log back in again to refresh the shell settings.  Check the availability of the cybersec commands in the path.
+7\. Source the shell config or log out and log back in again to refresh the shell settings.  Check the availability of the cybersec commands in the path.
 ```shell script
 [cduby@cduby-csa-081423-master0 ~]$ source .bash_profile
 [cduby@cduby-csa-081423-master0 ~]$ which cs-restart-parser
@@ -102,60 +102,67 @@ export PATH
 
 #### CDP Base
 1. Copy the files in examples/setup/templates to example/pipelines
-
 ```
 cd cybersec/flink-cyber/
 ```
 2. Edit the .properties files in example/pipelines with the correct settings for the cluster.
-2. If the Hbase service is not in the same cluster as Flink, download the Hbase client configs from Cloudera Manager.  Move the hbase config zip to the pipelines directory.  Unzip the hbase configuration files.     
-3. If the Hive service is not in the same cluster as Flink, download the Hive on tez client configs from Cloudera Manager.  Move the hive config zip to the pipelines directory.  Unzip the hive config files.
-4. If using a separate Hive cluster, remove the hive_conf/core-site.xml and hive-conf/yarn-site.xml files.  
+3. If the Hbase service is not in the same cluster as Flink, download the Hbase client configs from Cloudera Manager.  Move the hbase config zip to the pipelines directory.  Unzip the hbase configuration files.     
+4. If the Hive service is not in the same cluster as Flink, download the Hive on tez client configs from Cloudera Manager.  Move the hive config zip to the pipelines directory.  Unzip the hive config files.
+5. If using a separate Hive cluster, remove the hive_conf/core-site.xml and hive-conf/yarn-site.xml files.  
 
 #### CDP Public Cloud
 1. If necessary, install the [CDP CLI client](https://docs.cloudera.com/cdp-public-cloud/cloud/cli/topics/mc-cli-client-setup.html).
-2. Run the command line ./create_datahub_config.sh <environment_name> <prefix>. When prompted enter your workload password.
+2. [Install the jq package](https://jqlang.github.io/jq/download/).
+3. Create a properties file with the names of the CDP cloud resources.  
+```shell script
+hive_datahub_name=name_of_hive_datahub
+kafka_datahub_name=name_of_kafka_datahub
+opdb_database_name=name_of_operational_db
+```
+Omit any lines for the hive datahubs or operational DB.  The minimal properties files is shown below:
+```shell script
+kafka_datahub_name=name_of_kafka_datahub
+```
+4\. Run the command line ./create_datahub_config.sh <environment_name> <properties_file>. When prompted enter your workload password.
 ```shell script
 cduby@cduby-MBP16-21649 examples % cd cybersec/flink-cyber/cyber-jobs/src/main/resources/examples/setup 
-cduby@cduby-MBP16-21649 setup % ./create_datahub_config.sh se-sandboxx-aws cduby
-cleaning up hive configs
+cduby@cduby-MBP16-21649 setup % ./create_datahub_config.sh se-sandboxx-aws datahub_setup_kafka_hive_opdb.properties 
+When prompted, enter your workload user password.
+INFO: resetting hive configs from datahub de-cduby-013024
 Enter host password for user 'cduby':
-  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                 Dload  Upload   Total   Spent    Left  Speed
-100 11102    0 11102    0     0   2870      0 --:--:--  0:00:03 --:--:--  2876
-x hive-conf/mapred-site.xml
-x hive-conf/hdfs-site.xml
-x hive-conf/hive-site.xml
-x hive-conf/atlas-application.properties
-x hive-conf/log4j.properties
 x hive-conf/hadoop-env.sh
+x hive-conf/hdfs-site.xml
 x hive-conf/log4j2.properties
-x hive-conf/redaction-rules.json
-x hive-conf/core-site.xml
+x hive-conf/beeline-site.xml
 x hive-conf/yarn-site.xml
+x hive-conf/mapred-site.xml
+x hive-conf/hive-site.xml
+x hive-conf/atlas-application.properties
 x hive-conf/hive-env.sh
-x hive-conf/beeline-site.xml
+x hive-conf/core-site.xml
+x hive-conf/redaction-rules.json
+x hive-conf/log4j.properties
+INFO: Resetting OPDB configs from datahub ckdodb-013024
 Enter host password for user 'cduby':
-  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                 Dload  Upload   Total   Spent    Left  Speed
-100  5828  100  5828    0     0   5394      0  0:00:01  0:00:01 --:--:--  5436
-x hbase-conf/hdfs-site.xml
-x hbase-conf/atlas-application.properties
 x hbase-conf/hbase-omid-client-config.yml
+x hbase-conf/hbase-site.xml
+x hbase-conf/jaas.conf
+x hbase-conf/hdfs-site.xml
 x hbase-conf/hbase-env.sh
 x hbase-conf/core-site.xml
+x hbase-conf/atlas-application.properties
 x hbase-conf/log4j.properties
-x hbase-conf/hbase-site.xml
-x hbase-conf/jaas.conf
+INFO: getting Phoenix connection settings
+Enter host password for user 'cduby':
 Certificate was added to keystore
-[email protected]
 ```
 ### (Optional) Download Maxmind Database
 1. Optionally download the binary (mmdb) version of the [Maxmind GeoLite2 City and ASN Databases](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data). Create a Maxmind account login if you don't have one already.  If you don't download the databases, the triaging job will operate but events will not have geocode (country, city, lat, lon) or asn (network) enrichments. 
 2. cp the GeoLite2 .tar.gz files to the examples/setup directory.
 ```shell script
 cduby@cduby-MBP16-21649 templates % cp GeoLite2-*.tar.gz examples/setup
 ```
-3. Scp the examples directory tree to the flink gateway host.
+3\. Scp the examples directory tree to the flink gateway host.
 ```shell script
 scp -r examples <user>@<flink_gateway_host>:/home/<user>
 ```
@@ -193,7 +200,7 @@ no mock server running
 100   138  100   138    0     0    874      0 --:--:-- --:--:-- --:--:--   878
 
 ```
-3. Start the basic pipeline.  If you have all the required services installed, start the full pipeline.
+3\. Start the basic pipeline.  If you have all the required services installed, start the full pipeline.
 ```shell script
 ./start_basic.sh
 ## if all required services are installed

flink-cyber/cyber-jobs/src/main/resources/examples/setup/create_datahub_config.sh

@@ -1,63 +1,125 @@
+#!/usr/bin/env bash
+
+report_usage_error() {
+   echo "$(basename $0) $1" >&2
+   exit 2
+}
+
+report_fail() {
+   echo "$(basename $0) ERROR: $1" >&2
+   exit 2
+}
+
+report_info() {
+  echo "INFO: $1" >&2
+}
+
 if [[ $# -ne 2 ]]; then
-    echo "$(basename $0) <environment_name> <cluster_prefix>" >&2
-    exit 2
+  report_usage_error "<environment_name> <properties_file>"
 fi
 
-echo "When prompted, enter your workload user password."
 get_dh_name() {
   cdp datahub list-clusters | jq -r '.clusters[] | select(.clusterName | contains ("'"$1"'")) | select(.workloadType | contains ("'$2'")) | .clusterName '
 }
 
+# read_properties_into_variables <property_file_name>
+# Read the name value pairs
+#   split at first equals to key and value
+#   convert property name to legal shell variable name
+#   set shell variable to property variable
+function read_properties_into_variables() {
+  while read -r line; do
+    [[ "$line" =~ ^([[:space:]]*|[[:space:]]*#.*)$ ]] && continue
+    value=${line#*=}
+    key=${line%"=$value"}
+    key=$(echo $key | tr '.' '_')
+    eval ${key}=${value}
+  done <$1
+}
+
 env_name=$1
-cluster_prefix=$2
+properties_file=$2
+
+if [[ -f "$properties_file" ]]; then
+  read_properties_into_variables "$properties_file"
+else
+    report_fail "Properties file $properties_file can't be read.  Check path and permissions."
+fi
+
+echo "When prompted, enter your workload user password."
+
+found_env=$(cdp environments list-environments | jq -r '.environments[] | select(.environmentName=="'"${env_name}"'") | .environmentName')
+if [[ -z "${found_env}" ]]; then
+    report_fail "Environment ${env_name} does not exist."
+fi
+
 config_dir=../pipelines
 workload_user=$(cdp iam get-user | jq -r '.user.workloadUsername')
 
 # discover hive configs
-hive_dh=$(get_dh_name "${cluster_prefix}" "Hive")
-if [[ ! -z "$hive_dh" ]]; then
-    hive_zip=$config_dir/hive-conf.zip
-    hive_conf="$config_dir/hive-conf"
-    hive_cm_api=$(cdp datahub describe-cluster --cluster-name "$hive_dh" | jq -r '.cluster.endpoints.endpoints[] | select (.serviceName | contains("CM-API")) | .serviceUrl')
-    echo resetting hive configs from datahub ${hive_dh}
-    rm -f "$hive_zip"
-    rm -rf "$hive_conf"
-    curl -S -s -o "${hive_zip}" -u "${workload_user}" ${hive_cm_api}/v41/clusters/${hive_dh}/services/hive_on_tez/clientConfig
-    if [[ -f "$hive_zip" ]]; then
-       tar -zxvf "$hive_zip" -C "$config_dir"
-       rm -f "$hive_conf/core-site.xml"
-       rm -f "$hive_conf/yarn-site.xml"
+if [[ ! -z "${hive_datahub_name}" ]]; then
+    hive_dh=$(get_dh_name "${hive_datahub_name}" "Hive")
+    if [[ ! -z "$hive_dh" ]]; then
+        hive_zip=$config_dir/hive-conf.zip
+        hive_conf="$config_dir/hive-conf"
+        hive_cm_api=$(cdp datahub describe-cluster --cluster-name "$hive_dh" | jq -r '.cluster.endpoints.endpoints[] | select (.serviceName | contains("CM-API")) | .serviceUrl')
+        report_info "resetting hive configs from datahub ${hive_dh}"
+        rm -f "$hive_zip"
+        rm -rf "$hive_conf"
+        curl -S -s -o "${hive_zip}" -u "${workload_user}" ${hive_cm_api}/v41/clusters/${hive_dh}/services/hive_on_tez/clientConfig
+        if [[ -f "$hive_zip" ]]; then
+           tar -zxvf "$hive_zip" -C "$config_dir"
+           rm -f "$hive_conf/core-site.xml"
+           rm -f "$hive_conf/yarn-site.xml"
+        else
+            report_fail "Could not get hive configuration."
+        fi
     else
-        echo "ERROR: could not get hive configuration."
-        exit 2
+      report_fail "Hive datahub '${hive_datahub_name}' not found in environment '${env_name}'"
     fi
+else
+  report_info "Hive is not configured.  Property hive_datahub_name not defined in properties file"
 fi
 
-# discover kafka connection config 
-kafka_dh_name=$(get_dh_name "${cluster_prefix}" "Kafka")
+
+
+# discover kafka connection config
+if [[ ! -z "${kafka_datahub_name}" ]]; then
+    kafka_dh_name=$(get_dh_name "${kafka_datahub_name}" "Kafka")
+    if [[ -z "${kafka_dh_name}" ]]; then
+        report_fail "Environment '${env_name}' does not contain a datahub named '${kafka_datahub_name}' containing Kafka"
+    fi
+else
+    report_fail "Kafka is not configured.  Property kafka_datahub_name not defined in properties file"
+fi
 schema_registry=$(cdp datahub describe-cluster --cluster-name ${kafka_dh_name} | jq -r '.cluster.instanceGroups[] | select(.name | contains("master")) | .instances[].fqdn')
 kafka_broker=$(cdp datahub describe-cluster --cluster-name  ${kafka_dh_name} | jq -r '.cluster.endpoints.endpoints[] | select (.serviceName | contains("KAFKA_BROKER")) | .serviceUrl' | sed 's/ //g')
 
 # opdb (hbase and phoenix) connection config
-opdb_cluster_name=$(cdp opdb list-databases --environment-name ${env_name} | jq -r '.databases[] | select(.databaseName | contains ("'"${cluster_prefix}"'")) | .databaseName') 
-phoenix_query_server_host=NO_OPDB_CLUSTER
-if [[ ! -z "$opdb_cluster_name" ]]; then
-    echo resetting opdb configs from datahub ${opdb_cluster_name}
-    opdb_client_url=$(cdp opdb describe-client-connectivity --environment-name ${env_name} --database-name ${opdb_cluster_name} | jq -r '.connectors[] | select(.name=="hbase") | .configuration.clientConfigurationDetails[].url')
-    hbase_zip="$config_dir/hbase-config.zip"
-    hbase_conf="$config_dir/hbase-conf"
-    rm -f "$hbase_zip"
-    rm -rf "$hbase_conf"
-    curl -S -s -f -o "$hbase_zip" -u "${workload_user}" "${opdb_client_url}"
-    if [[ -f "$hbase_zip" ]]; then
-       tar -zxvf "$hbase_zip" -C "$config_dir"
-    else 
-        echo "ERROR: could not get hbase configuration."
-        exit 2
+if [[ ! -z "${opdb_database_name}" ]]; then
+    opdb_cluster_name=$(cdp opdb list-databases --environment-name ${env_name} | jq -r '.databases[] | select(.databaseName | contains ("'"${opdb_database_name}"'")) | .databaseName')
+    phoenix_query_server_host=NO_OPDB_CLUSTER
+    if [[ ! -z "$opdb_cluster_name" ]]; then
+        report_info "Resetting OPDB configs from datahub ${opdb_cluster_name}"
+        opdb_client_url=$(cdp opdb describe-client-connectivity --environment-name ${env_name} --database-name ${opdb_cluster_name} | jq -r '.connectors[] | select(.name=="hbase") | .configuration.clientConfigurationDetails[].url')
+        hbase_zip="$config_dir/hbase-config.zip"
+        hbase_conf="$config_dir/hbase-conf"
+        rm -f "$hbase_zip"
+        rm -rf "$hbase_conf"
+        curl -S -s -f -o "$hbase_zip" -u "${workload_user}" "${opdb_client_url}"
+        if [[ -f "$hbase_zip" ]]; then
+           tar -zxvf "$hbase_zip" -C "$config_dir"
+        else
+            report_fail "Could not get HBase configuration."
+        fi
+        base_opdb_services_url=$(echo ${opdb_client_url} | sed -e 's/hbase\/clientConfig//')
+        report_info "getting Phoenix connection settings"
+        phoenix_query_server_host=$(curl -S -s -u ${workload_user} ${base_opdb_services_url}/phoenix/roles | jq -r '.items[] | select (.type | contains("PHOENIX_QUERY_SERVER")) | .hostRef.hostname')
+    else
+        report_fail "OPDB database ${hive_datahub_name} not found in environment ${env_name}"
     fi
-    base_opdb_services_url=$(echo ${opdb_client_url} | sed -e 's/hbase\/clientConfig//')
-    echo "getting phoenix connection settings"
-    phoenix_query_server_host=$(curl -S -s -u ${workload_user} ${base_opdb_services_url}/phoenix/roles | jq -r '.items[] | select (.type | contains("PHOENIX_QUERY_SERVER")) | .hostRef.hostname')
+else
+    report_info "HBase and Phoenix are not configured.  Property opdb_database_name not defined in properties file" >&2
 fi 
 
 cdp environments get-keytab --environment-name $env_name | jq -r '.contents' | base64 --decode > ${config_dir}/krb5.keytab