Develop (#941)

* Fixing Issue #440: Adding example w/ gradle props

Adding a new example that utilizes gradle properties within
deployment configuration files for substitutions for variables. This
eliminates the need to utilize the mlAppConfig extension in
build.gradle files

* E2E/bug fixes -- tests for some bug fixes (#927)

* modify verification on advanced setting

* add tests on primary key and duplicate properties

* modify verification on trace headers

* added step on run input flow to set output uri suffix

* added mlcpInput page object

* DHFPROD-496 make consistent with current tutorial (#933)

Remove title declarations since we don't define a title property in the tutuorial.

* DHFPROD-675 add index confirm for save new entity

* added step to update index after initial entity creation (#940)

Author: aebadirad

examples/README.md

@@ -3,9 +3,11 @@
 This folder contains working examples of various DHF usage scenarios.
 
 1. [barebones](https://github.com/marklogic/marklogic-data-hub/tree/master/examples/barebones) - an example of the minimum configuration necessary to run a Gradle based Data Hub
+1. [custom-tokens](https://github.com/marklogic/marklogic-data-hub/tree/master/examples/custom-tokens) - an example where gradle environment properties are used in substitutions of configuration files that is based off of the barebones example
 1. [healthcare](https://github.com/marklogic/marklogic-data-hub/tree/master/examples/healthcare) - an example of a Healthcare 360 Data Hub
 1. [hr-hub](https://github.com/marklogic/marklogic-data-hub/tree/master/examples/hr-hub) - an example used for our 1.x tutorial. This example harmonizes data from various HR systems
 1. [load-binaries](https://github.com/marklogic/marklogic-data-hub/tree/master/examples/load-binaries) - an example of how to ingest binaries via an MLCP Input Flow
 1. [online-store](https://github.com/marklogic/marklogic-data-hub/tree/master/examples/online-store) - the example we use for our [Tutorial](https://marklogic.github.io/marklogic-data-hub/tutorial/)
+1. [single-step-ingest](https://github.com/marklogic/marklogic-data-hub/tree/master/examples/single-step-ingest) - an example of a custom REST endpoint that a user can call that will ingest a document into the STAGING database and then harmonize the same document immediately after the document was ingested
 1. [spring-batch](https://github.com/marklogic/marklogic-data-hub/tree/master/examples/spring-batch) - an example of how to load relational data into a Data Hub using Spring Batch
 1. [ssl](https://github.com/marklogic/marklogic-data-hub/tree/master/examples/ssl) - an example of how to configure your Data Hub to use SSL for added security

examples/custom-tokens/README.md

@@ -0,0 +1,57 @@
+# Custom Tokens Example
+
+This example demonstrates how to utilize gradle environment properties that are substituted in configuraton files within the MarkLogic deployment process.  We are basing this example off of the [barebones example](https://github.com/marklogic/marklogic-data-hub/tree/master/examples/barebones).
+
+This example utilizes gradle environment properties that are substituted in the configuration files for a user defined database and application server.  You can see the environment properties that are defined at the bottom of the `gradle.properties` file.
+
+Inside of the `gradle.properties` file, you can see the following information:
+
+```
+# Custom properties defined here
+TEST_DATABASE_NAME=custom-tokens-test-database
+TEST_SERVER_NAME=custom-tokens-test-database-server
+TEST_SERVER_PORT=8014
+TEST_TRACE_AUTH=digest
+```
+
+To utilize the custom tokens, then you will need to refer to them as `%%TEST_DATABASE_NAME%%` if you want to reference the `TEST_DATABASE_NAME` name token within your gradle deployment.  You can change the default token prefix and suffix from "%%" by utilizing the following tokens in your `gradle.properties` file by setting the appropriate prefix and suffix values according:
+
+```
+mlTokenPrefix=
+mlTokenSuffix=
+```
+For more information regarding this, then you can refer to the [ml-gradle wiki](https://github.com/marklogic-community/ml-gradle/wiki/Configuring-resources)
+
+You can see that we are referencing the new custom tokens within the `custom-tokens-test-server.json` file.  We are utilizing the four (4) custom tokens that we defined in our properties file which are the following: `%%TEST_SERVER_NAME%%`, `%%TEST_SERVER_PORT%%`, `%%TEST_DATABASE_NAME%%`, and `%%TEST_TRACE_AUTH%%`.
+
+```json
+{
+  "server-name": "%%TEST_SERVER_NAME%%",
+  "server-type": "http",
+  "root": "/",
+  "group-name": "%%GROUP%%",
+  "port": "%%TEST_SERVER_PORT%%",
+  "modules-database": "%%mlModulesDbName%%",
+  "content-database": "%%TEST_DATABASE_NAME%%",
+  "authentication": "%%TEST_TRACE_AUTH%%",
+  "default-error-format": "json",
+  "error-handler": "/MarkLogic/rest-api/error-handler.xqy",
+  "url-rewriter": "/MarkLogic/rest-api/rewriter.xml",
+  "rewrite-resolves-globally": true
+}
+```
+
+Next, Initialize your DHF app:
+
+```bash
+gradle hubInit
+```
+
+Then Bootstrap your DHF app with the user defined database and application that utilized custom tokens:
+
+```bash
+gradle mlDeploy
+```
+Once the deployment is complete, then you can login to the admin console and see the new test database and test database application server that were created with the values that were specified from our properties file.  Now that you've mastered utilizing custom tokens for your gradle deployment, you can continue on with your DHF app development.
+
+For a complete list of gradle tasks, check here: [https://github.com/marklogic/marklogic-data-hub/wiki/Gradle-Tasks](https://github.com/marklogic/marklogic-data-hub/wiki/Gradle-Tasks)

examples/custom-tokens/build.gradle

@@ -0,0 +1,11 @@
+plugins {
+    // this plugin lets you create properties files
+    // for multiple environments... like dev, qa, prod
+    id 'net.saliman.properties' version '1.4.6'
+
+    // this is the data hub framework gradle plugin
+    // it includes ml-gradle. This plugin is what lets you
+    // run DHF (Data Hub Framework) tasks from the
+    // command line
+    id 'com.marklogic.ml-data-hub' version '2.0.4'
+}

examples/custom-tokens/gradle.properties

@@ -0,0 +1,89 @@
+# These settings are used by the Data Hub Framework when
+# communicating with MarkLogic.
+# The values in this file are meant as project-wide settings.
+# You can override these properties for a specific environment
+# by creating a gradle-{environment}.properties file.
+# For example, to create a properties file for your prod environment create a file
+# named gradle-prod.properties.
+#
+# ....
+mlHost=localhost
+
+# Your MarkLogic Username and Password
+mlUsername=admin
+mlPassword=admin
+
+# If specified, the manage username/password combo is used with the ML Management REST API for managing application
+# resources; this user must have the manage-admin and rest-admin roles.
+#
+# If these are not set, then mlUsername/mlPassword is used for managing application resources.
+# mlManageUsername=
+# mlManagePassword=
+
+# If specified, the admin username/password combo is used with the ML Management REST API for creating users and roles. This
+# user must have the manage-admin or admin role. A good practice is to use your admin account here to create app-specific
+# users and roles, which can then be used as mlManageUsername/mlManagePassword and mlUsername/mlPassword.
+#
+# These properties are also used for connecting to the admin application on port 8001 - e.g. for initializing ML and for
+# waiting for ML to restart.
+#
+# If these properties are not set, then mlUsername/mlPassword will be used.
+# mlAdminUsername=
+# mlAdminPassword=
+
+# If specified, these values can override where the DHF thinks
+# MarkLogic default ports are at. You would only use this if you
+# have changed the ports on which MarkLogic listens
+#
+# mlAppServicesPort=8000
+# mlAdminPort=8001
+# mlManagePort=8002
+
+mlStagingAppserverName=data-hub-STAGING
+mlStagingPort=8010
+mlStagingDbName=data-hub-STAGING
+mlStagingForestsPerHost=4
+mlStagingAuth=digest
+
+mlFinalAppserverName=data-hub-FINAL
+mlFinalPort=8011
+mlFinalDbName=data-hub-FINAL
+mlFinalForestsPerHost=4
+mlFinalAuth=digest
+
+mlTraceAppserverName=data-hub-TRACING
+mlTracePort=8012
+mlTraceDbName=data-hub-TRACING
+mlTraceForestsPerHost=1
+mlTraceAuth=digest
+
+mlJobAppserverName=data-hub-JOBS
+mlJobPort=8013
+mlJobDbName=data-hub-JOBS
+mlJobForestsPerHost=1
+mlJobAuth=digest
+
+mlModulesDbName=data-hub-MODULES
+mlModulesForestsPerHost=1
+
+mlTriggersDbName=data-hub-TRIGGERS
+mlTriggersForestsPerHost=1
+
+mlSchemasDbName=data-hub-SCHEMAS
+mlSchemasForestsPerHost=1
+
+# You can override this to specify an alternate folder for your
+# custom forest info. Defaults to user-config/forests/
+# mlCustomForestPath=forests
+
+# The name of the Role to create for Hub Access
+mlHubUserRole=data-hub-role
+mlHubUserName=data-hub-user
+# this password is autogenerated for you via the 'gradle hubInit' task
+mlHubUserPassword=b$I7'3Ya|&;Ohw.ZzsDY
+
+# Custom properties defined here
+TEST_DATABASE_NAME=custom-tokens-test-database
+TEST_SERVER_NAME=custom-tokens-test-database-server
+TEST_SERVER_PORT=8014
+TEST_TRACE_AUTH=digest

examples/custom-tokens/user-config/databases/test-database.json

@@ -0,0 +1,4 @@
+{
+  "database-name": "%%TEST_DATABASE_NAME%%"
+}
+

examples/custom-tokens/user-config/servers/test-server.json

@@ -0,0 +1,15 @@
+{
+  "server-name": "%%TEST_SERVER_NAME%%",
+  "server-type": "http",
+  "root": "/",
+  "group-name": "%%GROUP%%",
+  "port": "%%TEST_SERVER_PORT%%",
+  "modules-database": "%%mlModulesDbName%%",
+  "content-database": "%%TEST_DATABASE_NAME%%",
+  "authentication": "%%TEST_TRACE_AUTH%%",
+  "default-error-format": "json",
+  "error-handler": "/MarkLogic/rest-api/error-handler.xqy",
+  "url-rewriter": "/MarkLogic/rest-api/rewriter.xml",
+  "rewrite-resolves-globally": true
+}
+

quick-start/e2e/specs/create/create.ts

@@ -90,6 +90,11 @@ export default function() {
       expect(entityPage.entityEditor.isPresent()).toBe(true);
       entityPage.entityTitle.sendKeys('Order');
       entityPage.saveEntity.click();
+      browser.wait(EC.elementToBeClickable(entityPage.confirmDialogYesButton));
+      expect(entityPage.confirmDialogYesButton.isPresent()).toBe(true);
+      entityPage.confirmDialogYesButton.click();
+      browser.wait(EC.presenceOf(entityPage.toast));
+      browser.wait(EC.stalenessOf(entityPage.toast));
       browser.wait(EC.visibilityOf(entityPage.getEntityBox('Order')));
       expect(entityPage.getEntityBox('Order').isDisplayed()).toBe(true);
       console.log('click edit Order entity');
@@ -113,6 +118,11 @@ export default function() {
       expect(entityPage.entityEditor.isPresent()).toBe(true);
       entityPage.entityTitle.sendKeys('Product');
       entityPage.saveEntity.click();
+      browser.wait(EC.elementToBeClickable(entityPage.confirmDialogYesButton));
+      expect(entityPage.confirmDialogYesButton.isPresent()).toBe(true);
+      entityPage.confirmDialogYesButton.click();
+      browser.wait(EC.presenceOf(entityPage.toast));
+      browser.wait(EC.stalenessOf(entityPage.toast));
       browser.wait(EC.visibilityOf(entityPage.getEntityBox('Product')));
       expect(entityPage.getEntityBox('Product').isDisplayed()).toBe(true);
       console.log('click edit Product entity');
@@ -316,6 +326,11 @@ export default function() {
       entityPage.getPropertyType(lastProperty).element(by.cssContainingText('option', 'string')).click();
       entityPage.getPropertyDescription(lastProperty).sendKeys('remove-prop1 description');
       entityPage.saveEntity.click();
+      browser.wait(EC.elementToBeClickable(entityPage.confirmDialogYesButton));
+      expect(entityPage.confirmDialogYesButton.isPresent()).toBe(true);
+      entityPage.confirmDialogYesButton.click();
+      browser.wait(EC.presenceOf(entityPage.toast));
+      browser.wait(EC.stalenessOf(entityPage.toast));
       browser.wait(EC.visibilityOf(entityPage.getEntityBox('removeEntity')));
       entityPage.toolsButton.click();
       //remove removeEntity entity
@@ -334,6 +349,9 @@ export default function() {
       expect(entityPage.entityEditor.isPresent()).toBe(true);
       entityPage.entityTitle.sendKeys('TestEntity');
       entityPage.saveEntity.click();
+      browser.wait(EC.elementToBeClickable(entityPage.confirmDialogNoButton));
+      expect(entityPage.confirmDialogNoButton.isPresent()).toBe(true);
+      entityPage.confirmDialogNoButton.click();
       browser.wait(EC.visibilityOf(entityPage.getEntityBox('TestEntity')));
       expect(entityPage.getEntityBox('TestEntity').isDisplayed()).toBe(true);
       entityPage.toolsButton.click();
@@ -376,8 +394,6 @@ export default function() {
       entityPage.getPropertyPrimaryKeyColumn(lastProperty).click();
       entityPage.getPropertyType(lastProperty).element(by.cssContainingText('option', 'integer')).click();
       entityPage.getPropertyDescription(lastProperty).sendKeys("this is our primary key");
-
-
       //let's save it now that it's populated
       entityPage.saveEntity.click();
       browser.wait(EC.elementToBeClickable(entityPage.confirmDialogNoButton));