Merge branch 'develop'

Author: paxtonhare

.travis.yml

@@ -11,6 +11,7 @@ services:
   - docker
 
 before_install:
+ - "/sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -ac -screen 0 1280x1024x16"
  - ./.travis/download.sh
 
 script:

CONTRIBUTING.md

@@ -25,14 +25,14 @@ You need these to get started
 
 - Java 8 JDK
 - Gradle (3.4 or greater)
-- Node JS 6.5 or newer
-- Typings `npm -g install typings`
 - A decent IDE. IntelliJ is nice.
 
 #### Building from the command line
 **First, a warning.** _The DHF has a ton of tests and they take a very long time to run. Considering you might not want to invest 30 minutes to wait for tests these instructions will show you how to skip the tests._
 
-To build the entire DHF (marklogic-data-hub.jar, quickstart.war, and ml-data-hub-plugin for gradle) simply run this command:
+**Do you need to do this?** - only if you are wanting to build the entire DHF final products (marklogic-data-hub.jar, quickstart.war, and ml-data-hub-plugin for gradle)
+
+Simply run this command:
 
 ```bash
 cd /path/to/data-hub-project/
@@ -41,9 +41,11 @@ cd /path/to/data-hub-project/
 
 #### Making Changes to the Hub Gradle Plugin
 
-This is for when you are making changes to the ml-data-hub-plugin. This is a gradle plugin that enables Hub Capabilities. Most likely you won't find yourself doing this. You can safely ignore this section.
+This is for when you really want to use a local copy of the Gradle Plugin in your Data Hub Framework Project. Perhaps you are testing out a change to the ml-data-hub Gradle plugin or you have a cutting edge development version. There are very few valid reasons for you to do this.
+
+Still here? Seems you really want to use a local copy of the Gradle Plugin in your Data Hub Framework Project. Here's how to tell Gradle to use your local copy instead of the one living up on the Cloud.
 
-Still here? Seems you really want to modify the Gradle Plugin. Here's how to tell Gradle to use your local copy instead of the one living up on the Cloud.
+First you must publish your Data Hub Plugin to the local maven repository.
 
 ```bash
 cd /path/to/data-hub-project/
@@ -52,7 +54,7 @@ cd /path/to/data-hub-project/ml-data-hub-plugin
 ./gradlew publishToMavenLocal
 ```
 
-Then in your build.gradle file you will need to use the local version:
+Then in your DHF project's build.gradle file you will need to use the local version:
 ```groovy
 
 // this goes at the top above the plugins section
@@ -78,6 +80,20 @@ plugins {
 apply plugin: "com.marklogic.ml-data-hub"
 ```
 
+To run the plugin's unit tests, cd to the ml-data-hub-plugin directory, then:
+
+Run all unit tests
+
+    ../gradlew test
+
+
+Run one unit test
+
+    ../gradlew -Dtest.single=CreateEntityTask test
+
+
+**Note**: This change goes in a DHF project's build.gradle. Not the DHF source code's build.gradle.
+
 #### Running the QuickStart UI from source
 Make sure you have the prerequisites installed.
 
@@ -89,6 +105,12 @@ cd /path/to/data-hub-project
 ./gradlew bootrun
 ```
 
+**BE AWARE** There will be a progress indicator that stops around 90%. This is normal. In gradle land, 100% means it finished running. This stays running indefinitely and thus shows 90%.
+
+```
+> Building 90% > :quick-start:bootRun
+```
+
 **Terminal window 2** - This runs the Quickstart UI
 ```
 cd /path/to/data-hub-project
@@ -97,6 +119,65 @@ cd /path/to/data-hub-project
 
 Now open your browser to [http://localhost:4200](http://localhost:4200) to use the debug version of the Quickstart UI.
 
+### Troubleshooting
+Did the `gradle runui` command fail for you? Here's a quick checklist to troubleshoot.
+
+#### Do you have Gradle 3.4 or newer?
+Using straight up gradle:
+```
+gradle -v
+```
+or if you are using the wrapper:
+```
+./gradlew -v
+```
+If your gradle wrapper is older than `3.4` then do this:
+```
+gradle wrapper --gradle-version 3.4
+```
+#### Are you on the develop branch?
+_hint: you should be_  
+Check like so:
+```bash
+git branch
+```
+
+To switch to the develop branch:
+```bash
+git checkout develop
+```
+
+#### Do you have the latest code?
+Better make sure...
+
+##### You clone from the github.com/marklogic-community/marklogic-data-hub repo
+
+```bash
+git pull origin develop
+```
+##### Your forked then cloned your fork
+Make sure you have the upstream set:
+```bash
+$ git remote add upstream git://github.com/marklogic-community/marklogic-data-hub.git
+```
+
+Then fetch the upstream:
+```bash
+git fetch upstream develop
+```
+
+Now merge it in:
+```bash
+git rebase upstream/develop
+```
+
+#### Try removing the `quick-start/node_modules` directory.  
+If you are seeing a bunch of javascript errors you might have a messed up node_modules directory. Try to remove it then run again.
+
+```bash
+rm -rf quick-start/node_modules
+```
+
 ## Submission Guidelines
 
 ### Submitting an Issue
@@ -193,6 +274,19 @@ Make sure the JUnit tests pass.
 $ ./gradlew test
 ```
 
+If you want to run a single test:
+
+```sh
+$ ./gradlew -Dtest.single=TestName test
+```
+
+for best results don't include the final word Test. Example:
+Say you want to run FlowRunnerTest.
+
+```sh
+$ ./gradlew -Dtest.single=FlowRunner test
+```
+
 Make sure that all tests pass. Please, do not submit patches that fail.
 
 #### Push your changes

examples/README.md

@@ -0,0 +1,11 @@
+# Data Hub Framework Examples
+
+This folder contains working examples of various DHF usage scenarios.
+
+1. [barebones](https://github.com/marklogic-community/marklogic-data-hub/tree/master/examples/barebones) - an example of the minimum configuration necessary to run a Gradle based Data Hub
+1. [healthcare](https://github.com/marklogic-community/marklogic-data-hub/tree/master/examples/healthcare) - an example of a Healthcare 360 Data Hub
+1. [hr-hub](https://github.com/marklogic-community/marklogic-data-hub/tree/master/examples/hr-hub) - an example used for our 1.x tutorial. This example harmonizes data from various HR systems
+1. [load-binaries](https://github.com/marklogic-community/marklogic-data-hub/tree/master/examples/load-binaries) - an example of how to ingest binaries via an MLCP Input Flow
+1. [online-store](https://github.com/marklogic-community/marklogic-data-hub/tree/master/examples/online-store) - the example we use for our [Tutorial](https://marklogic-community.github.io/marklogic-data-hub/tutorial/)
+1. [spring-batch](https://github.com/marklogic-community/marklogic-data-hub/tree/master/examples/spring-batch) - an example of how to load relational data into a Data Hub using Spring Batch
+1. [ssl](https://github.com/marklogic-community/marklogic-data-hub/tree/master/examples/ssl) - an example of how to configure your Data Hub to use SSL for added security

examples/ssl/README.md

@@ -0,0 +1,108 @@
+# SSL Example
+This example demonstrates how to enable SSL for your entire MarkLogic instance. This includes the builtin AppServers and your Data Hub Appservers.
+
+For additional information on SSL and Appservers see [Chapter 8 of the Security Guide](http://docs.marklogic.com/guide/security/SSL).
+
+# What's required
+### A certificate template
+We provide one for you in `user-config/security/certificate-templates/my-template.xml`
+```xml
+<certificate-template-properties xmlns="http://marklogic.com/manage">
+  <template-name>dhf-cert</template-name>
+  <template-description>Sample description</template-description>
+  <key-type>rsa</key-type>
+  <key-options />
+  <req>
+    <version>0</version>
+    <subject>
+      <countryName>US</countryName>
+      <stateOrProvinceName>VA</stateOrProvinceName>
+      <localityName>McLean</localityName>
+      <organizationName>MarkLogic</organizationName>
+      <organizationalUnitName>Consulting</organizationalUnitName>
+      <emailAddress>[email protected]</emailAddress>
+    </subject>
+  </req>
+</certificate-template-properties>
+```
+
+### Set the certificate Template for each appserver
+We do this for you in 
+`user-config/servers/*-server.json`
+
+```json
+{
+  "ssl-certificate-template": "dhf-cert"
+}
+```
+
+### Enable SSL properties
+We do this for you in `gradle.properties`
+
+```
+# To use SSL on port 8001
+mlAdminScheme=https
+mlAdminSimpleSsl=true
+
+# To use SSL on port 8002
+mlManageScheme=https
+mlManageSimpleSsl=true
+
+# To use SSL on port 8000
+mlAppServicesSimpleSsl=true
+
+# Set these to true to use SSL for you Hub App Servers
+mlStagingSimpleSsl=true
+mlFinalSimpleSsl=true
+mlTraceSimpleSsl=true
+mlJobSimpleSsl=true
+```
+
+# TLDR; How do I run it?
+
+### Step 1: Configure the `gradle.properties` files
+Open `gradle.properties` and put in your username and password
+next to:
+
+```properties
+mlUsername=
+mlPassword=
+```
+
+### Step 2: Enable SSL for builtin AppServers
+First you should enable SSL for the builtin Appservers. You can do this manually via the instructions in [Chapter 8 of the Security Guide](http://docs.marklogic.com/guide/security/SSL) or you can use the shortcut we provide in this example.
+
+To use the shortcut simply run:
+
+```bash
+gradle enableSSL
+```
+
+### Step 3: Initialize your Hub Project
+To initialize the Hub project simply type:
+
+```bash
+gradle hubInit
+```
+
+### Step 4: Deploy the Data Hub into MarkLogic
+To deploy your Data Hub into MarkLogic simply type:
+
+```bash
+gradle mlDeploy
+```
+
+### Step 5: Deploy your custom modules
+```bash
+gradle mlLoadModules
+```
+
+### Scaffolding Commands
+You can use scaffolding commands to configure flows.
+
+```bash
+gradle hubCreateInputFlow
+gradle hubCreateHarmonizeFlow
+```
+
+For a complete list of gradle tasks, check here: [https://github.com/marklogic-community/marklogic-data-hub/wiki/Gradle-Tasks](https://github.com/marklogic-community/marklogic-data-hub/wiki/Gradle-Tasks)