Merge branch 'master' of https://github.com/cfpb/hmda-platform

Author: Kibrael

Documents/panel.md

@@ -4,22 +4,29 @@
 The panel loader is designed to read a CSV file and load the data onto the HMDA-Platform.  The CSV file should use the `|` (pipe) delimiter, and should include a header row as the first line.
 
 ## Environment Variables
-There are two environment variables used by the panel loader.  Both must be set correctly in order for the data to be sent to the admin API.
+There is only one environment variable used by the panel loader.  It must be set correctly in order for the data to be sent to the admin API.
 
-For testing locally, no changes need to be made.  The defaults for both of these variables will point to the correct local admin API.
+For testing on an API running in SBT, no changes need to be made.  The default for this variable will point to the correct local admin API.
 
-For loading panel data into a remote system, you'll need to set the following environment variables:
+For loading panel data into a remote system or into a local Docker container, you'll need to set the following environment variable:
 ```shell
-> export HMDA_HTTP_ADMIN_HOST={ip address}
-> export HMDA_HTTP_ADMIN_PORT={port #}
+> export HMDA_HTTP_ADMIN_URL={base URL}
+```
+
+**IMPORTANT NOTE:** The base URL should *include* `http://` or `https://`, but *exclude* any trailing backslash `/`.  For example:
+
+```shell
+> export HMDA_HTTP_ADMIN_URL=http://192.168.99.100:8081
 ```
 
 ## Running the parser
 A small example file is located at `panel/src/main/resources/inst_data_2017_dummy.csv`
 
 The real panel file is located at `panel/src/main/resources/inst_data_2017.csv`
 
-In order for the panel data to be loaded locally, the API project must be up and running, along with Docker containers running Cassandra and Zookeper.  Otherwise, no other running services are needed (but make sure your environment variables are set).  In a terminal, execute the following commands:
+In order for the panel data to be loaded locally, the API project must be up and running, along with Docker containers running Cassandra and Zookeper, or run the full `docker-compose` setup.  To load panel data into the cluster, simply find the URL of the admin api (for the release branch: `https://hmda-ops-api.demo.cfpb.gov/admin`).  No other running services are necessary.
+
+In a terminal, execute the following commands:
 
 ```shell
 > sbt

README.md

@@ -144,6 +144,16 @@ export ZOOKEEPER_HOST=192.168.99.100
 export ZOOKEEPER_PORT=2181
 ```
 
+Alternatively, these dependencies (`Cassandra`, `Zookeeper` and `PostgreSQL`) can be started from `docker` providing default resources for the `HMDA Platform`:
+
+`docker-compose -f docker-dev.yml up`
+
+* If you want to use the sample files in this repo for testing the app, run the edits in demo mode. Otherwise, edit S025 will trigger for all files.
+
+```shell
+export EDITS_DEMO_MODE=true
+```
+
 * Start `sbt`
 
 ```shell
@@ -208,51 +218,124 @@ The Filing API will run on `$(docker-machine ip):8080`
 The Public API will run on `$(docker-machine ip):8082`
 
 #### To run the entire platform
-Clone the [HMDA Platform UI](https://github.com/cfpb/hmda-platform-ui) repo and the [HMDA Platform Auth](https://github.com/cfpb/hmda-platform-auth) repo into sibling directories of this one. Your directory structure should look like this:
-```shell
-~/dev/hmda-project$ ls -la
-total 16
-drwxr-xr-x   6 lortone  staff   204B Jul 25 17:44 ./
-drwxr-xr-x   9 lortone  staff   306B Jul 25 17:50 ../
-drwxr-xr-x  22 lortone  staff   748B Jul 27 16:28 hmda-platform/
-drwxr-xr-x  25 lortone  staff   850B Jul 25 17:13 hmda-platform-ui/
-drwxr-xr-x  23 lortone  staff   796B Jul 28 17:15 hmda-platform-auth/
-```
 
-From the _`hmda-platform-ui`'s_ root directory, run `yarn`. (Get yarn [here](https://yarnpkg.com/lang/en/docs/install/) if you don't have it installed.)
+1. Dedicate appropriate resources to your Docker environment.  We've found
+    that for the full stack to run efficiently, you need approximately:
 
-From _`hmda-platform`'s_ root directory, run the following:
+    * 4 CPUs
+    * 6 GB RAM
+    * 80 GB Disk space
 
-```shell
-sbt clean assembly
-docker-compose up
-```
+    Assuming you are using Docker Machine to provision your Docker
+    environment, you can check you current settings with the following 
+    (ignore the second `Memory`):
 
-This will bring up all the HMDA Platform services. The first run may take several minutes.
+        $ docker-machine inspect | grep 'CPU\|Memory\|DiskSize'
+            "CPU": 4,
+            "Memory": 6144,
+            "DiskSize": 81920,
+            "Memory": 0,
 
-Next, find your docker machine's endpoint.
+    If your settings are below these suggestions, you should create a new
+    Docker VM. The following will create a VM named `hmda-platform` with 
+    the appropriate resources:
 
-```shell
-# Typically defaults to 192.168.99.100, which will be used in the following examples
-docker-machine ip dev
-```
+        $ docker-machine create \
+        --driver virtualbox \
+        --virtualbox-disk-size 81920 \
+        --virtualbox-cpu-count 4 \
+        --virtualbox-memory 6144 \
+        hmda-platform
+
+1. Clone [hmda-platform-ui](https://github.com/cfpb/hmda-platform-ui) and 
+    [hmda-platform-auth](https://github.com/cfpb/hmda-platform-auth) into the same
+    directory as hmda-platform.
+
+        ~/dev/hmda-project$ ls -l
+        drwxr-xr-x  22 lortone  staff   748B Jul 27 16:28 hmda-platform/
+        drwxr-xr-x  25 lortone  staff   850B Jul 25 17:13 hmda-platform-ui/
+        drwxr-xr-x  23 lortone  staff   796B Jul 28 17:15 hmda-platform-auth/
+
+1. Build hmda-platform-ui
+
+        cd hmda-platform-ui && \
+        yarn && \
+        cd ..
+
+    **Note:** This requires [yarn](https://yarnpkg.com/lang/en/docs/install/) to be installed.
+
+1. Build hmda-platform
+
+        cd hmda-platform && \
+        sbt clean assembly
+
+1. Launch the stack with Docker Compose
+
+        docker-compose up
+
+    This will bring up all the HMDA Platform services. The first run may take several minutes.
+
+1. Discover your Docker host's IP
+
+        echo $DOCKER_HOST
+
+    ...or if using Docker Machine...
+
+        docker-machine ip
+
+    **Note:** Docker Machine generally defaults to `192.168.99.100`.  We reference that
+    IP throught this doc.  If your Docker host IP differs, please adjust these instructions
+    to match the Docker host IP provided by your system.
+
+1. Use it!  Below are steps representing a standard HMDA filing:
+
+    1. Browse to the app at http://192.168.99.100.
+    1. Select the "Login" button.  This will redirect your browser to the Keycloak login screen.
+    1. Select "create and account" on the login screen.
+    1. Enter you account information and select "Register".
+
+        **Note:** You must register with an email address from our whitelist of email domains.
+        For convenience, `bank0.com` and `bank1.com` address should be available automatically.
+
+    1. Browse to the mock email server at https://192.168.99.100:8443/mail/, and select the 
+        verification link in the email found there.  This should take you back to the HMDA
+        filing web app, now logged in.
+
+        **Note:** This "MailDev" services is for development purposes only.  In the case of
+        an actual HMDA filing, you would receive a confirmation to your actual email account.
+
+    1. Submit a HMDA filing.  Several sample files can be found [here](https://github.com/cfpb/hmda-platform/tree/master/parser/jvm/src/test/resources/txt).
 
-Then, visit the following URLS and click advanced -> proceed. This will bypass self-signed cert errors from your browser when running the app.
+##### Updating an existing system
 
-- https://192.168.99.100:8443/
-- https://192.168.99.100:4443/
-- https://192.168.99.100:9443/
+If you've updated any of the hmda-platform services, and would like to see those 
+changes reflected in the Docker Compose setup, the simplest way to do this is to
+rebuild everything from scratch.  The following command should be executed from
+within the `hmda-platform` directory.
 
-Visit the app at http://192.168.99.100, click the "Login" button, and click "Register" when redirected to the keycloak login screen.
+    docker-compose stop -t0 && \
+    docker-compose rm -vf && \
+    cd ../hmda-platform-ui && \
+    yarn && \
+    cd ../hmda-platform && \
+    sbt clean assembly && \
+    docker-compose build --no-cache && \
+    docker-compose up
 
-To use demo data there are two institutions available; Bank 0 and Bank 1. To register for either of these institutions you have to use the corresponding domain:
 
-- Bank 0 = bank0.com
-- Bank 1 = bank1.com
+##### Service URLs
 
-Confirm your signup via MailDev by visiting http://192.168.99.100:1080, opening the email, and clicking the verifying link.
+When running the full stack via Docker Compose, the following services are available:
 
-You can now interact with the app/begin uploading files, etc.
+| Service                | URL                                 |
+|------------------------|-------------------------------------|
+| Filing UI              | https://192.168.99.100              |
+| Filing API (Unsecured) | http://192.168.99.100:8080          |
+| Filing API (Secured)   | https://192.168.99.100:4443/hmda/   |
+| Admin API              | http://192.168.99.100:8081          |
+| Public API             | https://192.168.99.100:4443/public/ |
+| Keycloak               | https://192.168.99.100:8443         |
+| MailDev                | https://192.168.99.100:8443/mail/   |
 
 #### Development conveniences
 

api/src/main/resources/application.conf

@@ -62,12 +62,29 @@ hmda {
     timeout = 10
     timeout = ${?HMDA_HTTP_TIMEOUT}
   }
-  isDemo = true
+  isDemo = false
   isDemo = ${?HMDA_IS_DEMO}
   zookeeperHost = "192.168.99.100"
   zookeeperHost = ${?ZOOKEEPER_HOST}
   zookeeperPort = 2181
   zookeeperPort = ${?ZOOKEEPER_PORT}
 }
 
+api-dispatcher {
+  type = Dispatcher
+  executor = "fork-join-executor"
+  fork-join-executor {
+    # Min number of threads to cap factor-based parallelism number to
+    parallelism-min = 2
+    # Parallelism (threads) ... ceil(available processors * factor)
+    parallelism-factor = 2.0
+    # Max number of threads to cap factor-based parallelism number to
+    parallelism-max = 10
+  }
+  # Throughput defines the maximum number of messages to be
+  # processed per actor before the thread jumps to the next actor.
+  # Set to 1 for as fair as possible.
+  throughput = 100
+}
+
 

api/src/main/scala/hmda/api/HmdaFilingApi.scala

@@ -1,15 +1,14 @@
 package hmda.api
 
-import akka.actor.{ ActorRef, ActorSystem, Props }
+import akka.actor.{ ActorSystem, Props }
 import akka.event.Logging
 import akka.http.scaladsl.Http
 import akka.http.scaladsl.Http.ServerBinding
 import akka.http.scaladsl.server.Directives._
 import akka.http.scaladsl.server.Route
-import akka.pattern.{ ask, pipe }
+import akka.pattern.pipe
 import akka.stream.ActorMaterializer
 import akka.util.Timeout
-import com.typesafe.config.ConfigFactory
 import hmda.api.http.{ BaseHttpApi, HmdaCustomDirectives, InstitutionsHttpApi, LarHttpApi }
 import hmda.api.HmdaConfig._
 import scala.concurrent.duration._

api/src/main/scala/hmda/api/HmdaPlatform.scala

@@ -24,7 +24,6 @@ import hmda.query.projections.filing.HmdaFilingDBProjection._
 import hmda.validation.ValidationStats
 import hmda.api.HmdaConfig._
 import hmda.query.HmdaProjectionQuery
-import hmda.query.projections.institutions.InstitutionCassandraProjection
 
 object HmdaPlatform {
 
@@ -35,7 +34,7 @@ object HmdaPlatform {
     val system = ActorSystem(configuration.getString("clustering.name"), configuration)
     val supervisor = createSupervisor(system)
     val querySupervisor = createQuerySupervisor(system)
-    implicit val ec = system.dispatcher
+    implicit val ec = system.dispatchers.lookup("api-dispatcher")
 
     startActors(system, supervisor, querySupervisor)
     startApi(system, querySupervisor)

api/src/main/scala/hmda/api/http/ValidationErrorConverter.scala

@@ -6,7 +6,6 @@ import hmda.model.edits.EditMetaDataLookup
 import hmda.model.fi.{ HmdaFileRow, HmdaRowError }
 import hmda.model.validation.ValidationError
 import hmda.persistence.processing.HmdaFileValidator.HmdaFileValidationState
-import hmda.validation.engine._
 import spray.json.{ JsNumber, JsObject, JsString, JsValue }
 
 trait ValidationErrorConverter {

api/src/main/scala/hmda/api/http/institutions/FilingPaths.scala

@@ -19,7 +19,7 @@ import hmda.persistence.HmdaSupervisor.{ FindFilings, FindSubmissions }
 import hmda.persistence.institutions.FilingPersistence.GetFilingByPeriod
 import hmda.persistence.institutions.{ FilingPersistence, SubmissionPersistence }
 
-import scala.concurrent.{ ExecutionContext, Future }
+import scala.concurrent.Future
 import scala.util.{ Failure, Success }
 
 trait FilingPaths extends InstitutionProtocol with ApiErrorProtocol with HmdaCustomDirectives {

api/src/main/scala/hmda/api/http/institutions/InstitutionPaths.scala

@@ -22,7 +22,7 @@ import hmda.persistence.model.HmdaSupervisorActor.FindActorByName
 import hmda.query.view.institutions.InstitutionView
 import hmda.query.view.institutions.InstitutionView.{ GetInstitutionById, GetInstitutionsById }
 
-import scala.concurrent.{ ExecutionContext, Future }
+import scala.concurrent.Future
 import scala.util.{ Failure, Success }
 
 trait InstitutionPaths extends InstitutionProtocol with ApiErrorProtocol with HmdaCustomDirectives {

api/src/main/scala/hmda/api/http/institutions/UploadPaths.scala

@@ -28,7 +28,6 @@ import hmda.persistence.processing.SubmissionManager
 import hmda.query.HmdaQuerySupervisor.FindHmdaFilingView
 import hmda.query.projections.filing.HmdaFilingDBProjection.{ CreateSchema, DeleteLars }
 
-import scala.concurrent.ExecutionContext
 import scala.util.{ Failure, Success }
 
 trait UploadPaths extends InstitutionProtocol with ApiErrorProtocol with SubmissionProtocol with HmdaCustomDirectives {

api/src/main/scala/hmda/api/http/institutions/submissions/SubmissionBasePaths.scala

@@ -12,15 +12,11 @@ import akka.util.Timeout
 import hmda.api.http.{ HmdaCustomDirectives, ValidationErrorConverter }
 import hmda.api.model._
 import hmda.api.protocol.processing.{ ApiErrorProtocol, EditResultsProtocol, InstitutionProtocol }
-import hmda.model.fi.{ Filing, Submission, SubmissionId }
-import hmda.model.validation.Macro
-import hmda.persistence.messages.CommonMessages.GetState
-import hmda.persistence.HmdaSupervisor.{ FindFilings, FindProcessingActor, FindSubmissions }
+import hmda.model.fi.{ Filing, Submission }
+import hmda.persistence.HmdaSupervisor.{ FindFilings, FindSubmissions }
 import hmda.persistence.institutions.FilingPersistence.GetFilingByPeriod
 import hmda.persistence.institutions.SubmissionPersistence.{ CreateSubmission, GetLatestSubmission }
 import hmda.persistence.institutions.{ FilingPersistence, SubmissionPersistence }
-import hmda.persistence.processing.HmdaFileValidator
-import hmda.persistence.processing.HmdaFileValidator.HmdaFileValidationState
 
 import scala.concurrent.ExecutionContext
 import scala.util.{ Failure, Success }