Merge pull request cfpb#1037 from nickgrippin/panel-docs

Update panel docs and error messages

Author: hkeeler

Documents/panel.md

@@ -1,17 +1,29 @@
-# Panel CSV Parser
+# Panel CSV Loader
 
 ## Overview
-The panel parser is designed to read a CSV file and load the data onto the HMDA-Platform.  The CSV file should use the `|` (pipe) delimiter, and should include a header row as the first line.
+The panel loader is designed to read a CSV file and load the data onto the HMDA-Platform.  The CSV file should use the `|` (pipe) delimiter, and should include a header row as the first line.
+
+## Environment Variables
+There are two environment variables used by the panel loader.  Both must be set correctly in order for the data to be sent to the admin API.
+
+For testing locally, no changes need to be made.  The defaults for both of these variables will point to the correct local admin API.
+
+For loading panel data into a remote system, you'll need to set the following environment variables:
+```shell
+> export HMDA_HTTP_ADMIN_HOST={ip address}
+> export HMDA_HTTP_ADMIN_PORT={port #}
+```
 
 ## Running the parser
-An example panel file is located at `hmda-platform/panel/src/main/resources/inst_data_2017_dummy.csv`
+A small example file is located at `panel/src/main/resources/inst_data_2017_dummy.csv`
 
-In order for the panel data to be loaded, the API project must be up and running, along with Docker containers running Cassandra and Zookeper.  In a terminal, execute the following commands:
+The real panel file is located at `panel/src/main/resources/inst_data_2017.csv`
+
+In order for the panel data to be loaded locally, the API project must be up and running, along with Docker containers running Cassandra and Zookeper.  Otherwise, no other running services are needed (but make sure your environment variables are set).  In a terminal, execute the following commands:
 
 ```shell
 > sbt
 sbt> project panel
-sbt> clean
 sbt> run /path/to/panelData.csv
 ```
 
@@ -21,6 +33,15 @@ sbt> clean
 sbt> assembly
 ```
 Then the panel loader can be run with `java -jar  panel/target/scala-2.12/panel.jar path/to/institution_file.csv`
+
+## Error codes
+There are four ways the panel loader can fail.  The exit code and error message should tell you what happened.
+
+1. There were no command line arguments passed to the loader
+2. The path passed to the loader didn't point to a file
+3. The call to `institutions/create` didn't return the correct response.  This can indicate that you don't have the correct environment variables set, or that something is wrong with the hmda-platform.
+4. The loader didn't finish processing all the institutions.  This will happen when running the real panel file, but unsure as to why this happens.
+
 ## Testing
 Make sure your authorization header is updated with a few real `id_rssd` fields from the given file.  This can be found in the API log output (first field argument in the `InstitutionQuery` object), or in the CSV file (seventh field).
 

README.md

@@ -104,7 +104,7 @@ $ sbt
 project panel
 run <full local path to sample file>
 ```
-A sample file is located in the following folder: `hmda-platform/persistence/src/main/resources/demoInstitutions.csv`
+A sample file is located in the following folder: `panel/src/main/resources/inst_data_2017_dummy.csv`
 
 
 * In order to support the read side, a local PostgreSQL and Cassandra server are needed. Assuming it runs on the default port, on the same machine as the API, the following environment variable needs to be set:

panel/src/main/scala/hmda/panel/PanelCsvLoader.scala

@@ -2,11 +2,11 @@ package hmda.panel
 
 import java.io.File
 
-import akka.{ Done, NotUsed }
+import akka.NotUsed
 import akka.actor.ActorSystem
 import akka.util.{ ByteString, Timeout }
 import akka.stream.ActorMaterializer
-import akka.stream.scaladsl.{ FileIO, Flow, Framing, Sink, Source }
+import akka.stream.scaladsl.{ FileIO, Flow, Framing, Sink }
 import akka.http.scaladsl.Http
 import akka.http.scaladsl.model.{ HttpEntity, _ }
 import akka.http.scaladsl.unmarshalling.Unmarshal
@@ -18,8 +18,6 @@ import hmda.parser.fi.InstitutionParser
 import org.slf4j.LoggerFactory
 import spray.json._
 
-import scala.util.Try
-
 object PanelCsvLoader extends WriteInstitutionProtocol {
   implicit val system: ActorSystem = ActorSystem("hmda")
   implicit val materializer = ActorMaterializer()
@@ -96,6 +94,7 @@ object PanelCsvLoader extends WriteInstitutionProtocol {
 
   private def exitSys(errorMessage: String, code: Int) = {
     log.error(errorMessage)
+    Thread.sleep(100)
     sys.exit(code)
   }
 }