Merge branch 'sync'

Author: sowerstl

README.md

@@ -1,6 +1,6 @@
-# DOECode Web Application
+# DOE CODE Web Application
 
-Consists of the "back-end" services and JAX/RS API calls for DOE Code, to be
+Consists of the "back-end" services and JAX/RS API calls for DOE CODE, to be
 accessed by the front-end or presentation layer.  This application is targeted
 at a non-EE Java container such as Tomcat, using JPA and JAX/RS (Jersey implementation)
 for persistence layer and web API implementation.
@@ -109,7 +109,7 @@ to obtain the output in YAML format.
 
 `POST /services/metadata`
 
-Store a given metadata JSON to the DOECode persistence layer in an incomplete or
+Store a given metadata JSON to the DOE CODE persistence layer in an incomplete or
 pending status.  The resulting JSON information is returned as the JSON object "metadata",
 including the generated unique IDs as appropriate if the operation was successful. Record
 is placed in the "Saved" work flow.
@@ -121,16 +121,16 @@ format.  Does not persist any data.
 
 `POST /services/metadata/publish`
 
-Store the metadata information to the DOECode persistence layer with a "Published"
+Store the metadata information to the DOE CODE persistence layer with a "Published"
 work flow.  JSON is returned as with the "Saved" service above, and this record
-is marked as available to the DOECode search output services.  If DataCite information
+is marked as available to the DOE CODE search output services.  If DataCite information
 has been configured, this step will attempt to register any DOI entered and update
 metadata information with DataCite.
 
 `POST /services/metadata/submit`
 
 Post the metadata to OSTI, attempt to register a DOI if possible, and persist
-the information on DOECode.  If workflow validations pass, the JSON will be returned
+the information on DOE CODE.  If workflow validations pass, the JSON will be returned
 with appropriate unique identifier information and DOI values posted in the
 JSON object "metadata".  Data is placed in "Published" state.
 
@@ -191,11 +191,11 @@ It is often useful to have a simple database for testing that is not your instit
 3) In Eclipse, open the "Database Development" perspective.
 4) Follow the [Eclipse Documentation](http://help.eclipse.org/kepler/index.jsp?topic=%2Forg.eclipse.datatools.common.doc.user%2Fdoc%2Fhtml%2Fasc1229700387729.html) to create a Derby Connector, create a connection profile, and connect to Derby.
 
-In step 4, be sure to select "Derby Client Driver" instead of "Derby Embedded Driver." DOE Code is not currently configured to work with the Embedded driver.
+In step 4, be sure to select "Derby Client Driver" instead of "Derby Embedded Driver." DOE CODE is not currently configured to work with the Embedded driver.
 
 ## Running on AWS
 
-The DOE Code server works well on AWS. For the default RHEL 7 instance, the server can be executed with a Derby database for storing using the following rough steps:
+The DOE CODE server works well on AWS. For the default RHEL 7 instance, the server can be executed with a Derby database for storing using the following rough steps:
 1) Create the instance. Make sure your security group is configured to let the necessary ports through (normally 8080).
 2) SSH into the instance using your key. Issue the following commands to download and install prerequisites including Java, Git, and Derby.
 ```bash
@@ -250,7 +250,7 @@ cd ~/server
 ## SOLR for Searching and Indexing (Dissemination)
 
 If configured in the deployment profile, Apache SOLR may be used for an indexing and
-searching service for DOECode.  Setting up SOLR distribution package version 6.6.0 
+searching service for DOE CODE.  Setting up SOLR distribution package version 6.6.0 
 as a stand-alone service as follows:
 
 1) Download the [SOLR package from Apache](http://http://lucene.apache.org/solr/)
@@ -285,7 +285,7 @@ http://localhost:{port}/solr/doecode/query
 ```
 in order to get JSON results back in expected formats for the dissemination/searching service.  
 
-These values assume that the DOECode back-end is deployed on the same server as the SOLR standalone service.  If not, alter the
+These values assume that the DOE CODE back-end is deployed on the same server as the SOLR standalone service.  If not, alter the
 URL host names and ports appropriately.  In order to terminate the SOLR standalone server, issue the command:
 ```bash
 $ bin/solr stop

SOLR-AWS-NOTES.md

@@ -59,7 +59,7 @@ Once this script runs, SOLR should be up and running on the {port} you specify.
 Creation of the search core and customization
 ---------------------------------------------
 Once SOLR is running, we will need to create a search and index core to use,
-and customize it according to the DOECode specific schema and configuration,
+and customize it according to the DOE CODE specific schema and configuration,
 which are available in the root of https://github.com/doecode/server.
 
 Switch to the newly-created "solr" user and issue the SOLR commands:
@@ -82,7 +82,7 @@ Issue the command to reload the SOLR core with the new configuration.
 [solr] $ curl http://localhost:{port}/solr/admin/cores?action=RELOAD\&core=doecode
 ```
 
-The DOECode SOLR search core should be up and ready to use on the indicated
+The DOE CODE SOLR search core should be up and ready to use on the indicated
 port.  All that remains is to configure the EC2 instance TCP access rules
 to allow inbound connections on the port to the desired group(s) or application
 server IPs for the "server" back-end process, which will allow it to index

pom.xml

@@ -4,22 +4,28 @@
 	<groupId>gov.osti</groupId>
 	<artifactId>doecode</artifactId>
 	<packaging>war</packaging>
-	<version>1.3</version>
-	<name>DOE Code Web Application (Server)</name>
+	<version>1.4.0</version>
+	<name>DOE CODE Web Application (Server)</name>
 	<url>http://maven.apache.org</url>
 
 	<properties>
 		<jdk.version>1.8</jdk.version>
-                <jersey.version>2.22.2</jersey.version>
-                <httpclient.version>4.5.2</httpclient.version>
-                <jackson.version>2.9.0</jackson.version>
-                <projectName>doecodeapi</projectName>
-                <datacite.url>https://mds.test.datacite.org/</datacite.url>
-                <environment>development</environment>
+        <jersey.version>2.22.2</jersey.version>
+        <httpclient.version>4.5.2</httpclient.version>
+        <jackson.version>2.9.0</jackson.version>
+		<commons-io.version>2.6</commons-io.version>
+        <projectName>doecodeapi</projectName>
+        <datacite.url>https://mds.test.datacite.org/</datacite.url>
+        <environment>development</environment>
 	 <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
 	</properties>
- 
+
 	<dependencies>
+		<dependency>
+			<groupId>commons-io</groupId>
+			<artifactId>commons-io</artifactId>
+			<version>${commons-io.version}</version>
+		</dependency>
 		<dependency>
 			<groupId>junit</groupId>
 			<artifactId>junit</artifactId>

schema.xml

@@ -461,6 +461,9 @@
   <field name="contributors.orcid" type="text_en" multiValued="true" indexed="true" stored="true"/>
   <field name="countryOfOrigin" type="text_en" indexed="true" stored="true"/>
   <field name="description" type="text_en" indexed="true" stored="true"/>
+  <field name="programmingLanguages" type="string" multiValued="true" indexed="true" stored="true"/>
+  <field name="versionNumber" type="text_en" indexed="true" stored="true"/>
+  <field name="documentationUrl" type="text_en" indexed="true" stored="true"/>
   <field name="developers.name" type="text_en" multiValued="true" indexed="true" stored="true"/>
   <field name="developers.orcid" type="text_en" multiValued="true" indexed="true" stored="true"/>
   <field name="doi" type="string"/>

src/main/java/gov/osti/connectors/BitBucket.java

@@ -97,11 +97,18 @@ public JsonNode read(String url) {
         try {
             // try to get the METADATA YAML first
             JsonNode results = HttpUtil.readMetadataYaml(BITBUCKET_RAW_URL + name + "/raw/master/metadata.yml");
-
             if (null!=results)
                 return results;
-            // try alternate names
+            // try alternate metadata name
+            results = HttpUtil.readMetadataYaml(BITBUCKET_RAW_URL + name + "/raw/master/.metadata.yml");
+            if (null!=results)
+                return results;
+            // try alternate name
             results = HttpUtil.readMetadataYaml(BITBUCKET_RAW_URL + name + "/raw/master/doecode.yml");
+            if (null!=results)
+                return results;
+            // try alternate doecode name
+            results = HttpUtil.readMetadataYaml(BITBUCKET_RAW_URL + name + "/raw/master/.doecode.yml");
             if (null!=results)
                 return results;
 

src/main/java/gov/osti/connectors/Connector.java

@@ -134,12 +134,20 @@ public JsonNode read(String url) {
             // if it's not empty, use that
             if (null!=yaml)
                 return yaml;
-            // try alternate names
+            // try alternate metadata name
+            yaml = HttpUtil.readMetadataYaml(GITHUB_RAW_BASE_URL + name + "/master/.metadata.yml");
+            if (null!=yaml)
+                return yaml;
+            // try alternate name
             yaml = HttpUtil.readMetadataYaml(GITHUB_RAW_BASE_URL + name + "/master/doecode.yml");
+            if (null!=yaml)
+                return yaml;
+            // try alternate doecode name
+            yaml = HttpUtil.readMetadataYaml(GITHUB_RAW_BASE_URL + name + "/master/.doecode.yml");
             if (null!=yaml)
                 return yaml;
 
-            // acquire the SourceForge API response as JSON
+            // acquire the GitHub API response as JSON
             HttpGet get = gitHubAPIGet(GITHUB_BASE_URL + name);
 
             // Convert the JSON into an Object we can handle

src/main/java/gov/osti/connectors/GitHub.java

@@ -0,0 +1,95 @@
+package gov.osti.connectors;
+
+import gov.osti.connectors.gitlab.Project;
+import gov.osti.connectors.api.GitLabAPI;
+import com.fasterxml.jackson.databind.JsonNode;
+import gov.osti.entity.DOECodeMetadata;
+import java.io.IOException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The GitLab metadata scraper class, to acquire/write relevant data
+ * from/to the GitLab public API (or local OSTI API for OSTI Hosted).
+ *
+ * @author sowerst
+ */
+public class GitLab implements ConnectorInterface {
+
+    /** a logger implementation * */
+    private static final Logger log = LoggerFactory.getLogger(GitLab.class);
+
+    /** authentication information for accessing GitLab API * */
+    /**
+     * Initialize and read the properties for configuration purposes.
+     *
+     * Obtains connector authentication information from properties files.
+     *
+     * @throws IOException on file IO errors
+     */
+    @Override
+    public void init() throws IOException {
+    }
+
+    /**
+     * Obtain the connection-driven metadata elements from GitLab public API
+     * requests, or OSTI GitLab authenticated requests.
+     *
+     * @param url the URL to process
+     *
+     * @return a JsonElement of the DOECodeMetadata filled in as possible from
+     * the API
+     */
+    @Override
+    public JsonNode read(String url) {
+        DOECodeMetadata md = new DOECodeMetadata();
+
+        try {
+            GitLabAPI glApi = new GitLabAPI(url);
+
+            // try to identify the NAME of the project
+            if (null == glApi.getProjectName())
+                return null;
+
+            // try to get the metadata YAML file first
+            JsonNode yaml = HttpUtil.readMetadataYaml(glApi.acquireRawFileGet("metadata.yml"));
+            // if it's not empty, use that
+            if (null != yaml)
+                return yaml;
+            // try alternate metadata name
+            yaml = HttpUtil.readMetadataYaml(glApi.acquireRawFileGet(".metadata.yml"));
+            if (null != yaml)
+                return yaml;
+            // try alternate name
+            yaml = HttpUtil.readMetadataYaml(glApi.acquireRawFileGet("doecode.yml"));
+            if (null != yaml)
+                return yaml;
+            // try alternate doecode name
+            yaml = HttpUtil.readMetadataYaml(glApi.acquireRawFileGet(".doecode.yml"));
+            if (null != yaml)
+                return yaml;
+
+            // Convert the JSON into an Object we can handle
+            Project project = glApi.fetchProject();
+
+            // parse the relevant response parts into Metadata
+            md.setSoftwareTitle(project.getName());
+            md.setAcronym(project.getFullPath());
+            md.setDescription(project.getDescription());
+
+            /*
+             * TODO:
+             * Scrape Contributors/Developers
+             * name/email/affiliations
+             */
+            return md.toJson();
+        } catch (IOException e) {
+            // here's where you'd warn about the IO error
+            log.warn("IO Error reading GitLab information: " + e.getMessage());
+            log.warn("Read from " + url);
+        }
+        // unable to process this one
+        return null;
+    }
+
+}

src/main/java/gov/osti/connectors/GitLab.java

@@ -72,26 +72,43 @@ protected static String fetch(HttpGet get) throws IOException {
             hc.close();
         }
     }
-    
+
     /**
      * Attempt to read URL (file) as a YAML metadata reference.  Returns null
      * if YAML file not found or could not process.
-     * 
+     *
      * @param url the URL to the file to attempt to read
      * @return JSON representation of the YAML read, or null if not found/invalid
      * @throws IOException on file IO errors
      */
     protected static JsonNode readMetadataYaml(String url) throws IOException {
+        try {
+            return readMetadataYaml(new HttpGet(url));
+        } catch ( IOException e ) {
+            // no YAML or illegal format, skip it
+            return null;
+        }
+    }
+
+    /**
+     * Attempt to read URL (file) as a YAML metadata reference.  Returns null
+     * if YAML file not found or could not process.
+     *
+     * @param url the HTTPGET URL to the file to attempt to read
+     * @return JSON representation of the YAML read, or null if not found/invalid
+     * @throws IOException on file IO errors
+     */
+    protected static JsonNode readMetadataYaml(HttpGet url) throws IOException {
         try {
             // read the YAML in as a DOECodeMetadata record
-            DOECodeMetadata yaml = YAML_MAPPER.readValue(HttpUtil.fetch(new HttpGet(url)), DOECodeMetadata.class);
+            DOECodeMetadata yaml = YAML_MAPPER.readValue(HttpUtil.fetch(url), DOECodeMetadata.class);
             return (null==yaml) ? null : yaml.toJson();
         } catch ( IOException e ) {
             // no YAML or illegal format, skip it
             return null;
         }
     }
-    
+
     /**
      * Write the Metadata in YAML format.
      * @param in the DOECodeMetadata to write

src/main/java/gov/osti/connectors/HttpUtil.java

@@ -3,7 +3,7 @@
 These are the currently-supported repositories from which the API will attempt
 to pre-populate information it can glean given the project URL.  Currently the
 information obtained is somewhat sparse, as this is intended to provoke 
-discussion on what elements of metadata we should map to DOECode's schema.
+discussion on what elements of metadata we should map to DOE CODE's schema.
 
 If any of these sources should have a "metadata.yml" file checked-in on the
 master branch, project root level, the Connector will attempt to read from and