Resolve competing web module war definitions

Date	July 7 2020	Contacts	Jody Garnett
Status	proposed	Release	3.12
Resources		Ticket #	pending
Source code	pending
Funding	GeoCat, GeoNetwork Enterprise and professional services

Overview

The web app contains competing build configurations (for both war:war and jetty:run) on the management of schema-plugins leading to build instability.

This proposal refactors web module structure to resolve this issue.

This proposal should simplify the build experience for everyone, and preserve the developer workflow you are used to, and document the result so it is easier to understand where to make any changes.

Technical Details:

Initially proposed as a simple change to take advantage public OSGeo repository; this change has shown several limitations of the core-geonetwork build system for schema-plugins:

• Use of schema-plugin version 3.7 across different branches of core-geonetwork, combined with transitive dependencies, ensure conflicts across active branches of core-geonetwork.

• web app build copies contents folder to folder, rather than make use of schema plugin artifact zip and jar.

  • The web app war and jetty configurations actually directly watch some folders across module boundaries (sidestepping the distinction between copying folder content or unpacking zip contents). This duplication gives some uncertainty when a change is noticed, and how to reliability pick up a change.

• web app contains competing build configurations (for both war:war and jetty:run) on the management of schema-plugins leading to build instability

• See "baseline" heading further down this page for a detailed review of the current build system

The above limitations result in an unstable build environment.

References:

• Bolsena 2020 Online Slides

• https://github.com/geonetwork/core-geonetwork/pull/4701

Proposal: Refactor into distinct modules and setup seperate war and jetty staging

This approach:

• Any required content is staged into a distinct target folder by process-resources

• Respect that target/geonetwork is reserved for use by the war plugin (and jetty:run-war testing) and never stage content here directly

• Make jetty:run a new folder for data_directory not in src (which is static) or target (which is subject to mvn clean).

• Focuses on locking down the war definition using unpack to extract distinct dependencies zip artifacts managed by maven (with no copying between folders).

  • Use of distinct target folders
  • Use of a custom war assembly, if required, to produce exactly the war we want.
  • There is an opportunity to use jetty:run-war to test the war build chain. While this would respond to mvn process-resources, it would not be intended for interactive development.

• Allow the jetty configuration to work across folders

  • In this workflow web-ui resources, and schema plugins folders would be used directly, no process-resources required.
  • Care taken to only use the maven dependencies to take advantage of jetty ability to watch the maven reactor and notice mvn install being used.
  • If this approach works using jetty:run would be quicker and just focused on rapid development, there would not be any need to use process-resources (just update files directly for html, javascript and xslt, or use mvn install for java files).

Discussion:

• This proposal has the advantage of clearly supporting the two distinct workflows enjoyed by geonetwork developers.

  This represent an improvement for developers as process-resources is no longer be required to pick up changes. However there is a duplication of effort, and there may be some frustration testing the war bundle as a result.

  Example:

  1. If you were using jetty:run and process-resources to try out a schematron rule change
  2. And then packaged a war for a customer
  3. You would of missed out including your rule change in the war

  To fix:

  4. Use mvn install to package the schema plugin into jar and zip files in the local maven repository
  5. And then package war for customer (it will pick up the rule change which is provided by schema plugin zip artifact)

• Consider use of web context to replace use of filters, have multiple jetty-context.xml files to test different dev, prod, inspire.

  The use of jetty:run is primarily used with dev filter. The war:war makes use of prod and inspire filters are used to produce distinct war bundles.

  This is a larger discussion, offering greater control to administrators, and will need to be the subject of its own proposal.

• Q: The gn database is created in the current directory, can this move to target folder?

  A: The only reason to keep in sources can be to avoid loosing the info with mvn clean, but maybe then can be configured a profile to externalize the data directory?

• Q: Is it desirable to keep data geonetwork and gn database?

  A: Yes, see above.

• How to handle schema plugins sch files being compiled into xslt files when GeoNetwork first starts up?

  If jetty:run is using schema plugin directly to ideally generate xlsx into a different location to avoid both the risk of accidental commit and accidentally including these files is generated war.

migrate

migrate module: a new module to package database migration into jar and zip artifacts.

• src/main/java
• src/main/resources

web-ui

Care needed to ensure zip artifact produced:

• src/main/resources

web

The package-resources and compile stages use:

• src/main/filters
• src/main/java
• src/main/resources
• src/main/webapp
• src/main/data
• src/main/webResources
• target/classes
• target/doc - copied docs
• target/schemas - schema plugins updated by mvn process-resources
• target/webGenerated - generated webapp contents from filters and webResources

The war configuration uses:

• src/assembly - custom war assembly to include license and avoid accidentally packaging contents
• src/main/resources
• src/main/webapp
• src/main/data
• target/classes
• target/doc
• target/schemas - schema plugins updated by mvn process-resources
• target/webGenerated
• target/migrate - unpacked content from migrate module
• target/geonetwork/WEB-INF/lib/ - gn-web jar and dependencies staged here

The jetty configuration uses:

• src/main/resources
• src/main/webapp
• test/java/org/fao/geonet/Start.java
• test/resources/jetty-context.xml - these dev settings (assume inspire or prod not needed here)
• test/webGenerated - generated webapp contents from filters/dev-env.properties and webResources
• target/classes
• data_directory/data/ - created from src/main/data for use by jetty
• ../web-ui/src/main/resources - direct use of web-ui files
• ../migrate/src/main/resources - direct use of sql files
• ../schemas/../**/plugin/ - direct use of schema plugins folders

Baseline GeoNetwork 3.10 web/pom.xml build process

The web module has quite a complex build chain supporting:

• war creation of a geonetwork.war that includes schema plugins, a default data directory, and some environment specific customizations

• jetty:run supporting something like:

  • immediate recognition of changes to html files and javascript files
  • delayed recognition of changes to XSLT files (using mvn process-resources)
  • creation of gn h2 database in the web folder
  • direct use of data directory in src/main/webapp

Background on Build Life Cycle

For this discussion keep in mind the maven build life cycle (common steps in italic, default war stages described):

• validate
• initialize
• generate-sources
• process-sources
• generate-resources
• process-resources - resources:resource goal
• compile - compiler:compile goal
• process-classes
• generate-test-sources
• process-test-sources
• generate-test-resources
• process-test-resources - resources:testResources goal
• test-compile - compiler:testCompile goal
• process-test-classes
• test - surefire:test goal
• prepare-package
• package - war:war goal
• pre-integration-test
• integration-test
• post-integration-test
• verify
• install - install:install goal
• deploy - deploy:deploy goal

Each pom packaging type (pom, jar, war, ...) registers maven plugins to specific stages of the maven life cycle, along with appropriate plugin configuration. As an example war plugin configuration defines a default assembly definition that includes src/webapp.

References:

• http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Lifecycle_Reference
• https://maven.apache.org/plugins/maven-war-plugin/index.html
• http://maven.apache.org/guides/introduction/introduction-to-the-standard-directory-layout.html

src folders

The war packaging for the module defines several src folders:

• src/main/java

  Define Geonetwork and other classes (compiled into target/classes)

• src/main/resources

  Define resources (compiled into target/classes)

• src/webapp

  Static outline of war contents.

  • conversion

  • doc

  • htmlCache

  • images

  • loc - translations

  • resources

  • WEB-INF

    • web.xml

  • xml

  • xsl

  • xslt

  • geonetwork.css

  • modalbox.css

The build generate

• src/main/filters

  Used to process webResources into target/webapp

• src/webResources

  • WEB-INF

  Content processed into target/webapp

The generate-sources registers two additional src folders:

• src/main/java - add-src folder containing Geonetwork class
• src/webapp/main/webapp/WEB-INF/classes/setup/sql/migrate - add-src folder for sql migration

process-resources

The process-resources stages content into:

• src/main/webapp/WEB-INF/data/config/schema_plugins

  Collected from schema plugin content

• src/webapp/doc - from documentation manuals

• target/webapp

  Copy filter src/main/webResource based on env configuration

compile

The compile stage builds:

• target/classes - from java and resources

package

The maven-war-plugin generates jar:

• target/geonetwork/WEB-INF/lib/web-app-3.11.0-SNAPSHOT.jar

  Containing:

  • src/main/java compiled classes

  • src/main/resources contents

  • src/webapp/main/webapp/WEB-INF/classes/setup/sql/migrate compiled classes

    This is unusual having java source code in src/webapp!

The maven-war-plugin generates war structure into target/geonetwork:

• catalog
• conversion
• doc
• htmlCache
• images
• loc
• META-INF
• resources
• WEB-INF
• xml
• xsl
• xslt
• modalbox.css
• geonetwork.css

Where content is collected from:

• target/geonetwork: src/main/webapp files

• target/geonetwork: target/webapp files that have been processed

• target/geonetwork: ../web-ui/src/main/resources files

• target/geonetwork/WEB-INF/data/config/schema_plugins: ../schemas files from each plugin folder

• target/geonetwork/WEB-INF/lib/: maven dependencies

• with many excludes ...

  • xml/schemas/
  • WEB-INF/data/*.db
  • WEB-INF/data/index/**
  • ...

  These appear to be a safety measure to avoid including files produced when src/main/webapp is used as a live directory by jetty below.

Observations:

• ⚠️ The handling of sql is very unusual

• The processing (using filters) of webResources appears to duplicate functionality that should be controlled by web container.

  • Some work put into making the application configurable should remove the need to make custom wars for different environments.
  • If needed for local testing a jetty context for each of these configurations would be a clean approach

• ⚠️ Working with the data directory in src/main/webapp results in lots of strange workarounds:

  • Excludes in the war definition to avoid packaging "running" content
  • Not quite documented process to "reset" your environment if you want a clean start

• ⚠️ : war definition including content form web-ui and schema plugins directly, while process-resource includes the same content

      <plugin>
        <artifactId>maven-war-plugin</artifactId>
        <configuration>
          <archive>
            <manifest>
              <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
            </manifest>
            <manifestEntries>
              <Implementation-Build>${git.commit.id}</Implementation-Build>
            </manifestEntries>
          </archive>
          <archiveClasses>true</archiveClasses>
          <!-- Filter resources according to current profile properties
              (see src/main/filters) and copy them to the webapp -->
          <webResources>
            <resource>
              <directory>${project.basedir}/../schemas</directory>
              <includes>
                <include>**/src/main/plugin/*</include>
              </includes>
              <targetPath>WEB-INF/data/config/schema_plugins</targetPath>
            </resource>
            <resource>
              <directory>${project.basedir}/../web-ui/src/main/resources</directory>
            </resource>
            <resource>
              <directory>${build.webapp.resources}</directory>
            </resource>
          </webResources>
          <!-- <packagingExcludes>WEB-INF/data/**</packagingExcludes> -->
          <packagingExcludes>
            xml/schemas/**,
            WEB-INF/data/*.db,
            WEB-INF/data/index/**,
            WEB-INF/data/data/backup/**,
            WEB-INF/data/data/resources/htmlcache/**,
            catalog/lib/style/bootstrap/docs/**,
            catalog/lib/style/bootstrap/fonts/**,
            catalog/lib/style/bootstrap/grunt/**,
            catalog/lib/style/bootstrap/test-infra/**,
            catalog/lib/style/font-awesome/css/**,
            catalog/lib/style/font-awesome/src/**,
            catalog/lib/style/font-awesome/scss/**,
            catalog/lib/style/font-awesome/*.json,
            catalog/lib/style/font-awesome/Gemfile*,
            catalog/lib/style/font-awesome/.*,
            catalog/lib/style/font-awesome/*.txt,
            catalog/lib/style/font-awesome/*.md,
            catalog/lib/style/bootstrap/*.json,
            catalog/lib/style/bootstrap/Gemfile*,
            catalog/lib/style/bootstrap/.*,
            catalog/lib/style/bootstrap/*.txt,
            catalog/lib/style/bootstrap/*.md,
            catalog/lib/bootstrap-table/*.json,
            catalog/lib/bootstrap-table/Gemfile*,
            catalog/lib/bootstrap-table/.*,
            catalog/lib/bootstrap-table/*.txt,
            catalog/lib/bootstrap-table/*.md,
            catalog/lib/bootstrap-table/docs/**
          </packagingExcludes>
          <!--          <warSourceDirectory>src/main/geonetwork</warSourceDirectory> -->
          <webXml>${build.webapp.resources}/WEB-INF/web.xml</webXml>
          <attachClasses>true</attachClasses>
          <warName>${application.name}</warName>
          <webappDirectory>${project.build.directory}/geonetwork</webappDirectory>
        </configuration>
      </plugin>

jetty

The jetty-maven-plugin defines web application using:

• maven dependencies
• target/geonetwork
• src/main/webapp
• ../web-ui/src/main/resources/
• target/webapp
• jett-context.xml: used to prevent jetty scanning jars during startup

Observations:

• This configuration has been setup to use the "live" src/main/webapp location and pick up any changes each time mvn process-resources is called above.

• ⚠️ schemas plugin folders are included twice:

  • in target/geonetwork via war definition
  • in src/main/webapp/ via process-resource copy

• ⚠️ jars are included twice:

  • target/geonetwork libs folder
  • as maven dependency
  • example: An error starting up (where domain jar has a conflict between jar included in libs and included by maven)

      <plugin>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-maven-plugin</artifactId>
        <configuration>
          <contextXml>${basedir}/jetty-context.xml</contextXml>
          <webAppSourceDirectory>${project.build.directory}/geonetwork</webAppSourceDirectory>
          <webApp>
            <contextPath>/${application.name}</contextPath>
            <descriptor>${project.build.directory}/WEB-INF/web.xml</descriptor>
            <baseResource implementation="org.eclipse.jetty.util.resource.ResourceCollection">
              <resourcesAsCSV>
                ${project.basedir}/src/main/webapp,
                ${rootProjectDir}/web-ui/src/main/resources/,
                ${build.webapp.resources}
              </resourcesAsCSV>
            </baseResource>
          </webApp>
          <systemProperties>
            <systemProperty>
              <name>org.eclipse.jetty.server.Request.maxFormContentSize</name>
              <value>5000000</value>
            </systemProperty>
          </systemProperties>
          <httpConnector>
            <port>${jetty.port}</port>
          </httpConnector>
          <stopKey>JETTY_TOP</stopKey>
          <stopPort>${jetty.stop.port}</stopPort>
        </configuration>
      </plugin>

Proposal Type:

• Type: Build system
• Module: schema plugins, metadata101, webapp, root

Voting History

Vote Proposed: 14 July 2020

Project Steering Committee (voting):

• Emanuele Tajariol
• Florent Gravin
• Francois Prunayre
• Jeroen Ticheler
• Jesse Eichar
• Jose Garcia
• Paul van Genuchten
• Simon Pigot
• Jo Cook

Community support (non-voting):

• @jodygarentt (initial proposal)

Participants

• @jodygarentt