Skip to content

Database / Liquibase#9146

Draft
fxprunayre wants to merge 36 commits intomainfrom
feature/liquibase
Draft

Database / Liquibase#9146
fxprunayre wants to merge 36 commits intomainfrom
feature/liquibase

Conversation

@fxprunayre
Copy link
Member

@fxprunayre fxprunayre commented Jan 28, 2026

Proposal

Liquibase is used to manage database schema creation and changes on a per-feature basis.

Main goals are:

  • manage database changes on a per feature basis (instead of a per version basis) to facilitate backporting features
  • improve migration which currently does not work with all possible migration paths
  • be able to setup/migrate databases in CI pipelines
  • be able to initialized the database with custom settings or custom admin user (using -Dgeonetwork.settings.system.server.host=...)
  • be able to manage database from GeoNetwork 4 and 5 with same system

GeoNetwork 4.4 (and before) is using its own custom mechanism based on Java and SQL to migrate from version to version. This mechanism does not work well when backporting changes to previous versions and does not support all possible migration paths.

It is proposed the following strategy to move to Liquibase: Custom GeoNetwork system is used until version 4.4.x, then starting from version 4.6.x Liquibase only is used.

Proposal to be discussed and planned :

  • Create a 4.4.x branch (with database schema freeze - or not?)
  • main branch is 4.6.x (use liquibase)
  • Release a 4.4.10 and a 4.6.0 version

See https://github.com/geonetwork/core-geonetwork/tree/feature/liquibase/database for details

Actions:

  • Databases tested (test: first run works, restart works)
    • Postgres
    • H2 (needed for test in GN4)
    • Other - discuss which database do we support and test?
  • Database creation
    • Document how to create database without the app running
    • Check creation in non default schema - currentSchema=gn in JDBC URL works
    • Database settings and admin user can be initialized with variables
    • Liquibase bean configuration with changelog.xml as main changelog file
    • Schema creation / Precondition: no table metadata exists
    • Initial data / Precondition: table settings is empty
    • Initial languages / Precondition: table language does not have eng
    • Other languages
    • Remove/Add language from the admin use the initial language data
  • Liquibase connection
    • Add username/password dedicated to liquibase (may have more privileges than the user running the app) with fallback to app user
    • Add liquibase schema (in order to put liquibase table in a custom schema) with fallback to app schema
  • Testing
    • Test and integration test ok
    • For test, populate database with minimal data (ie. only one language) - see context=test
    • Some test requires db with no data eg. domain - see context=schema-only
  • Liquibase utility
    • Test connection
    • Populate database
    • Diff between 2 databases
    • Generate change logs from existing database using Liquibase command line tool
    • Generate change logs from existing database in Intellij
    • Generate a changeset file directly from a ref and a target
  • Release
    • Create changeset when making a release
  • Cleanup
    • Remove past database migrations
    • Remove old configuration initial_data.xml and database_migration.xml
    • Remove db.migration_onstartup property / Hibernate hbm2ddl property set to none
    • Remove TestDatabasePopulator
    • Remove tests related to old Java database migrations.
  • Configure log level for liquibase https://docs.liquibase.com/oss/user-guide-4-33/use-environment-variables-to-control-log-level
  • GeoNetwork 5 / Make a PR for using this module to create the database (see README and Database / Liquibase geonetwork#153)
  • Additional future tasks
    • Admin > Languages / Q: One limitation in future changeset, is how do we load additional language specific data not in initial changeset (eg. 00023-initial-data-language-tur.sql)?

Checklist

  • I have read the contribution guidelines
  • Pull request provided for main branch, backports managed with label
  • Good housekeeping of code, cleaning up comments, tests, and documentation
  • Clean commit history broken into understandable chucks, avoiding big commits with hundreds of files, cautious of reformatting and whitespace changes
  • Clean commit messages, longer verbose messages are encouraged
  • API Changes are identified in commit messages
  • Testing provided for features or enhancements using automatic tests
  • User documentation provided for new features or enhancements in manual
  • Build documentation provided for development instructions in README.md files
  • Library management using pom.xml dependency management. Update build documentation with intended library use and library tutorials or documentation

Inspired by work done by @joachimnielandt in https://metadata.vlaanderen.be/

Funded by BRGM

@fxprunayre fxprunayre marked this pull request as draft January 28, 2026 17:35
@fxprunayre fxprunayre force-pushed the feature/liquibase branch 2 times, most recently from 8fe59c0 to 8662416 Compare January 29, 2026 09:20
@sonarqubecloud
Copy link

@josegar74
Copy link
Member

@fxprunayre, I think for this pull request it should be relevant to create a 4.4.x branch and update main to 4.6.x, what do you think?

@fxprunayre
Copy link
Member Author

@fxprunayre, I think for this pull request it should be relevant to create a 4.4.x branch and update main to 4.6.x, what do you think?

Yes if it is decided to move forward on this, a dedicated new main branch would be needed. The main reason would be because of the past migrations steps that would be quite hard to migrate in liquibase changesets I think. Maybe you've other ideas about how to deal with that?

For now, this PR is more a place to investigate if this is a good option or not and what are difficulties. So when to create the new main branch is more dependent on others "major" 4.4.x PRs and confirmation for funding for this one.

@joachimnielandt
Copy link
Contributor

joachimnielandt commented Feb 9, 2026

As a comment: liquibase has been a part of our fork update strategy for a while now. It has been integral in, e.g.,

  • keeping the various environments in sync
  • making test records available in specific environments
  • regenerating configuration each time a deployment happens (asserting default values)
  • tracking core model changes and merging them with our own updates

@jahow jahow moved this to Open for Voting in GeoNetwork 4 Roadmap Feb 11, 2026
…looking for category (Category table is now empty on init).
…e is now empty on init)

To run one test, we can use
mvn integration-test -Pit -Dtest=None -DfailIfNoTests=false -DskipIntegrationTests=false -DskipTests=false -Dit.test=org.fao.geonet.kernel.datamanager.BaseMetadataCategoryTest
…ed when webapp is used. Liquibase CLI does not work with it.
…(for domain test), with minimal test data (ie. no extra languages) for core testing.
* Centralize plugin config
* Reduce log level
fxprunayre and others added 16 commits March 11, 2026 09:25
…which does not use proper type - Fixed in Page and TextFile entity by adding correct annotation.
NPE does not happen on github but this test is failing locally on mac
and linux

```
cd core
mvn integration-test -Pit -DfailIfNoTests=false -DskipIntegrationTests=false -DskipTests=false -Dtest=None -Dit.test=org.fao.geonet.kernel.XmlSerializerIntegrationTest
```

Adding a check to avoid the NPE.
mvn jetty:run -Dgeonetwork.system.site.name=CustomName
can be use to customize setting when populating the database.
To use a custom schema, use env variable GEONETWORK_DB_CONNECTION_PROPERTIES. For example, for Postgres: GEONETWORK_DB_CONNECTION_PROPERTIES=currentSchema=test
…tead of removed SQL files).

A question is later, how do we load additional language specific data not in initial changeset?
@sonarqubecloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)
B Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
// Search for file ending with -initial-data-language-{langCode}.sql eg. 00012-initial-data-language-geo.sql
// TODO: Q: How to deal with future language data?
org.springframework.core.io.Resource[] resources = resolver.getResources("classpath*:db/changesets/*-initial-data-language-" + langCode + ".sql");

Check notice

Code scanning / SonarCloud

Logging should not be vulnerable to injection attacks Low

Change this code to not log user-controlled data. See more on SonarQube Cloud
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Open for Voting

Development

Successfully merging this pull request may close these issues.

4 participants