STAND-127: Optimising OpenMRS Standalone 3.x Initialization by Using Pre-Initialized Demo and Empty Database and Auto-Refreshing Search Index

[Muta-Jonathan]

See https://openmrs.atlassian.net/browse/STAND-127

Description

Currently, the standalone rebuilds the full OpenMRS schema and demo content from XML on first run, which is slow and doesn't include an active Lucene search index.

This ticket proposes:

• Supporting a .sql dump from a pre-initialized SDK 3.x instance to serve as the source DB (demo/empty).

• Detecting the appropriate dump (demo-db-{refapp.version}.sql or empty-db-{refapp.version}.sql) from src/main/db/.

• Rebuilding the Lucene search index automatically at startup after importing the dump.

• Updating README to guide developers on how to generate dumps using mysqldump from their SDK environments.

• Uses docker compose to run the 3.x instance at build time and copy the checksums using

   docker cp "$CONTAINER_ID":/openmrs/data/configuration_checksums "$DISTRO_DIR/web/openmrs_config_checksums"; 
  

and then zip it along with the standalone zip to quicken the startup process

This speeds up load time significantly and ensures a smoother UX with patients immediately searchable after first launch.

ScreenRecord for demo db

Screencast.from.07-08-25.14.38.40.webm

ScreenRecord for Empty db

Screencast.from.07-08-25.14.42.20.webm

[Muta-Jonathan]

cc @dkayiwa @wikumChamith

[dkayiwa]

Is this also for the starter implementation without demo data?

[Muta-Jonathan]

Is this also for the starter implementation without demo data?

Yes using empty-db-*.sql

[dkayiwa]

Do you know why i get this for the O3 standalone and yet i never got it for the platform standalone? zsh: permission denied: ./run-on-unix.sh

[Muta-Jonathan]

Do you know why i get this for the O3 standalone and yet i never got it for the platform standalone? zsh: permission denied: ./run-on-unix.sh

Not really seems to me as a file permission failure

[dkayiwa]

Why on the O3 standalone but not for the platform one?

Have you also noticed that the ID Generation module fails to start with this error?

Error while trying to start module Unable to update data model using liquibase.xml. Module: ID Generation

[Muta-Jonathan]

Why on the O3 standalone but not for the platform one?

Have you also noticed that the ID Generation module fails to start with this error?

Error while trying to start module Unable to update data model using liquibase.xml. Module: ID Generation

Let me take a deep dive however let me push this to fix the file permission first ...

[Muta-Jonathan]

Error while trying to start module Unable to update data model using liquibase.xml. Module: ID Generation

oh i see according to this

org.openmrs.module.ModuleException: Unable to update data model using liquibase.xml. Module: ID Generation
	at org.openmrs.module.ModuleFactory.runLiquibase(ModuleFactory.java:992) ~[openmrs-api-2.7.4.jar:?]
	at org.openmrs.module.ModuleFactory.startModuleInternal(ModuleFactory.java:715) ~[openmrs-api-2.7.4.jar:?]
	at org.openmrs.api.context.Daemon.lambda$startModule$0(Daemon.java:84) ~[openmrs-api-2.7.4.jar:?]
	at org.openmrs.api.context.Daemon.lambda$runInDaemonThreadInternal$4(Daemon.java:428) ~[openmrs-api-2.7.4.jar:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: liquibase.exception.ValidationFailedException: Validation Failed:
     2 changesets check sum
          liquibase.xml::daa579e7-b8de-4858-bfe5-c9ef2606db5e::Samuel Male was: 9:3d316de8f6d09b6c8e35ac1e324c91cc but is now: 9:a1434668047ebfb0bc35377e338b72b9
          liquibase.xml::9b5c919c-2223-41d2-824a-b65869666028::Ian Bacher was: 9:a1434668047ebfb0bc35377e338b72b9 but is now: 9:3d316de8f6d09b6c8e35ac1e324c91cc

	at liquibase.changelog.DatabaseChangeLog.validate(DatabaseChangeLog.java:384) ~[liquibase-core-4.31.1.jar:?]
	at liquibase.changelog.DatabaseChangeLog.validate(DatabaseChangeLog.java:353) ~[liquibase-core-4.31.1.jar:?]
	at org.openmrs.util.DatabaseUpdater.executeChangelog(DatabaseUpdater.java:237) ~[openmrs-api-2.7.4.jar:?]
	at org.openmrs.module.ModuleFactory.runLiquibase(ModuleFactory.java:984) ~[openmrs-api-2.7.4.jar:?]
	... 7 more

It seems to be an issue Caused by: liquibase.exception.ValidationFailedException in openmrs- core -2.7.4 am using due to these two changesets conflicting

liquibase.xml::daa579e7-b8de-4858-bfe5-c9ef2606db5e::Samuel Male was: 9:3d316de8f6d09b6c8e35ac1e324c91cc but is now: 9:a1434668047ebfb0bc35377e338b72b9
          liquibase.xml::9b5c919c-2223-41d2-824a-b65869666028::Ian Bacher was: 9:a1434668047ebfb0bc35377e338b72b9 but is now: 9:3d316de8f6d09b6c8e35ac1e324c91cc

[Muta-Jonathan]

Error while trying to start module Unable to update data model using liquibase.xml. Module: ID Generation

let me try to solve it manually by adjusting them in sql directly

[Muta-Jonathan]

cc @dkayiwa fixed the module fails however now noticed that the module also when started adds some more load time

[dkayiwa]

It is taking long because it is initialising the database on the first run, just like it was before.

[Muta-Jonathan]

It is taking long because it is initialising the database on the first run, just like it was before.

But this is done on first run and takes less on second or other time
Isn't this the expected approach as 3.x loads?

[dkayiwa]

When i looked at the logs, it had those things that usually happen when intializing.

[Muta-Jonathan]

When i looked at the logs, it had those things that usually happen when intializing.

seems like the ID Generation module when solved to load increased the load time and makes the sql to initialise

[dkayiwa]

@Muta-Jonathan can we take advantage of this to fix the famous database error in where clause? openmrs/openmrs-core@474ce37

[Muta-Jonathan]

Oh yes let me look into it

[Muta-Jonathan]

@Muta-Jonathan can we take advantage of this to fix the famous database error in where clause? openmrs/openmrs-core@474ce37

@dkayiwa have fixed this however should it be included in this PR for 3.x? Is it okay?

[dkayiwa]

should it be included in this PR for 3.x? Is it okay?

You can create a separate pull request?

[Muta-Jonathan]

Great ... What is really missing for this so i can look into it so it goes in?

[dkayiwa]

When i tested it, it too long because it was running those database initialisation steps which we wanted to avoid by simply dumping an already initialised database.

[Muta-Jonathan]

oh let me run a new instance of sdk and deal with a newer dump and see how it behaves

[dkayiwa]

I did not expect to see logs like WARN - DefaultTCAppointmentPatientEmailNotifier.isApplicable(56) |2025-08-04T15:47:07,454| Email notification for tele-consultation not configured to be sent to patient.

[Muta-Jonathan]

oh yes Am now trying the new sql however seems introduces a new error but am on it to roll it out

Basically ScriptRunner the plugin we use to import the dump doesnot interprete the following sql lines basically DELIMITER ;; and /*!50003 but hopefully am finding a work around on it without custom changes in sql

/*!40000 ALTER TABLE `location` ENABLE KEYS */;
UNLOCK TABLES;
/*!50003 SET @saved_cs_client      = @@character_set_client */ ;
/*!50003 SET @saved_cs_results     = @@character_set_results */ ;
/*!50003 SET @saved_col_connection = @@collation_connection */ ;
/*!50003 SET character_set_client  = utf8mb4 */ ;
/*!50003 SET character_set_results = utf8mb4 */ ;
/*!50003 SET collation_connection  = utf8mb4_0900_ai_ci */ ;
/*!50003 SET @saved_sql_mode       = @@sql_mode */ ;
/*!50003 SET sql_mode              = 'ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION' */ ;
DELIMITER ;;
/*!50003 CREATE*/ /*!50017 DEFINER=`root`@`localhost`*/ /*!50003 TRIGGER `stockmgmt_location_after_insert` AFTER INSERT ON `location` FOR EACH ROW BEGIN
	INSERT INTO stockmgmt_location_tree (parent_location_id, child_location_id, depth, uuid)
    VALUES (NEW.location_id, NEW.location_id, 0, uuid());

	INSERT INTO stockmgmt_location_tree (parent_location_id, child_location_id, depth, uuid)
    SELECT x.parent_location_id, NEW.location_id, x.depth + 1, uuid()
    FROM stockmgmt_location_tree x
    WHERE x.child_location_id = NEW.parent_location;

    INSERT INTO stockmgmt_party(location_id, creator, date_created, uuid)
    VALUES(NEW.location_id,NEW.creator, NEW.date_created, uuid());
END */;;
DELIMITER ;

this happens during stockmgmt_location_after_insert

[Muta-Jonathan]

cc @dkayiwa

[Muta-Jonathan]

Also have updated the videos on PR

[dkayiwa]

Are you able to successfully run mvn package?

[Muta-Jonathan]

Are you able to successfully run mvn package?

Oh yes it is successful

[dkayiwa]

This is what i get: https://pastebin.com/45Q237W7

[Muta-Jonathan]

This is what i get: https://pastebin.com/45Q237W7

Is this happening on mac?

[Muta-Jonathan]

Oh seen the issue timeout is not recognised on mac let me fix it

[Muta-Jonathan]

cc @dkayiwa this should work now

[dkayiwa]

Does this work for you? mvn package -Drefapp.version=3.5.0-SNAPSHOT

[Muta-Jonathan]

Seems the 3.5.0-SNAPSHOT is not available in the referenceapplication-distro

[ERROR] Failed to execute goal org.openmrs.maven.plugins:openmrs-sdk-maven-plugin:6.4.0:build-distro (build-distro) on project referenceapplication-standalone-zip-stage: Unable to find/resolve artifact.: Failed to read artifact descriptor for org.openmrs.distro:referenceapplication-distro:zip:3.5.0-SNAPSHOT: Could not transfer artifact org.openmrs.distro:referenceapplication-distro:pom:3.5.0-SNAPSHOT from/to openmrs-repo-thirdparty (https://mavenrepo.openmrs.org/thirdparty): Transfer failed for https://mavenrepo.openmrs.org/thirdparty/org/openmrs/distro/referenceapplication-distro/3.5.0-SNAPSHOT/referenceapplication-distro-3.5.0-SNAPSHOT.pom 409 -> [Help 1]
[ERROR] 

[dkayiwa]

How about this? mvn package -Drefapp.version=3.6.0-SNAPSHOT?

[Muta-Jonathan]

same issue. could it be that the 3.x snapshots are not available/ published in the location org.openmrs.distro:referenceapplication-distro:pom?

[dkayiwa]

How does the sdk get them?

[Muta-Jonathan]

Am also wondering because am using the same version .. let me take a search in the sdk code

[Muta-Jonathan]

So basically when we use -Drefapp.version=3.4.0 it locates distro-emr-configuration-3.4.0.zip Downloading from openmrs-repo: https://mavenrepo.openmrs.org/public/org/openmrs/distro-emr-configuration/3.4.0/
4:28

however when we specify -Drefapp.version=3.6.0-SNAPSHOT its unable to locate distro-emr-configuration-3.6.0-SNAPSHOT.zip because in the location https://openmrs.jfrog.io/ui/native/public/org/openmrs/distro-emr-configuration/3.6.0-SNAPSHOT/ i see we have distro-emr-configuration-3.6.0-20250807.131409-1.zip with release dates included which makes distro-emr-configuration-3.6.0-SNAPSHOT.zip unavailable

And this also happens for other versions with similar versioning in their location folders

[dkayiwa]

Merging this. Let us deal with that in a separate pull request.