refactor(backend): System Shutdown Cleanup (dotCMS#32477) (dotCMS#32493)

### Proposed Changes
* Creates a shutdown coordinator to properly shutdown and cleanup
application resources when a signal to shutdown has been sent.
* Makes sure the main application is shutdown before tomcat shuts down
itself including the Log4J logging so the status of shutdown can be
properly logged.
* Cleans up the shutdown process and order of some of our common
services like the ReindexThread and JobQueue to ensure they cannot block
shutting down.

Shutdown process can be checked by stopping the main container, it
should cleanly shutdown with logs finishing with "Exiting JVM to
complete container shutdown" Docker has a default 10s shutdown anything
longer and a kill is required.

This should also resolve "just dev-stop" from not shutting down the main
dotcms container properly due to it requiring a kill to shut down fully.



### Checklist
- [ ] Tests
- [ ] Translations
- [ ] Security Implications Contemplated (add notes if applicable)

### Additional Info
This PR resolves dotCMS#32477 (System Shutdown Cleanup).


### Screenshots
Original             |  Updated
:-------------------------:|:-------------------------:
** original screenshot **  |  ** updated screenshot **

Author: spbolton

.cursor/rules/dotcms-configuration-conventions.mdc

@@ -0,0 +1,159 @@
+---
+description: 
+globs: 
+alwaysApply: false
+---
+# dotCMS Configuration Conventions
+
+## Property File Hierarchy
+
+### Main Configuration File
+- **Primary file**: [dotmarketing-config.properties](mdc:dotCMS/src/main/resources/dotmarketing-config.properties)
+- **Use for**: General application configuration, new features, most properties
+
+### Specialized Configuration Files  
+For properties related to specific domains that already have dedicated files:
+
+- **Cluster config**: [dotcms-config-cluster.properties](mdc:dotCMS/src/main/resources/dotcms-config-cluster.properties)
+  - Use for: Clustering, ES endpoints, reindexing, heartbeat settings
+  - Example properties: `ES_ENDPOINTS`, `HEARTBEAT_TIMEOUT`, `REINDEX_THREAD_*`
+
+- **SAML config**: [dotcms-saml-default.properties](mdc:dotCMS/src/main/resources/dotcms-saml-default.properties)  
+  - Use for: SAML authentication configuration
+
+- **Database config**: [db.properties](mdc:dotCMS/src/main/resources/db.properties)
+  - Use for: Database connection settings
+
+- **Portal config**: [portal.properties](mdc:dotCMS/src/main/resources/portal.properties)
+  - Use for: Portal/legacy configuration
+
+### Decision Rules
+1. **Check existing files first**: If your property relates to functionality already configured in a specialized file, add it there
+2. **Use main config for new domains**: For new features or general configuration, use [dotmarketing-config.properties](mdc:dotCMS/src/main/resources/dotmarketing-config.properties)
+3. **Don't create new specialized files**: Avoid creating new `.properties` files unless absolutely necessary
+
+## Property Naming Convention
+
+### In Properties File
+Use **lowercase dot-separated** format:
+```properties
+# Correct format
+shutdown.timeout.seconds=30
+shutdown.component.timeout.seconds=10
+experiments.enabled=true
+health.detailed.authentication.required=true
+```
+
+### In Java Code
+Use the **same lowercase dot-separated** format with `Config` class:
+```java
+// Correct usage
+Config.getIntProperty("shutdown.timeout.seconds", 30)
+Config.getBooleanProperty("experiments.enabled", false)
+Config.getStringProperty("health.detailed.authentication.required", "true")
+```
+
+### Environment Variables (Automatic)
+The `Config` class automatically transforms properties:
+- `shutdown.timeout.seconds` → `DOT_SHUTDOWN_TIMEOUT_SECONDS`
+- `experiments.enabled` → `DOT_EXPERIMENTS_ENABLED`
+- `health.detailed.authentication.required` → `DOT_HEALTH_DETAILED_AUTHENTICATION_REQUIRED`
+
+## Config Class Resolution Order
+1. **Environment variables** (with `DOT_` prefix)
+2. **System table** (database)
+3. **Properties files** (all files are loaded)
+
+## Configuration Documentation Pattern
+When adding new properties, include:
+```properties
+# Purpose description (default: value)
+# Environment variable: DOT_PROPERTY_NAME
+property.name=default_value
+```
+
+## Examples from Codebase
+
+### Health Check Configuration (Main Config)
+```properties
+# In dotmarketing-config.properties
+health.detailed.authentication.required=true
+
+# In Java code
+Config.getBooleanProperty("health.detailed.authentication.required", true)
+
+# Environment variable override
+DOT_HEALTH_DETAILED_AUTHENTICATION_REQUIRED=false
+```
+
+### Cluster Configuration (Specialized File)
+```properties
+# In dotcms-config-cluster.properties
+ES_ENDPOINTS=http://localhost:9200
+HEARTBEAT_TIMEOUT=600
+
+# In Java code
+Config.getStringProperty("ES_ENDPOINTS", "http://localhost:9200")
+Config.getIntProperty("HEARTBEAT_TIMEOUT", 600)
+```
+
+### Shutdown Configuration (Main Config)
+```properties
+# In dotmarketing-config.properties  
+shutdown.timeout.seconds=30
+shutdown.graceful.logging=true
+
+# In Java code - ShutdownCoordinator
+Config.getIntProperty("shutdown.timeout.seconds", 30)
+Config.getBooleanProperty("shutdown.graceful.logging", true)
+
+# Environment variable override
+DOT_SHUTDOWN_TIMEOUT_SECONDS=45
+DOT_SHUTDOWN_GRACEFUL_LOGGING=false
+```
+
+## Anti-Patterns to Avoid
+
+### ❌ Wrong Property Naming
+```properties
+# Don't use uppercase
+SHUTDOWN_TIMEOUT_SECONDS=30
+
+# Don't use dotcms prefix in properties file
+dotcms.shutdown.timeout=30
+```
+
+### ❌ Wrong Config Usage
+```java
+// Don't use uppercase in code
+Config.getIntProperty("SHUTDOWN_TIMEOUT_SECONDS", 30)
+
+// Don't use System.getProperty directly
+System.getProperty("shutdown.timeout.seconds")
+```
+
+### ❌ Wrong File Usage
+```properties
+# Don't put cluster config in main file if it belongs in cluster file
+# In dotmarketing-config.properties (WRONG)
+es.endpoints=http://localhost:9200
+
+# Should be in dotcms-config-cluster.properties (CORRECT)
+ES_ENDPOINTS=http://localhost:9200
+```
+
+### ❌ Creating Unnecessary Files
+```
+# Don't create new specialized files unnecessarily
+dotCMS/src/main/resources/shutdown.properties  # Bad
+dotCMS/src/main/resources/my-feature.properties  # Bad
+```
+
+## Key Points
+- Check specialized files first for domain-specific properties
+- Use `Config.getProperty()` for all configuration access
+- Properties use lowercase dot-separated naming in code
+- Environment variables automatically get `DOT_` prefix and uppercase conversion
+- Add general properties to [dotmarketing-config.properties](mdc:dotCMS/src/main/resources/dotmarketing-config.properties)
+- Add domain-specific properties to existing specialized files
+- Document environment variable names in comments

dotCMS/pom.xml

@@ -2195,6 +2195,39 @@
                 </plugins>
             </build>
         </profile>
+
+        <profile>
+            <id>docker-stop-no-remove</id>
+            <activation>
+                <activeByDefault>false</activeByDefault>
+            </activation>
+            <build>
+                <defaultGoal>validate</defaultGoal>
+                <plugins>
+                    <plugin>
+                        <groupId>io.fabric8</groupId>
+                        <artifactId>docker-maven-plugin</artifactId>
+                        <configuration>
+                            <skip>false</skip>
+                            <removeVolumes>false</removeVolumes>
+                        </configuration>
+                        <executions>
+                            <execution>
+                                <id>docker-stop-no-remove</id>
+                                <goals>
+                                    <goal>stop</goal>
+                                </goals>
+                                <phase>validate</phase>
+                                <configuration>
+                                    <keepContainer>true</keepContainer>
+                                    <removeVolumes>false</removeVolumes>
+                                </configuration>
+                            </execution>
+                        </executions>
+                    </plugin>
+                </plugins>
+            </build>
+        </profile>
     </profiles>
     <reporting>
         <plugins>

dotCMS/src/enterprise/java/com/dotcms/enterprise/license/LicenseCleanupShutdownTask.java

@@ -0,0 +1,27 @@
+package com.dotcms.enterprise.license;
+
+import com.dotcms.enterprise.LicenseUtil;
+import com.dotcms.shutdown.ShutdownOrder;
+import com.dotcms.shutdown.ShutdownTask;
+import com.dotmarketing.util.Logger;
+
+import javax.enterprise.context.ApplicationScoped;
+
+@ApplicationScoped
+@ShutdownOrder(10)
+public class LicenseCleanupShutdownTask implements ShutdownTask {
+
+    @Override
+    public String getName() {
+        return "License cleanup";
+    }
+
+    @Override
+    public void run() {
+        try {
+            LicenseUtil.freeLicenseOnRepo();
+        } catch (Exception e) {
+            Logger.warn(this, "License cleanup failed: " + e.getMessage());
+        }
+    }
+}

dotCMS/src/main/java/com/dotcms/business/CacheableEagerFactory.java

@@ -96,7 +96,21 @@ default Optional<V> find(final K key) throws DotDataException {
 
             value = this.getCache().get(key);
             if (Objects.isNull(value) && Objects.isNull((getCache().get(this.getLoadedKey())))) {
-
+                
+                // Check if shutdown is in progress - avoid database operations during shutdown
+                boolean shutdownInProgress = false;
+                try {
+                    shutdownInProgress = com.dotcms.shutdown.ShutdownCoordinator.isShutdownStarted();
+                } catch (Exception e) {
+                    // If we can't check shutdown status, assume not shutting down
+                }
+                
+                if (shutdownInProgress) {
+                    // During shutdown, don't attempt database operations that might fail
+                    // Return empty result to avoid infinite recursion
+                    return Optional.empty();
+                }
+                
                 try {
                     this.findAll();
                     // Only retry if findAll() succeeded - check if cache is now loaded
@@ -122,6 +136,21 @@ default Optional<V> find(final K key) throws DotDataException {
      */
     default Map<K, V> findAll() throws DotDataException {
 
+        // Check if shutdown is in progress - avoid database operations during shutdown
+        boolean shutdownInProgress = false;
+        try {
+            shutdownInProgress = com.dotcms.shutdown.ShutdownCoordinator.isShutdownStarted();
+        } catch (Exception e) {
+            // If we can't check shutdown status, assume not shutting down
+        }
+        
+        if (shutdownInProgress) {
+            // During shutdown, don't attempt database operations that might fail
+            // Return empty map to avoid database connectivity issues
+            com.dotmarketing.util.Logger.debug(this, "Skipping database operations during shutdown - returning empty result");
+            return new HashMap<>();
+        }
+
         final Map<K, V> records = new HashMap<>();
 
         final List<Map<String, Object>> result = new DotConnect()
@@ -152,6 +181,20 @@ default Map<K, V> findAll() throws DotDataException {
     default void saveOrUpdate(final K key, final V value) throws DotDataException {
 
         if (Objects.nonNull(key) && Objects.nonNull(value)) {
+            
+            // Check if shutdown is in progress - avoid database operations during shutdown
+            boolean shutdownInProgress = false;
+            try {
+                shutdownInProgress = com.dotcms.shutdown.ShutdownCoordinator.isShutdownStarted();
+            } catch (Exception e) {
+                // If we can't check shutdown status, assume not shutting down
+            }
+            
+            if (shutdownInProgress) {
+                // During shutdown, don't attempt database operations that might fail
+                com.dotmarketing.util.Logger.debug(this, "Skipping database save/update operation during shutdown");
+                throw new DotDataException("Database operations not available during shutdown");
+            }
 
             saveOrUpdateInternal (key, value);
 
@@ -170,6 +213,20 @@ default void saveOrUpdate(final K key, final V value) throws DotDataException {
     default void delete(final K key) throws DotDataException {
 
         if (Objects.nonNull(key)) {
+            
+            // Check if shutdown is in progress - avoid database operations during shutdown
+            boolean shutdownInProgress = false;
+            try {
+                shutdownInProgress = com.dotcms.shutdown.ShutdownCoordinator.isShutdownStarted();
+            } catch (Exception e) {
+                // If we can't check shutdown status, assume not shutting down
+            }
+            
+            if (shutdownInProgress) {
+                // During shutdown, don't attempt database operations that might fail
+                com.dotmarketing.util.Logger.debug(this, "Skipping database delete operation during shutdown");
+                throw new DotDataException("Database operations not available during shutdown");
+            }
 
             deleteInternal(key);
             this.clearCache();

dotCMS/src/main/java/com/dotcms/cdi/CDIUtils.java

@@ -1,7 +1,9 @@
 package com.dotcms.cdi;
 
 import com.dotmarketing.util.Logger;
+import java.util.List;
 import java.util.Optional;
+import java.util.stream.Collectors;
 import javax.enterprise.inject.spi.CDI;
 
 /**
@@ -46,4 +48,21 @@ public static <T> T getBeanThrows(Class<T> clazz) {
         }
     }
 
+    /**
+     * Get all beans of a given type from CDI container
+     * @param clazz the class of the beans
+     * @return a List containing all beans of the given type
+     * @param <T> the type of the beans
+     */
+    public static <T> List<T> getBeans(Class<T> clazz) {
+        try {
+            return CDI.current().select(clazz).stream()
+                    .collect(Collectors.toList());
+        } catch (Exception e) {
+            String errorMessage = String.format("Unable to find beans of class [%s]: %s", clazz, e.getMessage());
+            Logger.error(CDIUtils.class, errorMessage);
+            throw new IllegalStateException(errorMessage, e);
+        }
+    }
+
 }

dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ESIndexAPI.java

@@ -1337,7 +1337,7 @@ public boolean waitUtilIndexReady() {
         }
         if (stats == null) {
             Logger.fatal(this.getClass(), "No Elasticsearch, dying an ugly death");
-            System.exit(1);
+            com.dotcms.shutdown.SystemExitManager.immediateExit(1, "Elasticsearch connection failed after maximum attempts");
         }
         return true;