Skip to content

Commit 31034df

Browse files
spboltonspbolton
authored
refactor(backend): System Shutdown Cleanup (dotCMS#32477) (dotCMS#32493)
### Proposed Changes * Creates a shutdown coordinator to properly shutdown and cleanup application resources when a signal to shutdown has been sent. * Makes sure the main application is shutdown before tomcat shuts down itself including the Log4J logging so the status of shutdown can be properly logged. * Cleans up the shutdown process and order of some of our common services like the ReindexThread and JobQueue to ensure they cannot block shutting down. Shutdown process can be checked by stopping the main container, it should cleanly shutdown with logs finishing with "Exiting JVM to complete container shutdown" Docker has a default 10s shutdown anything longer and a kill is required. This should also resolve "just dev-stop" from not shutting down the main dotcms container properly due to it requiring a kill to shut down fully. ### Checklist - [ ] Tests - [ ] Translations - [ ] Security Implications Contemplated (add notes if applicable) ### Additional Info This PR resolves dotCMS#32477 (System Shutdown Cleanup). ### Screenshots Original | Updated :-------------------------:|:-------------------------: ** original screenshot ** | ** updated screenshot **
1 parent 3aace99 commit 31034df

File tree

56 files changed

+4162
-976
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+4162
-976
lines changed

‎.cursor/rules/dotcms-configuration-conventions.mdc‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
---
2+
description:
3+
globs:
4+
alwaysApply: false
5+
---
6+
# dotCMS Configuration Conventions
7+
8+
## Property File Hierarchy
9+
10+
### Main Configuration File
11+
- **Primary file**: [dotmarketing-config.properties](mdc:dotCMS/src/main/resources/dotmarketing-config.properties)
12+
- **Use for**: General application configuration, new features, most properties
13+
14+
### Specialized Configuration Files
15+
For properties related to specific domains that already have dedicated files:
16+
17+
- **Cluster config**: [dotcms-config-cluster.properties](mdc:dotCMS/src/main/resources/dotcms-config-cluster.properties)
18+
- Use for: Clustering, ES endpoints, reindexing, heartbeat settings
19+
- Example properties: `ES_ENDPOINTS`, `HEARTBEAT_TIMEOUT`, `REINDEX_THREAD_*`
20+
21+
- **SAML config**: [dotcms-saml-default.properties](mdc:dotCMS/src/main/resources/dotcms-saml-default.properties)
22+
- Use for: SAML authentication configuration
23+
24+
- **Database config**: [db.properties](mdc:dotCMS/src/main/resources/db.properties)
25+
- Use for: Database connection settings
26+
27+
- **Portal config**: [portal.properties](mdc:dotCMS/src/main/resources/portal.properties)
28+
- Use for: Portal/legacy configuration
29+
30+
### Decision Rules
31+
1. **Check existing files first**: If your property relates to functionality already configured in a specialized file, add it there
32+
2. **Use main config for new domains**: For new features or general configuration, use [dotmarketing-config.properties](mdc:dotCMS/src/main/resources/dotmarketing-config.properties)
33+
3. **Don't create new specialized files**: Avoid creating new `.properties` files unless absolutely necessary
34+
35+
## Property Naming Convention
36+
37+
### In Properties File
38+
Use **lowercase dot-separated** format:
39+
```properties
40+
# Correct format
41+
shutdown.timeout.seconds=30
42+
shutdown.component.timeout.seconds=10
43+
experiments.enabled=true
44+
health.detailed.authentication.required=true
45+
```
46+
47+
### In Java Code
48+
Use the **same lowercase dot-separated** format with `Config` class:
49+
```java
50+
// Correct usage
51+
Config.getIntProperty("shutdown.timeout.seconds", 30)
52+
Config.getBooleanProperty("experiments.enabled", false)
53+
Config.getStringProperty("health.detailed.authentication.required", "true")
54+
```
55+
56+
### Environment Variables (Automatic)
57+
The `Config` class automatically transforms properties:
58+
- `shutdown.timeout.seconds` → `DOT_SHUTDOWN_TIMEOUT_SECONDS`
59+
- `experiments.enabled` → `DOT_EXPERIMENTS_ENABLED`
60+
- `health.detailed.authentication.required` → `DOT_HEALTH_DETAILED_AUTHENTICATION_REQUIRED`
61+
62+
## Config Class Resolution Order
63+
1. **Environment variables** (with `DOT_` prefix)
64+
2. **System table** (database)
65+
3. **Properties files** (all files are loaded)
66+
67+
## Configuration Documentation Pattern
68+
When adding new properties, include:
69+
```properties
70+
# Purpose description (default: value)
71+
# Environment variable: DOT_PROPERTY_NAME
72+
property.name=default_value
73+
```
74+
75+
## Examples from Codebase
76+
77+
### Health Check Configuration (Main Config)
78+
```properties
79+
# In dotmarketing-config.properties
80+
health.detailed.authentication.required=true
81+
82+
# In Java code
83+
Config.getBooleanProperty("health.detailed.authentication.required", true)
84+
85+
# Environment variable override
86+
DOT_HEALTH_DETAILED_AUTHENTICATION_REQUIRED=false
87+
```
88+
89+
### Cluster Configuration (Specialized File)
90+
```properties
91+
# In dotcms-config-cluster.properties
92+
ES_ENDPOINTS=http://localhost:9200
93+
HEARTBEAT_TIMEOUT=600
94+
95+
# In Java code
96+
Config.getStringProperty("ES_ENDPOINTS", "http://localhost:9200")
97+
Config.getIntProperty("HEARTBEAT_TIMEOUT", 600)
98+
```
99+
100+
### Shutdown Configuration (Main Config)
101+
```properties
102+
# In dotmarketing-config.properties
103+
shutdown.timeout.seconds=30
104+
shutdown.graceful.logging=true
105+
106+
# In Java code - ShutdownCoordinator
107+
Config.getIntProperty("shutdown.timeout.seconds", 30)
108+
Config.getBooleanProperty("shutdown.graceful.logging", true)
109+
110+
# Environment variable override
111+
DOT_SHUTDOWN_TIMEOUT_SECONDS=45
112+
DOT_SHUTDOWN_GRACEFUL_LOGGING=false
113+
```
114+
115+
## Anti-Patterns to Avoid
116+
117+
### ❌ Wrong Property Naming
118+
```properties
119+
# Don't use uppercase
120+
SHUTDOWN_TIMEOUT_SECONDS=30
121+
122+
# Don't use dotcms prefix in properties file
123+
dotcms.shutdown.timeout=30
124+
```
125+
126+
### ❌ Wrong Config Usage
127+
```java
128+
// Don't use uppercase in code
129+
Config.getIntProperty("SHUTDOWN_TIMEOUT_SECONDS", 30)
130+
131+
// Don't use System.getProperty directly
132+
System.getProperty("shutdown.timeout.seconds")
133+
```
134+
135+
### ❌ Wrong File Usage
136+
```properties
137+
# Don't put cluster config in main file if it belongs in cluster file
138+
# In dotmarketing-config.properties (WRONG)
139+
es.endpoints=http://localhost:9200
140+
141+
# Should be in dotcms-config-cluster.properties (CORRECT)
142+
ES_ENDPOINTS=http://localhost:9200
143+
```
144+
145+
### ❌ Creating Unnecessary Files
146+
```
147+
# Don't create new specialized files unnecessarily
148+
dotCMS/src/main/resources/shutdown.properties # Bad
149+
dotCMS/src/main/resources/my-feature.properties # Bad
150+
```
151+
152+
## Key Points
153+
- Check specialized files first for domain-specific properties
154+
- Use `Config.getProperty()` for all configuration access
155+
- Properties use lowercase dot-separated naming in code
156+
- Environment variables automatically get `DOT_` prefix and uppercase conversion
157+
- Add general properties to [dotmarketing-config.properties](mdc:dotCMS/src/main/resources/dotmarketing-config.properties)
158+
- Add domain-specific properties to existing specialized files
159+
- Document environment variable names in comments

‎dotCMS/pom.xml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2195,6 +2195,39 @@
21952195
</plugins>
21962196
</build>
21972197
</profile>
2198+
2199+
<profile>
2200+
<id>docker-stop-no-remove</id>
2201+
<activation>
2202+
<activeByDefault>false</activeByDefault>
2203+
</activation>
2204+
<build>
2205+
<defaultGoal>validate</defaultGoal>
2206+
<plugins>
2207+
<plugin>
2208+
<groupId>io.fabric8</groupId>
2209+
<artifactId>docker-maven-plugin</artifactId>
2210+
<configuration>
2211+
<skip>false</skip>
2212+
<removeVolumes>false</removeVolumes>
2213+
</configuration>
2214+
<executions>
2215+
<execution>
2216+
<id>docker-stop-no-remove</id>
2217+
<goals>
2218+
<goal>stop</goal>
2219+
</goals>
2220+
<phase>validate</phase>
2221+
<configuration>
2222+
<keepContainer>true</keepContainer>
2223+
<removeVolumes>false</removeVolumes>
2224+
</configuration>
2225+
</execution>
2226+
</executions>
2227+
</plugin>
2228+
</plugins>
2229+
</build>
2230+
</profile>
21982231
</profiles>
21992232
<reporting>
22002233
<plugins>

‎dotCMS/src/enterprise/java/com/dotcms/enterprise/license/LicenseCleanupShutdownTask.java‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
package com.dotcms.enterprise.license;
2+
3+
import com.dotcms.enterprise.LicenseUtil;
4+
import com.dotcms.shutdown.ShutdownOrder;
5+
import com.dotcms.shutdown.ShutdownTask;
6+
import com.dotmarketing.util.Logger;
7+
8+
import javax.enterprise.context.ApplicationScoped;
9+
10+
@ApplicationScoped
11+
@ShutdownOrder(10)
12+
public class LicenseCleanupShutdownTask implements ShutdownTask {
13+
14+
@Override
15+
public String getName() {
16+
return "License cleanup";
17+
}
18+
19+
@Override
20+
public void run() {
21+
try {
22+
LicenseUtil.freeLicenseOnRepo();
23+
} catch (Exception e) {
24+
Logger.warn(this, "License cleanup failed: " + e.getMessage());
25+
}
26+
}
27+
}

‎dotCMS/src/main/java/com/dotcms/business/CacheableEagerFactory.java‎

Lines changed: 58 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,21 @@ default Optional<V> find(final K key) throws DotDataException {
9696

9797
value = this.getCache().get(key);
9898
if (Objects.isNull(value) && Objects.isNull((getCache().get(this.getLoadedKey())))) {
99-
99+
100+
// Check if shutdown is in progress - avoid database operations during shutdown
101+
boolean shutdownInProgress = false;
102+
try {
103+
shutdownInProgress = com.dotcms.shutdown.ShutdownCoordinator.isShutdownStarted();
104+
} catch (Exception e) {
105+
// If we can't check shutdown status, assume not shutting down
106+
}
107+
108+
if (shutdownInProgress) {
109+
// During shutdown, don't attempt database operations that might fail
110+
// Return empty result to avoid infinite recursion
111+
return Optional.empty();
112+
}
113+
100114
try {
101115
this.findAll();
102116
// Only retry if findAll() succeeded - check if cache is now loaded
@@ -122,6 +136,21 @@ default Optional<V> find(final K key) throws DotDataException {
122136
*/
123137
default Map<K, V> findAll() throws DotDataException {
124138

139+
// Check if shutdown is in progress - avoid database operations during shutdown
140+
boolean shutdownInProgress = false;
141+
try {
142+
shutdownInProgress = com.dotcms.shutdown.ShutdownCoordinator.isShutdownStarted();
143+
} catch (Exception e) {
144+
// If we can't check shutdown status, assume not shutting down
145+
}
146+
147+
if (shutdownInProgress) {
148+
// During shutdown, don't attempt database operations that might fail
149+
// Return empty map to avoid database connectivity issues
150+
com.dotmarketing.util.Logger.debug(this, "Skipping database operations during shutdown - returning empty result");
151+
return new HashMap<>();
152+
}
153+
125154
final Map<K, V> records = new HashMap<>();
126155

127156
final List<Map<String, Object>> result = new DotConnect()
@@ -152,6 +181,20 @@ default Map<K, V> findAll() throws DotDataException {
152181
default void saveOrUpdate(final K key, final V value) throws DotDataException {
153182

154183
if (Objects.nonNull(key) && Objects.nonNull(value)) {
184+
185+
// Check if shutdown is in progress - avoid database operations during shutdown
186+
boolean shutdownInProgress = false;
187+
try {
188+
shutdownInProgress = com.dotcms.shutdown.ShutdownCoordinator.isShutdownStarted();
189+
} catch (Exception e) {
190+
// If we can't check shutdown status, assume not shutting down
191+
}
192+
193+
if (shutdownInProgress) {
194+
// During shutdown, don't attempt database operations that might fail
195+
com.dotmarketing.util.Logger.debug(this, "Skipping database save/update operation during shutdown");
196+
throw new DotDataException("Database operations not available during shutdown");
197+
}
155198

156199
saveOrUpdateInternal (key, value);
157200

@@ -170,6 +213,20 @@ default void saveOrUpdate(final K key, final V value) throws DotDataException {
170213
default void delete(final K key) throws DotDataException {
171214

172215
if (Objects.nonNull(key)) {
216+
217+
// Check if shutdown is in progress - avoid database operations during shutdown
218+
boolean shutdownInProgress = false;
219+
try {
220+
shutdownInProgress = com.dotcms.shutdown.ShutdownCoordinator.isShutdownStarted();
221+
} catch (Exception e) {
222+
// If we can't check shutdown status, assume not shutting down
223+
}
224+
225+
if (shutdownInProgress) {
226+
// During shutdown, don't attempt database operations that might fail
227+
com.dotmarketing.util.Logger.debug(this, "Skipping database delete operation during shutdown");
228+
throw new DotDataException("Database operations not available during shutdown");
229+
}
173230

174231
deleteInternal(key);
175232
this.clearCache();

‎dotCMS/src/main/java/com/dotcms/cdi/CDIUtils.java‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
package com.dotcms.cdi;
22

33
import com.dotmarketing.util.Logger;
4+
import java.util.List;
45
import java.util.Optional;
6+
import java.util.stream.Collectors;
57
import javax.enterprise.inject.spi.CDI;
68

79
/**
@@ -46,4 +48,21 @@ public static <T> T getBeanThrows(Class<T> clazz) {
4648
}
4749
}
4850

51+
/**
52+
* Get all beans of a given type from CDI container
53+
* @param clazz the class of the beans
54+
* @return a List containing all beans of the given type
55+
* @param <T> the type of the beans
56+
*/
57+
public static <T> List<T> getBeans(Class<T> clazz) {
58+
try {
59+
return CDI.current().select(clazz).stream()
60+
.collect(Collectors.toList());
61+
} catch (Exception e) {
62+
String errorMessage = String.format("Unable to find beans of class [%s]: %s", clazz, e.getMessage());
63+
Logger.error(CDIUtils.class, errorMessage);
64+
throw new IllegalStateException(errorMessage, e);
65+
}
66+
}
67+
4968
}

‎dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ESIndexAPI.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1337,7 +1337,7 @@ public boolean waitUtilIndexReady() {
13371337
}
13381338
if (stats == null) {
13391339
Logger.fatal(this.getClass(), "No Elasticsearch, dying an ugly death");
1340-
System.exit(1);
1340+
com.dotcms.shutdown.SystemExitManager.immediateExit(1, "Elasticsearch connection failed after maximum attempts");
13411341
}
13421342
return true;
13431343

0 commit comments

Comments
 (0)