Merge pull request #1117 from oracle/owls-70496

Start WLS with -Dweblogic.SituationalConfig.failBootOnError=true

Author: rjeberhard

docs-source/content/userguide/managing-domains/configoverrides/_index.md

@@ -336,7 +336,12 @@ spec:
 ---
 ### Debugging
 
-Incorrectly formatted override files may be accepted without warnings or errors and will not prevent WebLogic pods from booting. So, it is important to make sure that the template files are correct in a QA environment, otherwise your WebLogic Servers may start even though critically required overrides are failing to take effect.
+Incorrectly formatted override files may be accepted without warnings or errors and may not prevent WebLogic pods from booting. So, it is important to make sure that the template files are correct in a QA environment, otherwise your WebLogic Servers may start even though critically required overrides are failing to take effect.
+
+On WebLogic Servers that support the `weblogic.SituationalConfig.failBootOnError` system property ( Note: It is not supported in WebLogic Server 12.2.1.3.0 ),
+by default the WebLogic server will fail to boot if any situational configuration files are invalid,
+or if it encounters an error while loading situational configuration files.
+By setting the `FAIL_BOOT_ON_SITUATIONAL_CONFIG_ERROR` environment variable in the Kubernetes containers for the WebLogic Servers to `false`, you can start up the WebLogic Servers even with incorrectly formatted override files.
 
 * Make sure you've followed each step in the [Step-by-step guide](#step-by-step-guide).
 
@@ -363,9 +368,17 @@ Incorrectly formatted override files may be accepted without warnings or errors
            ...
          ```
   * Look in your `DOMAIN_HOME/optconfig` directory.
-      * This directory, or a subdirectory within this directory, should contain each of your custom situational configuration files.
-      * If it doesn't, then this likely indicates your domain resource `configOverrides` was not set to match your custom override configuration map name, or that your custom override configuration map does not contain your override files.
-
+    * This directory, or a subdirectory within this directory, should contain each of your custom situational configuration files.
+    * If it doesn't, then this likely indicates your domain resource `configOverrides` was not set to match your custom override configuration map name, or that your custom override configuration map does not contain your override files.
+
+* If the Administration Server pod does start but fails to reach ready state or tries to restart:
+  * Check for this message ` WebLogic server failed to start due to missing or invalid situational configuration files` in the Administration Server pod's `kubectl log`
+    * This suggests that the Administration Server failure to start may have been caused by errors found in a configuration override file.
+      * Lines containing the String `situational` may be found in the Administration Server pod log to provide more hints. 
+      * For example:
+        * `<Jun 20, 2019 3:48:45 AM GMT> <Warning> <Management> <BEA-141323> <The situational config file has an invalid format, it is being ignored: XMLSituationalConfigFile[/shared/domains/domain1/optconfig/jdbc/testDS-0527-jdbc-situational-config.xml] because org.xml.sax.SAXParseException; lineNumber: 8; columnNumber: 3; The element type "jdbc:jdbc-driver-params" must be terminated by the matching end-tag "</jdbc:jdbc-driver-params>".`
+      * The warning message suggests a syntax error is found in the provided configuration override file for the testDS JDBC datasource.
+  
 * If you'd like to verify that the situational configuration is taking effect in the WebLogic MBean tree, then one way to do this is to compare the `server config` and `domain config` MBean tree values.
   * The `domain config` value should reflect the original value in your domain home configuration.
   * The `server config` value should reflect the overridden value.

operator/src/main/resources/scripts/monitorLog.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# Copyright 2019, Oracle Corporation and/or its affiliates.  All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at
+# http://oss.oracle.com/licenses/upl.
+
+#
+# This script is used to monitor the server log until the server is
+# started successfully.
+# It is separate from startServer.sh so that it is easier to quickly 
+# kill the process running this script.
+#
+
+echo $$ > /tmp/monitorLog-pid
+
+SCRIPTPATH="$( cd "$(dirname "$0")" > /dev/null 2>&1 ; pwd -P )"
+source ${SCRIPTPATH}/traceUtils.sh
+
+trace "Monitoring server log file $1 every $2 seconds for selected known log messages."
+
+while true; do
+  if grep -q "BEA-141335" $1 ; then
+    msg=("WebLogic server failed to start due to missing or invalid"
+         "situational configuration files, which may be due to invalid"
+         "configOverride templates (these are specified via the"
+         "configOverride attribute in the Domain custom resource)."
+         "For details, please search your pod log or"
+         "${SERVER_OUT_FILE} for the keyword 'situational'."
+        )
+    trace "${msg[*]}"
+    exit 0
+  fi
+  if grep -q "BEA-000360" $1 ; then
+    trace "WebLogic Server started successfully."
+    exit 0
+  fi
+  sleep $2
+done
+

operator/src/main/resources/scripts/startNodeManager.sh

@@ -27,6 +27,9 @@
 #                       Default:
 #                          Use LOG_HOME.  If LOG_HOME not set, use NODEMGR_HOME.
 #   ADMIN_PORT_SECURE = "true" if the admin protocol is secure. Default is false
+#   FAIL_BOOT_ON_SITUATIONAL_CONFIG_ERROR = "true" if WebLogic server should fail to 
+#                       boot if situational configuration related errors are 
+#                       found. Default to "true" if unspecified.
 #
 # If SERVER_NAME is set, then this NM is for a WL Server and these must also be set:
 # 
@@ -113,12 +116,14 @@ export NODEMGR_HOME=${NODEMGR_HOME}/${DOMAIN_UID}/${SERVER_NAME}
 createFolder ${NODEMGR_HOME} 
 
 NODEMGR_LOG_HOME=${NODEMGR_LOG_HOME:-${LOG_HOME:-${NODEMGR_HOME}/${DOMAIN_UID}}}
+FAIL_BOOT_ON_SITUATIONAL_CONFIG_ERROR=${FAIL_BOOT_ON_SITUATIONAL_CONFIG_ERROR:-true}
 
 trace "Info: NODEMGR_HOME='${NODEMGR_HOME}'"
 trace "Info: LOG_HOME='${LOG_HOME}'"
 trace "Info: SERVER_NAME='${SERVER_NAME}'"
 trace "Info: DOMAIN_UID='${DOMAIN_UID}'"
 trace "Info: NODEMGR_LOG_HOME='${NODEMGR_LOG_HOME}'"
+trace "Info: FAIL_BOOT_ON_SITUATIONAL_CONFIG_ERROR='${FAIL_BOOT_ON_SITUATIONAL_CONFIG_ERROR}'"
 
 createFolder ${NODEMGR_LOG_HOME}
 
@@ -239,7 +244,7 @@ RestartInterval=3600
 NumberOfFilesLimited=true
 FileTimeSpan=24
 NMHostName=${SERVICE_NAME}
-Arguments=${USER_MEM_ARGS} -XX\\:+UnlockExperimentalVMOptions -XX\\:+UseCGroupMemoryLimitForHeap ${serverOutOption} ${JAVA_OPTIONS}
+Arguments=${USER_MEM_ARGS} -XX\\:+UnlockExperimentalVMOptions -XX\\:+UseCGroupMemoryLimitForHeap -Dweblogic.SituationalConfig.failBootOnError=${FAIL_BOOT_ON_SITUATIONAL_CONFIG_ERROR} ${serverOutOption} ${JAVA_OPTIONS}
 
 EOF
 

operator/src/main/resources/scripts/startServer.sh

@@ -102,6 +102,11 @@ function waitUntilShutdown() {
     trace "Showing the server out file from ${SERVER_OUT_FILE}"
     ${SCRIPTPATH}/tailLog.sh ${SERVER_OUT_FILE} &
   fi
+  FAIL_BOOT_ON_SITUATIONAL_CONFIG_ERROR=${FAIL_BOOT_ON_SITUATIONAL_CONFIG_ERROR:-true} 
+  SERVER_OUT_MONITOR_INTERVAL=${SERVER_OUT_MONITOR_INTERVAL:-3}
+  if [ ${FAIL_BOOT_ON_SITUATIONAL_CONFIG_ERROR} == 'true' ] ; then
+    ${SCRIPTPATH}/monitorLog.sh ${SERVER_OUT_FILE} ${SERVER_OUT_MONITOR_INTERVAL} &
+  fi
   waitForShutdownMarker
 }
 
@@ -137,6 +142,8 @@ function copySitCfg() {
   if [ $? = 0 ]; then
     for local_fname in ${src_dir}/${fil_prefix}*.xml ; do
       copyIfChanged $local_fname $tgt_dir/`basename ${local_fname/${fil_prefix}//}`
+      trace "Printing contents of situational configuration file $local_fname:"
+      cat $local_fname
     done
   fi
 

operator/src/main/resources/scripts/tailLog.sh

@@ -10,5 +10,5 @@
 # running this script.
 #
 
-echo $$ > /weblogic-operator/pid
-tail -F -n +0 $1
+echo $$ > /tmp/tailLog-pid
+tail -F -n +0 $1

operator/src/test/java/oracle/kubernetes/operator/helpers/ConfigMapHelperTest.java

@@ -67,7 +67,8 @@ public class ConfigMapHelperTest {
     "traceUtils.py",
     "traceUtils.sh",
     "wlst.sh",
-    "tailLog.sh"
+    "tailLog.sh",
+    "monitorLog.sh"
   };
   private static final String DOMAIN_NS = "namespace";
   private static final String OPERATOR_NS = "operator";