Merge branch 'develop' of https://github.com/oracle/weblogic-kubernetes-operator into feature/java-integration-tests

Author: vanajamukkara

README.md

@@ -20,7 +20,7 @@ In addition, we've provided an open-source Oracle WebLogic Server Kubernetes Ope
 * Scale WebLogic domains by starting and stopping Managed Servers on demand, or by integrating with a REST API to initiate scaling based on WLDF, Prometheus, Grafana, or other rules.
 * Publish operator and WebLogic Server logs into Elasticsearch and interact with them in Kibana.
 
-The fastest way to experience the operator is to follow the Quick Start Guide, or you can peruse our [documentation](site), read our blogs, try out the [samples](kubernetes/samples/README.md), or check us out on our public Slack channel.
+The fastest way to experience the operator is to follow the [Quick start guide](site/quickstart.md), or you can peruse our [documentation](site), read our blogs, or try out the [samples](kubernetes/samples/README.md).
 
 
 # Important terms
@@ -45,7 +45,7 @@ Before using the operator, you might want to read the [design philosophy](site/d
 
 ## Prerequisites
 
-*	Kubernetes 1.10+, 1.11+, and 1.12.0+  (check with `kubectl version`).
+*	Kubernetes 1.10.11+, 1.11.5+, and 1.12.3+  (check with `kubectl version`).
 *	Flannel networking v0.9.1-amd64 (check with `docker images | grep flannel`)
 *	Docker 17.03.1.ce (check with `docker version`)
 *	Oracle WebLogic Server 12.2.1.3.0
@@ -81,7 +81,7 @@ In your Kubernetes cluster you can have one or more operators that manage one or
 * Edit the operator input YAML
 * Start the operator with a Helm chart
 
-### Modifying the Operator
+### Modifying the operator
 
 (images, RBAC roles, ...)
 
@@ -91,16 +91,17 @@ In your Kubernetes cluster you can have one or more operators that manage one or
 ## Create and manage WebLogic domains
 
 In this version of the operator, a WebLogic domain can be persisted either to a persistent volume (PV) or in a Docker image.
-(We can add a paragraph that describes the pros and cons of both these approaches.)
+(Describe the pros and cons of both these approaches.)
 
-You can create the WebLogic domain inside of a Docker image or in a PV using WebLogic Scripting Tool (WLST) or WebLogic Deploy Tooling (WDT).  
-* Advantages of using WDT.  (See samples, Domain in image WDT, Domain in image WLST, Domain in PV WDT, Domain in PV WLST)
-
-(What images do we need before we get started? Operator 2.0 requires you to patch your WebLogic image 12.2.1.3 with patch #.)
 * WebLogic binary image when domain is persisted to a PV (as in Operator v1.1)
 * WebLogic domain image where the domain is persisted to a Docker image (new for Operator v2.0).  The WebLogic domain image will contain the WebLogic binaries, domain configuration, and applications.
 
-### Preparing the Kubernetes cluster to run the WebLogic domain
+You create the WebLogic domain inside of a Docker image or in a PV using WebLogic Scripting Tool (WLST) or WebLogic Deploy Tooling (WDT).  
+* (Describe the advantages of using WDT. See samples, Domain in image WDT, Domain in image WLST, Domain in PV WDT, Domain in PV WLST.)
+
+(What images do we need before we get started? Operator 2.0 requires you to patch your WebLogic image 12.2.1.3 with patch #.)
+
+### Preparing the Kubernetes cluster to run WebLogic domains
 
 Perform these steps to prepare your Kubernetes cluster to run a WebLogic domain:
 
@@ -110,9 +111,9 @@ Perform these steps to prepare your Kubernetes cluster to run a WebLogic domain:
 * Optionally, [create a PV & persistent volume claim (PVC)](kubernetes/samples/scripts/create-weblogic-domain-pv-pvc/README.md) which can hold the domain home, logs, and application binaries.
 * [Configure a load balancer](kubernetes/samples/charts/README.md) to manage the domains and ingresses.
 
-### Creating and managing the WebLogic domain
+### Creating and managing WebLogic domains
 
-To create and manage a WebLogic domain in Kubernetes we create a deployment type, the domain custom resource.   The operator always looks at the custom resource and manages the domain deployment to adjust to the definitions in the custom resource.   This custom resource can also be managed using the Kubernetes command-line interface `kubectl`.  
+To create and manage a WebLogic domain in Kubernetes we create a deployment type, the domain custom resource.   The operator introspects the custom resource and manages the domain deployment to adjust to the definitions in the custom resource.   This custom resource can also be managed using the Kubernetes command-line interface `kubectl`.  
 * (Point to documentation how to edit the domain inputs YAML and how to create the domain custom resource.)
 * Create Ingress controllers if you are using a load balancer that supports them, such as Traefik or Voyager.
 
@@ -122,16 +123,16 @@ In Operator 2.0 you can perform lifecycle operations on WebLogic servers, cluste
 * Point to the documentation on how to manage lifecycle operations.
 
 ### Modifying domain configurations
-You can modify the WebLogic domain configuration for both domain in PV and domain in image:
+You can modify the WebLogic domain configuration for both the domain in PV and the domain in image:
 
 * When the domain is in a PV, use WLST or WDT to change the configuration.
-* Use configuration overrides when using the domain in image.  (Point to the documentation.)
+* Use configuration overrides when using the domain in image.(Check with Tom B, "The automatic and custom overrides apply to all domains - not just domain-in-image domains." Point to the documentation.)
 
 ### Patching WebLogic and performing application updates
 
-###  Shutting down the domain
+###  Shutting down domains
 
-###  Deleting the domain
+###  Deleting domains
 (Point to sample)
 
 # Developer guide
@@ -144,7 +145,9 @@ Please take a look at our [wish list](https://github.com/oracle/weblogic-kuberne
 
 Documentation for APIs is provided here:
 
-* The operator provides a REST API that you can use to obtain information about the configuration and to initiate scaling actions. For details about how to use the REST APIs, see [Using the operator's REST services](site/rest.md). Also, see the [Swagger](https://oracle.github.io/weblogic-kubernetes-operator/swagger/index.html) documentation for the operator's REST interface.
+* The operator provides a REST API that you can use to obtain information about the configuration and to initiate scaling actions. For details about how to use the REST APIs, see [Using the operator's REST services](site/rest.md).
+
+* See the [Swagger](https://oracle.github.io/weblogic-kubernetes-operator/swagger/index.html) documentation for the operator's REST interface.
 
 * [Javadoc](https://oracle.github.io/weblogic-kubernetes-operator/apidocs/index.html) for the operator.
 
@@ -154,12 +157,12 @@ We have a public Slack channel where you can get in touch with us to ask questio
 
 ## Recent changes
 
-See [Recent changes](site/recent-changes.md) for recent changes to the operator, including any backward incompatible changes.
+See [Recent changes](site/recent-changes.md) for changes to the operator, including any backward incompatible changes.
 
 
 # Contributing to the operator
 
-Oracle welcomes contributions to this project from anyone.  Contributions may be reporting an issue with the operator, or submitting a pull request.  Before embarking on significant development that may result in a large pull request, it is recommended that you create an issue and discuss the proposed changes with the existing developers first.
+Oracle welcomes contributions to this project from anyone.  Contributions may be reporting an issue with the operator or submitting a pull request.  Before embarking on significant development that may result in a large pull request, it is recommended that you create an issue and discuss the proposed changes with the existing developers first.
 
 If you want to submit a pull request to fix a bug or enhance an existing feature, please first open an issue and link to that issue when you submit your pull request.
 

integration-tests/src/test/java/oracle/kubernetes/operator/utils/Domain.java

@@ -370,7 +370,7 @@ public void create() throws Exception {
     cmd.append(userProjectsDir)
         .append("/weblogic-domains/")
         .append(domainUid)
-        .append("/domain-custom-resource.yaml");
+        .append("/domain.yaml");
     logger.info("Running " + cmd);
     ExecResult result = ExecCommand.exec(cmd.toString());
     if (result.exitValue() != 0) {
@@ -399,7 +399,7 @@ public void destroy() throws Exception {
     cmd.append(userProjectsDir)
         .append("/weblogic-domains/")
         .append(domainUid)
-        .append("/domain-custom-resource.yaml");
+        .append("/domain.yaml");
     ExecResult result = ExecCommand.exec(cmd.toString());
     if (result.exitValue() != 0) {
       throw new RuntimeException(

kubernetes/samples/scripts/create-weblogic-domain/domain-home-on-pv/common/create-domain-job.sh

@@ -3,6 +3,13 @@
 # Licensed under the Universal Permissive License v 1.0 as shown at http://oss.oracle.com/licenses/upl.
 #
 
+function exitIfError {
+  if [ "$1" != "0" ]; then
+    echo "$2"
+    exit $1
+  fi
+}
+
 # Include common utility functions
 source ${CREATE_DOMAIN_SCRIPT_DIR}/utility.sh
 
@@ -15,9 +22,11 @@ prepareDomainHomeDir
 
 # Execute the script to create the domain
 source $script
+exitIfError $? "ERROR: $script failed."
 
 # Create the node manager
 wlst.sh -skipWLSModuleScanning ${CREATE_DOMAIN_SCRIPT_DIR}/setup-nodemanager.py
+exitIfError $? "ERROR: setup node manager failed."
 
 # DON'T REMOVE THIS
 # This script has to contain this log message. 

kubernetes/samples/scripts/create-weblogic-domain/domain-home-on-pv/create-domain.sh

@@ -84,7 +84,7 @@ function initAndValidateOutputDir {
     create-domain-inputs.yaml \
     create-domain-job.yaml \
     delete-domain-job.yaml \
-    domain-custom-resource.yaml
+    domain.yaml
 }
 
 #
@@ -216,7 +216,7 @@ function initialize {
     validationError "The template file ${deleteJobInput} for deleting a WebLogic domain_home folder was not found"
   fi
 
-  dcrInput="${scriptDir}/domain-custom-resource-template.yaml"
+  dcrInput="${scriptDir}/domain-template.yaml"
   if [ ! -f ${dcrInput} ]; then
     validationError "The template file ${dcrInput} for creating the domain custom resource was not found"
   fi
@@ -283,7 +283,7 @@ function createYamlFiles {
 
   createJobOutput="${domainOutputDir}/create-domain-job.yaml"
   deleteJobOutput="${domainOutputDir}/delete-domain-job.yaml"
-  dcrOutput="${domainOutputDir}/domain-custom-resource.yaml"
+  dcrOutput="${domainOutputDir}/domain.yaml"
 
   enabledPrefix=""     # uncomment the feature
   disabledPrefix="# "  # comment out the feature
@@ -469,34 +469,35 @@ function createDomainHome {
   while [ "$JOB_STATUS" != "Completed" -a $count -lt $max ] ; do
     sleep 30
     count=`expr $count + 1`
-    JOB_STATUS=`kubectl get pods --show-all -n ${namespace} | grep ${JOB_NAME} | awk ' { print $3; } '`
-    JOB_INFO=`kubectl get pods --show-all -n ${namespace} | grep ${JOB_NAME} | awk ' { print "pod", $1, "status is", $3; } '`
+    JOBS=`kubectl get pods --show-all -n ${namespace} | grep ${JOB_NAME}`
+    JOB_ERRORS=`kubectl logs jobs/$JOB_NAME -n ${namespace} | grep "ERROR:" `
+    JOB_STATUS=`echo $JOBS | awk ' { print $3; } '`
+    JOB_INFO=`echo $JOBS | awk ' { print "pod", $1, "status is", $3; } '`
     echo "status on iteration $count of $max"
     echo "$JOB_INFO"
 
     # Terminate the retry loop when a fatal error has already occurred.  Search for "ERROR:" in the job log file
     if [ "$JOB_STATUS" != "Completed" ]; then
-      JOB_ERRORS=`kubectl logs jobs/$JOB_NAME -n ${namespace} | grep "ERROR:" `
       ERR_COUNT=`echo $JOB_ERRORS | grep "ERROR:" | wc | awk ' {print $1; }'`
       if [ "$ERR_COUNT" != "0" ]; then
-        echo A failure was detected in the log file for job $JOB_NAME
-        echo $JOB_ERRORS
-        echo Check the log output for additional information
-        fail "Exiting due to failure - the job has failed"
+        echo "A failure was detected in the log file for job $JOB_NAME."
+        echo "$JOB_ERRORS"
+        echo "Check the log output for additional information."
+        fail "Exiting due to failure - the job has failed!"
       fi
     fi
   done
 
   # Confirm the job pod is status completed
-  JOB_POD=`kubectl get pods --show-all -n ${namespace} | grep ${JOB_NAME} | awk ' { print $1; } '`
   if [ "$JOB_STATUS" != "Completed" ]; then
-    echo The create domain job is not showing status completed after waiting 300 seconds
-    echo Check the log output for errors
+    echo "The create domain job is not showing status completed after waiting 300 seconds."
+    echo "Check the log output for errors."
     kubectl logs jobs/$JOB_NAME -n ${namespace}
     fail "Exiting due to failure - the job status is not Completed!"
   fi
 
   # Check for successful completion in log file
+  JOB_POD=`kubectl get pods --show-all -n ${namespace} | grep ${JOB_NAME} | awk ' { print $1; } '`
   JOB_STS=`kubectl logs $JOB_POD -n ${namespace} | grep "Successfully Completed" | awk ' { print $1; } '`
   if [ "${JOB_STS}" != "Successfully" ]; then
     echo The log file for the create domain job does not contain a successful completion status

kubernetes/samples/scripts/create-weblogic-domain/domain-home-on-pv/domain-custom-resource-template.yaml renamed to kubernetes/samples/scripts/create-weblogic-domain/domain-home-on-pv/domain-template.yaml

@@ -196,6 +196,17 @@ public class DomainSpec extends BaseConfiguration {
   @Description("Configuration for the clusters")
   protected Map<String, Cluster> clusters = new HashMap<>();
 
+  /**
+   * Adds a Cluster to the DomainSpec
+   *
+   * @param cluster
+   * @return
+   */
+  public DomainSpec withCluster(Cluster cluster) {
+    clusters.put(cluster.getClusterName(), cluster);
+    return this;
+  }
+
   AdminServer getOrCreateAdminServer(String adminServerName) {
     if (adminServer != null) return adminServer;
 
@@ -676,7 +687,7 @@ private void setAdminServer(AdminServer adminServer) {
     this.adminServer = adminServer;
   }
 
-  Map<String, ManagedServer> getManagedServers() {
+  public Map<String, ManagedServer> getManagedServers() {
     return managedServers;
   }
 

model/src/main/java/oracle/kubernetes/weblogic/domain/v2/DomainSpec.java

@@ -15,6 +15,7 @@
 import io.kubernetes.client.util.Watch;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Map;
 import java.util.Objects;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ConcurrentMap;
@@ -32,6 +33,8 @@
 import oracle.kubernetes.operator.helpers.JobHelper;
 import oracle.kubernetes.operator.helpers.PodHelper;
 import oracle.kubernetes.operator.helpers.ResponseStep;
+import oracle.kubernetes.operator.helpers.Scan;
+import oracle.kubernetes.operator.helpers.ScanCache;
 import oracle.kubernetes.operator.helpers.ServerKubernetesObjects;
 import oracle.kubernetes.operator.helpers.ServiceHelper;
 import oracle.kubernetes.operator.helpers.StorageHelper;
@@ -45,6 +48,7 @@
 import oracle.kubernetes.operator.steps.ListPersistentVolumeClaimStep;
 import oracle.kubernetes.operator.steps.ManagedServersUpStep;
 import oracle.kubernetes.operator.steps.WatchPodReadyAdminStep;
+import oracle.kubernetes.operator.wlsconfig.WlsDomainConfig;
 import oracle.kubernetes.operator.work.Component;
 import oracle.kubernetes.operator.work.Fiber;
 import oracle.kubernetes.operator.work.Fiber.CompletionCallback;
@@ -645,7 +649,8 @@ public StartPlanStep(DomainPresenceInfo info, Step next) {
 
     @Override
     public NextAction apply(Packet packet) {
-      Step strategy = new RegisterStep(info, getNext());
+      Step strategy =
+          Step.chain(readExistingSituConfigMap(info), new RegisterStep(info, getNext()));
       if (!info.isPopulated()) {
         strategy = Step.chain(readExistingPods(info), readExistingServices(info), strategy);
       }
@@ -684,6 +689,14 @@ private static Step readExistingServices(DomainPresenceInfo info) {
         .listServiceAsync(info.getNamespace(), new ServiceListStep(info));
   }
 
+  private static Step readExistingSituConfigMap(DomainPresenceInfo info) {
+    String situConfigMapName =
+        ConfigMapHelper.SitConfigMapContext.getConfigMapName(info.getDomainUID());
+    return new CallBuilder()
+        .readConfigMapAsync(
+            situConfigMapName, info.getNamespace(), new ReadSituConfigMapStep(info));
+  }
+
   private static class PodListStep extends ResponseStep<V1PodList> {
     private final DomainPresenceInfo info;
 
@@ -817,7 +830,7 @@ static Step createDomainUpPlan(DomainPresenceInfo info) {
     Domain dom = info.getDomain();
     Step managedServerStrategy =
         bringManagedServersUp(DomainStatusUpdater.createEndProgressingStep(new TailStep()));
-    Step adminServerStrategy = bringAdminServerUp(dom, managedServerStrategy);
+    Step adminServerStrategy = bringAdminServerUp(info, managedServerStrategy);
 
     return new UpHeadStep(
         info,
@@ -893,17 +906,53 @@ public NextAction apply(Packet packet) {
 
   // pre-conditions: DomainPresenceInfo SPI
   // "principal"
-  private static Step bringAdminServerUp(Domain dom, Step next) {
-    return StorageHelper.insertStorageSteps(dom, Step.chain(bringAdminServerUpSteps(dom, next)));
+  private static Step bringAdminServerUp(DomainPresenceInfo info, Step next) {
+    return StorageHelper.insertStorageSteps(
+        info.getDomain(), Step.chain(bringAdminServerUpSteps(info, next)));
   }
 
-  private static Step[] bringAdminServerUpSteps(Domain dom, Step next) {
+  private static class ReadSituConfigMapStep extends ResponseStep<V1ConfigMap> {
+    private final DomainPresenceInfo info;
+
+    ReadSituConfigMapStep(DomainPresenceInfo info) {
+      this.info = info;
+    }
+
+    @Override
+    public NextAction onFailure(Packet packet, CallResponse<V1ConfigMap> callResponse) {
+      return callResponse.getStatusCode() == CallBuilder.NOT_FOUND
+          ? onSuccess(packet, callResponse)
+          : super.onFailure(packet, callResponse);
+    }
+
+    @Override
+    public NextAction onSuccess(Packet packet, CallResponse<V1ConfigMap> callResponse) {
+      V1ConfigMap result = callResponse.getResult();
+      if (result != null) {
+        Map<String, String> data = result.getData();
+        String topologyYaml = data.get("topology.yaml");
+        if (topologyYaml != null) {
+          ConfigMapHelper.DomainTopology domainTopology =
+              ConfigMapHelper.parseDomainTopologyYaml(topologyYaml);
+          WlsDomainConfig wlsDomainConfig = domainTopology.getDomain();
+          ScanCache.INSTANCE.registerScan(
+              info.getNamespace(), info.getDomainUID(), new Scan(wlsDomainConfig, new DateTime()));
+        }
+      }
+
+      return doNext(packet);
+    }
+  }
+
+  private static Step[] bringAdminServerUpSteps(DomainPresenceInfo info, Step next) {
+    Domain dom = info.getDomain();
     List<Step> resources = new ArrayList<>();
     resources.add(new ListPersistentVolumeClaimStep(null));
     resources.add(
         JobHelper.deleteDomainIntrospectorJobStep(
             dom.getDomainUID(), dom.getMetadata().getNamespace(), null));
-    resources.add(JobHelper.createDomainIntrospectorJobStep(PodHelper.createAdminPodStep(null)));
+    resources.add(JobHelper.createDomainIntrospectorJobStep(null));
+    resources.add(PodHelper.createAdminPodStep(null));
     resources.add(new BeforeAdminServiceStep(null));
     resources.add(ServiceHelper.createForServerStep(null));
     resources.add(new WatchPodReadyAdminStep(Main.podWatchers, null));

operator/src/main/java/oracle/kubernetes/operator/DomainProcessorImpl.java

@@ -304,7 +304,7 @@ private static String getOperatorNamespace() {
     }
   }
 
-  static class SitConfigMapContext extends ConfigMapContext {
+  public static class SitConfigMapContext extends ConfigMapContext {
     Map<String, String> data;
     String cmName;
 
@@ -513,7 +513,7 @@ static String extractFilename(String line) {
     return fname;
   }
 
-  static DomainTopology parseDomainTopologyYaml(String topologyYaml) {
+  public static DomainTopology parseDomainTopologyYaml(String topologyYaml) {
     ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
 
     try {