INITIAL version of CI/CD doc (#1035)

* add first version of ci/cd documenation

* review and edit, jenkins-ignore

* fix typo, jenkins-ignore

* updates from review comments

* updates from review

Author: markxnelson

docs-source/content/userguide/cicd/_index.md

@@ -0,0 +1,65 @@
+---
+title: "CI/CD considerations"
+date: 2019-04-11T13:01:55-04:00
+weight: 5
+description: "Learn about managing domain images with CI/CD."
+draft: false
+---
+
+### Overview
+
+In this section, we will discuss the recommended techniques for managing the evolution
+and mutation of Docker images to run WebLogic in Kubernetes.  There are several
+approaches and techniques available, and the choice of which to use depends very
+much on your particular requirements.  We will start with a review of the "problem
+space," and then talk about the considerations that would lead us to choose various
+approaches.  We will provide details about several approaches to implementing
+CI/CD and links to examples.
+
+### Review of the problem space
+
+Kubernetes makes a fundamental assumption that Docker images are immutable,
+that they contain no state, and that updating them is as simple as throwing
+away a pod/container and replacing it with a new one that uses a newer version
+of the Docker image.  These assumptions work very well for microservices
+applications, but for more traditional workloads, we need to do some extra
+thinking and some extra work to get the behavior we want.
+
+CI/CD is an area where the standard assumptions aren't always suitable.  In the
+microservices architecture, you typically minimize dependencies and build
+images from scratch with all of the dependencies in them.  You also typically
+keep all of the configuration outside of the image, for example, in Kubernetes config
+maps or secrets, and all of the state outside of the image too.  This makes
+it very easy to update running pods with a new image.
+
+Let's consider how a WebLogic image is different.  There will, of course, be a
+base layer with the operating system; let's assume it is
+[Oracle Linux "slim"](https://hub.docker.com/_/oraclelinux/).  Then you need
+a JDK and this is very commonly in another layer.  Many people will use
+the officially supported JDK images from the Docker Store, like the
+[Server JRE image](https://hub.docker.com/_/oracle-serverjre-8), for example.  On
+top of this, you need the WebLogic Server binaries (the "Oracle Home").  On top
+of that, you may wish to have some patches or updates installed.  And then
+you need your domain, that is the configuration.
+
+There is also other information associated with a domain that needs to live
+somewhere, for example leasing tables, message and transaction stores, and so
+on.  We recommend that these be kept in a database to take advantage of built-in 
+database server HA, and the fact that disaster recovery of sites across all 
+but the shortest distances almost always requires using a single database 
+server to consolidate and replicated data (DataGuard).
+
+There are two common approaches on how to structure these components.  The first,
+which we call "domain on persistent volume," places the JDK and WebLogic binaries
+in the Docker image, but the domain is kept on a separate persistent storage
+outside of the image.  The second approach puts the JDK, WebLogic binaries
+and the domain all in the Docker image.  Both of these approaches are perfectly
+valid (and fully supported) and they have various advantages and disadvantages.
+
+We have listed the [relative advantages of these two approaches here]({{< relref "/userguide/managing-domains/choosing-a-model/_index.md" >}}).
+
+One of the key differences between these approaches is how many Docker images
+you have, and therefore, how you build and maintain them - your image CI/CD
+process.  Let's take a short detour and talk about Docker image layering.
+
+{{% children style="h4" description="true" %}}

docs-source/content/userguide/cicd/choose-an-approach.md

@@ -0,0 +1,60 @@
+---
+title: "Choose an approach"
+date: 2019-04-11T13:36:57-04:00
+draft: false
+weight: 3
+description: "How to choose an approach."
+---
+
+Let's review what we have discussed and talk about when we might want to use
+various approaches.  We can start by asking ourselves questions like these:
+
+
+- *Can you make the desired change with a configuration override?*  
+  The WebLogic Kubernetes Operator allows you to inject a number of [configuration
+  overrides]({{< relref "/userguide/managing-domains/configoverrides/_index.md" >}})
+  into your pods before starting any servers in the domain.  This allows you to use 
+  the same image for multiple
+  different configurations.  A good example would be changing the settings for a data
+  source, for example. You may wish to have a larger connection pool in your production
+  environment than you do in your development/test environments.  You probably also
+  want to have different credentials.  You may want to change the service name, and
+  so on.  All of these kinds of updates can be made with configuration overrides.
+  These are placed in a Kubernetes config map, that is, they are outside of the image, so
+  they do not require rebuilding the Docker image.  If all of your changes fit into
+  this category, it is probably much better to just use configuration overrides
+  instead of building a new image.  
+- *Are you only changing the WebLogic configuration, for example, deploying or updating an
+  application, changing a resource configuration in a way that is not supported by
+  configuration overrides, and such?*  
+  If your changes fit into this category, and you have used the "domain-in-image"
+  approach and the Docker layering model, then you only need to update the top layer
+  of your image.  This is relatively easy compared to making changes in lower layers.
+  You could create a new layer with the changes, or you could rebuild/replace the
+  existing top layer with a new one.  Which approach you choose depends mainly on
+  whether you need to maintain the same domain encryption keys or not.
+- *Do you need to be able to do a rolling restart?*
+  If you need to do a rolling restart, for example to maintain the availability of
+  your applications, then you need to make sure the new domain layer has the same
+  domain encryption keys.  You cannot perform a rolling restart of a domain if the
+  new members have a different encryption key.
+- *Do you need to mutate something in a lower layer, for example, patch WebLogic, the JDK, or Linux?*  
+  If you need to make an update in a lower layer, then you will need to rebuild that
+  layer and all of the layers above it.  This means that you will need to rebuild the
+  domain layer.  You will need to determine if you need to keep the same domain encryption keys.
+
+The diagram below summarizes these concerns in a decision tree for the “domain in image” case:
+
+![Decision model for the "domain in image" approach](/weblogic-kubernetes-operator/images/flowchart.png)
+
+If you are using the "domain on persistent storage" approach, many of these concerns become
+moot because you have an effective separation between your domain and the Docker image.
+There is still the possibility that an update in the Docker image could affect your domain;
+for example, if you updated the JDK, you may need to update some of your domain scripts
+to reflect the new JDK path.  
+
+However, in this scenario, your environment is much closer to what you are probably used
+to in a traditional (non-Kubernetes) environment, and you will probably find that all of
+the practices you used from that pre-Kubernetes environment are directly applicable here
+too, with just some small modifications.  For example, applying a WebLogic patch would
+now involve building a new Docker image.

docs-source/content/userguide/cicd/how-to-copy-domains.md

@@ -0,0 +1,42 @@
+---
+title: "How to copy domains"
+date: 2019-04-11T13:48:15-04:00
+draft: false
+weight: 5
+description: "How to copy domains."
+---
+
+The recommended approach to save a copy of a domain is to simply ZIP (or tar)
+the domain directory.  However, there is a very important caveat with this
+recommendation - when you unzip the domain, it must go back into exactly
+the same location (Domain Home) in the (new) file system.  Using this
+approach will maintain the same domain encryption key.  
+
+The best practice/recommended approach is to create a "primordial domain"
+which does not contain any applications or resources,
+and to create a ZIP file of this domain before starting any servers.  
+
+> The domain ZIP must be created before starting servers.  
+
+When servers are started the first time, they will encrypt various other data.
+Make sure you create the ZIP file before starting servers for the first time.
+The primordial domain ZIP file should be stored in a safe place where the CI/CD
+can get it when needed, for example in a secured Artifactory repository (or
+something similar).  
+
+{{% notice warning %}}
+Remember, anyone who gets access to this ZIP file can get access
+to the domain encryption key, so it needs to be protected appropriately.
+{{% /notice %}}
+
+Every time you run your CI/CD pipeline to create a new mutation of the domain,
+it should retrieve and unzip the primordial domain first, and then apply changes
+to that domain using tools like WDT or WLST (see [here]({{< relref "/userguide/cicd/tools.md" >}})).
+
+
+> Always use external state.
+
+You should always keep state outside the Docker image.  This means that you should
+use JDBC stores for leasing tables, JMS and Transaction stores,
+EJB timers, JMS queues, and so on.  This ensures that data will not be lost when
+a container is destroyed.

docs-source/content/userguide/cicd/layering.md

@@ -0,0 +1,58 @@
+---
+title: "Docker image layering"
+date: 2019-04-11T13:15:32-04:00
+weight: 1
+draft: false
+description: "Learn about Docker image layering and why it is important."
+---
+
+
+Docker images are composed of layers, as shown in the diagram below.  If you download
+the standard `weblogic:12.2.1.3` image from the [Docker Store](https://hub.docker.com/_/oracle-weblogic-server-12c),
+then you can see these layers using the command  `docker inspect store/oracle/weblogic:12.2.1.3`
+(the domain layer will not be there).  You are not required to use layers, but
+efficient use of layers is considered a best practice.
+
+![Docker image layers](/weblogic-kubernetes-operator/images/layers.png)
+
+#### Why is it important to maintain the layering of images?
+
+Layering is an important technique in Docker images.  Layers are important because they
+are shared between images.  Let's consider an example.  In the diagram below, we have
+two domains that we have built using layers.  The second domain has some additional
+patches that we needed on top of those provided in the standard WebLogic image.  Those
+are installed in their own layer, and then the second domain is created in another
+layer on top of that.
+
+Let's assume we have a three-node Kubernetes cluster and we are running both domains
+in this cluster.  Sooner or later, we will end up with servers in each domain running
+on each node, so eventually all of the image layers are going to be needed on all of
+the nodes.  Using the approach shown below (that is, standard Docker layering techniques)
+we are going to need to store all six of these layers on each node.  If you add up the
+sizes, then you will see that it comes out to about 1.5GB per node.
+
+![Docker images with layers](/weblogic-kubernetes-operator/images/more-layers.png)
+
+Now, let's consider the alternative, where we do not use layers, but instead,
+build images for each domain and put everything in one big layer (this is often
+called "squashing" the layers).  In this case, we have the same content, but if
+you add up the size of the images, you get 2.9GB per node. That’s almost twice the size!
+
+![Docker images without layers](/weblogic-kubernetes-operator/images/no-layers.png)
+
+With only two domains, you start to see the problem.  In the layered approach, each
+new domain is adding only a relatively very small increment.  In the non-layered
+approach, each new domain is essentially adding the entire stack over again.  Imagine
+if we had ten domains, now the calculation looks like this:
+
+|                            | With Layers       | Without Layers    |
+| -------------------------- | ----------------- | ----------------- |
+| Shared Layers              | 1.4GB             | 0GB               |
+| Dedicated/different layers | 10 x 10MB = 100MB | 10 x 1.5GB = 15GB |
+| Total per node             | 1.5GB             | 15GB              |
+
+
+You can see how the amount of storage for images really starts to add up, and it
+is not just a question of storage.  When Kubernetes creates a container from an
+image, the size of the image has an impact on how long it takes to create and
+start the container.

docs-source/content/userguide/cicd/mutate-the-domain-layer.md

@@ -0,0 +1,43 @@
+---
+title: "Mutate the domain layer"
+date: 2019-04-11T13:43:41-04:00
+draft: false
+weight: 4
+description: "How to mutate the domain layer."
+---
+
+If you need to mutate the domain layer, and keep the same domain encryption keys,
+then there are some choices about how to implement that, as alluded to previously.
+Let's explore those in some more detail now.
+
+The first option is to implement each mutation as a delta to the previous state.
+This is conceptually similar to how immutable objects (like Java Strings) are
+implemented, a "copy on write" approach applied to the domain configuration as a
+unit.  This does have the advantage that it is simple to implement, but the
+disadvantage that your builds would depend on the previous good build and
+this is somewhat contrary to typical CI/CD practices.  You also have to work
+out what to do with the bad builds, or "holes" in the sequence.
+
+![Mutating the previous state](/weblogic-kubernetes-operator/images/n-1.png)
+
+An alternative is to capture a "primordial state" of the domain before starting
+the sequence.  In practical terms, this might mean creating a very simple domain
+with no applications or resources in it, and "saving" it before ever starting
+any servers.  This primordial domain (let’s call it t=0) would then be used
+to build each mutation.  So each state is built from t=0, plus all of the
+changes up to that point.  
+
+Said another way, each build would start with t=0 as the base image and extend it.
+This eliminates the need to keep each intermediate state, and would also likely
+have benefits when you remove things from the domain, because you would not have
+"lost" ("whited out" is the Docker layer term) space in the intermediate layers.
+Although, these layers tend to be relatively small, so this is possibly not a big issue.
+
+![Rebuilding from a primordial state](/weblogic-kubernetes-operator/images/primordial.png)
+
+This approach is probably an improvement.  It does get interesting though when you
+update a lower layer, for example when you patch WebLogic or update the JDK.  When
+this happens, you need to create another base image, shown in the diagram as v2 t-0.
+All of the mutations in this new chain are based on this new base image.  So that
+still leaves us with the problem of how to take the domain from the first series
+(v1 t=0 to t=3) and "copy" it across to the second series (v2).

docs-source/content/userguide/cicd/tools.md

@@ -0,0 +1,64 @@
+---
+title: "Tools"
+date: 2019-04-11T13:50:15-04:00
+draft: false
+weight: 6
+description: "Tools that are available to build CI/CD pipelines."
+---
+
+### WebLogic Deploy Tooling (WDT)
+
+You can use several of the [WDT tools](https://github.com/oracle/weblogic-deploy-tooling)
+in a CI/CD pipeline. For example, the
+`createDomain` tool creates a new domain based on a simple model, and
+`updateDomain` (and `deployApps`) uses the same model concept to update
+an existing domain (preserving the same domain encryption key). The `deployApps`
+tool is very similar to the updateDomain tool, but limits what can be updated
+to application related configuration attributes such as data sources and
+application archives.  The model used by these tools is a sparse set of
+attributes needed to create or update the domain. A model can be as sparse
+as providing only the WebLogic Server administrative password, although not very
+interesting.  A good way to get a jumpstart on a model is to use the
+`discoverDomain` tool in WDT which builds a model based on an existing domain.  
+
+Other than the tools themselves, there are three components to the WDT tools:  
+
+- *The Domain Model* - Metadata model describing the desired domain.  
+  The metadata domain model can be YAML or JSON and is documented [here](https://github.com/oracle/weblogic-deploy-tooling#the-metadata-model).
+- *The Archive Zip* - Binaries to supplement the model.  
+  All binaries needed to supplement the model must be specified in an archive
+  file, which is just a ZIP file with a specific directory structure. Optionally,
+  the model can be stored inside the ZIP file, if desired. Any binaries not
+  already on the target system must be in the ZIP file so that the tooling
+  can extract them in the target domain.
+- *The Properties File* - A standard Java properties file.  
+  A property file used to provide values to placeholders in the model.
+
+### WDT Create Domain Samples
+
+- (Docker) A sample for creating a domain in a Docker image with WDT can be found
+  [here](https://github.com/oracle/weblogic-deploy-tooling/tree/master/samples/docker-domain).
+- (Kubernetes) A similar sample of creating a domain in a Docker image with WDT
+  can be found in the WebLogic Kubernetes Operator project for creating a
+  [domain-in-image with WDT](https://oracle.github.io/weblogic-kubernetes-operator/samples/simple/domains/domain-home-in-image/).
+
+### WebLogic Scripting Tool (WLST)
+
+You can use WLST scripts to create and/or update domains in a CI/CD pipeline.
+We recommend that you use offline WLST for this purpose.  There may be some
+scenarios where it is necessary to use WLST online, but we recommend that
+you do that only as an exception, and when absolutely necessary.
+
+If you do not already have WLST scripts, we recommend that you consider
+using WebLogic Deploy Tooling (WDT) instead.  It provides a more declarative
+approach to domain creation, whereas WLST is more of an imperative scripting
+language.  WDT provides advantages like being able to use the same model with
+different versions of WebLogic, whereas you may need to update WLST scripts
+manually when migrating to a new version of WebLogic for example.
+
+### WebLogic pack and unpack tools
+
+WebLogic Server provides tools called "pack" and "unpack" that can be used to
+"clone" a domain.  These tools do not preserve the domain encryption key.
+You can use these tools to make copies of domains in scenarios when you do
+not need the same domain encryption key.