Deploying to Git

Deploying K8s Objects to Git

Process

The change from direct API calls to creating manifests in a Git repo changes the way OpenUnison communicates with the API server. The process for GitOps is:

1. In a workflow, add objects to a Secret
2. Once ready to commit, launch a Job that will checkout the repo, add the files from the Secret, and push to the repo
3. An OpenUnison scheduled task will cleanup Job and Secret objects

From an audit perspective, objects are logged against the k8s target, just as before.

Parts List

1. Git repo with SSH support
2. An SSH public and private key, with the public key added as a "Deploy key" to the git repo
3. Create a new secret in the openunison namespace named git-ssh-key from the deploy key's private key. Name the private key id_rsa
4. Import docker.io/tremolosecurity/git-push:latest into you local container registry

Updating Workflows

First, create a Secret for storing the objects. This should be done at the beginning of the workflow:

<customTask className="com.tremolosecurity.provisioning.tasks.CreateK8sObject">
                  <param name="targetName" value="k8s"/>
                  <param name="template" >
<![CDATA[ 
---
kind: Secret
apiVersion: v1
metadata:
  name: git-secret-$nameSpace$
  labels:
    deleteAfterJob: "true"
    forJob: push-to-git-$nameSpace$
data: {}
]]>

                  </param>
                  <param name="kind" value="Secret" />
                  <param name="srcType" value="yaml" />
                  <param name="url" value="/api/v1/namespaces/openunison/secrets" />
              </customTask>

Note the label forJob must be present.

Each CreateK8sObject task in a workflow needs to be updated to support writing to git. As an example, the existing creation of the Namespace looks like:

<customTask className="com.tremolosecurity.provisioning.tasks.CreateK8sObject">
                    <param name="targetName" value="k8s"/>
                    <param name="template" value="{&quot;kind&quot;:&quot;Namespace&quot;,&quot;apiVersion&quot;:&quot;v1&quot;,&quot;metadata&quot;:{&quot;name&quot;:&quot;$nameSpace$&quot;,&quot;labels&quot;:{&quot;name&quot;:&quot;$nameSpace$&quot;}}}" />

                    </param>
                    <param name="kind" value="Namespace" />
                    <param name="url" value="/api/v1/namespaces" />
                </customTask>

it needs to be converted to:

<customTask className="com.tremolosecurity.provisioning.tasks.CreateK8sObject">
                    <param name="targetName" value="k8s"/>
                    <param name="template" >
<![CDATA[ 
---
kind: Namespace
apiVersion: v1
metadata:
  name: "$nameSpace$"
  labels:
    name: "$nameSpace$"
]]>

                    </param>
                    <param name="kind" value="Namespace" />
                    <param name="url" value="/api/v1/namespaces" />
                    <param name="srcType" value="yaml" />
                    <param name="writeToSecret" value="true" />
                    <param name="secretName" value="git-secret-$nameSpace$" />
                    <param name="path" value="src/main/ns/$nameSpace$/namespace-$nameSpace$.yaml" />
                    <param name="secretNamespace" value="openunison" />
                </customTask>

First, convert the value of the template parameter by reversing the XML escaping the JSON and converting to YAML. The param tag now supports embedding the yaml directly in the value of the tag instead of an attribute using the <![CDATA[ ]]> notation. Also, add the the following parameters to each task being written to git:

<!-- tells openunison to interpret the template as YAML instead of JSON -->
<param name="srcType" value="yaml" />
<!-- tells openunison to write to the Secret instead of directly to the API server -->
<param name="writeToSecret" value="true" />
<!-- The secret to write to, created at the beginning of the workflow -->
<param name="secretName" value="git-secret-$nameSpace$" />
<!-- The path, relative to the root of the git repo, to write the yaml to -->
<param name="path" value="src/main/ns/$nameSpace$/namespace-$nameSpace$.yaml" />
<!-- The namespace where the Secret created at the beginning of the workflow is stored -->
<param name="secretNamespace" value="openunison" />

At then end of the workflow, when you wish to provision the objects to git, add a Job:

<customTask className="com.tremolosecurity.provisioning.tasks.CreateK8sObject">
                  <param name="targetName" value="k8s"/>
                  <param name="template" >
<![CDATA[ 
---
kind: Job
apiVersion: batch/v1
metadata:
  name: push-to-git-$nameSpace$
  namespace: openunison
  labels:
    jobtype: gitpush
    forNamespace: $nameSpace$
spec:
  parallelism: 1
  completions: 1
  backoffLimit: 1
  selector:
    matchLabels:
      job-name: push-to-git-$nameSpace$
  template:
    metadata:
      labels:
        job-name: push-to-git-$nameSpace$
    spec:
      volumes:
        - name: ssh-keys
          secret:
            secretName: git-ssh-key
            defaultMode: 420
        - name: ssh-dir
          emptyDir: {}
        - name: git-dir
          emptyDir: {}
        - name: secret-dir
          secret:
            secretName: git-secret-$nameSpace$
            defaultMode: 420
      containers:
        - name: push-to-git
          image: 'docker.io/tremolosecurity/git-push:latest'
          command:
            - /usr/local/openunison/pushtogit.sh
            - /usr/src
            - /usr/git
            - git@github.com:repo/test-git-push.git
          resources: {}
          volumeMounts:
            - name: ssh-keys
              readOnly: true
              mountPath: /etc/ssh-keys
            - name: ssh-dir
              mountPath: /usr/local/openunison/.ssh
            - name: secret-dir
              mountPath: /usr/src
            - name: git-dir
              mountPath: /usr/git
          env:
            - name: GIT_EMAIL
              value: user@domain.com
            - name: GIT_USERNAME
              value: user
            - name: GIT_COMMIT_MSG
              value: For workflow $WORKFLOW.id$
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
      restartPolicy: Never
      serviceAccountName: default
      serviceAccount: default
]]>

                  </param>
                  <param name="kind" value="Job" />
                  <param name="url" value="/apis/batch/v1/namespaces/openunison/jobs" />
                  <param name="srcType" value="yaml" />
                  <param name="writeToSecret" value="false" />
                  
              </customTask>

Update the Job per your needs (ie pointing to a local docker repository).

Cleanup

Add the following job to unison.xml inside of the <scheduler> tag:

<job className="com.tremolosecurity.provisioning.jobs.ClearJobs" name="clearJobs" group="management">
            <!-- When to run the job -->
          	<cronSchedule
            seconds="0"
            minutes="*"
            hours="*"
            dayOfMonth="*"
            month="*"
            dayOfWeek="?"
            year="*"
          />
          <param name="target" value="k8s" />
          <param name="uri" value="/apis/batch/v1/namespaces/openunison/jobs" />  
          <param name="labels" value="jobtype=gitpush" />
          <param name="workflow" value="deleteNewNSJob" />
          <param name="runWorkflowAsUsername" value="system" />
          <param name="runWorkflowAsUsernameAttribute" value="sub" />

            </job>

This will delete completed Job objects and their associated Secret once a minute. Add the following workflow as src/man/webapp/WEB-INF/workflows/deleteNewNSJob.xml:

<workflow  name="deleteNewNSJob" label="deleteNewNsJob" description="Delete new namespace job" inList="false" orgid="687da09f-8ec1-48ac-b035-f2f182b9bd1e">
    <tasks>
        <customTask className="com.tremolosecurity.provisioning.tasks.DeleteK8sObject">
            <param name="targetName" value="k8s"/>
            <param name="kind" value="Secret" />
            <param name="url" value="/api/v1/namespaces/openunison/secrets/git-secret-$job_labels_forNamespace$" />
        </customTask>
        <customTask className="com.tremolosecurity.provisioning.tasks.DeleteK8sObject">
            <param name="targetName" value="k8s"/>
            <param name="kind" value="Job" />
            <param name="url" value="/apis/batch/v1/namespaces/openunison/jobs/$job_name$" />
        </customTask>
    </tasks>
</workflow>