|
| 1 | +# Batch Changes design |
| 2 | + |
| 3 | +Why is the [Batch Changes](../../../batch_changes/index.md) feature designed the way it is? |
| 4 | + |
| 5 | +## Principles |
| 6 | + |
| 7 | +- **Declarative API** (not imperative). You declare your intent, such as "lint files in all repositories with a `package.json` file"{/* <!-- TODO(sqs): thorsten had a suggestion to make this quote use non-imperative language but I can't find the comment -->. Batch Changes figures out how to achieve your desired state. The external state (of repositories, changesets, code hosts, access tokens, etc.) can change at any time, and temporary errors frequently occur when reading and writing to code hosts. These factors would make an imperative API very cumbersome because each API client would need to handle the complexity of the distributed system. */} |
| 8 | + |
| 9 | +- **Define a batch change in a file** (not some online API). The source of truth of a batch change's definition is a file that can be stored in version control, reviewed in code review, and re-applied by CI. This is in the same spirit as IaaC (infrastructure as code; e.g., storing your Terraform/Kubernetes/etc. files in Git). We prefer this approach over a (worse) alternative where you define a batch change in a UI with a bunch of text fields, checkboxes, buttons, etc., and need to write a custom API client to import/export the batch change definition. |
| 10 | +- **Shareable and portable.** You can share your batch specs, and it's easy for other people to use them. A batch spec expresses an intent that's high-level enough to (usually) not be specific to your own particular repositories. You declare and inject configuration and secrets to customize it instead of hard-coding those values. |
| 11 | +- **Large-scale.** You can run batch changes across 10,000s of repositories. It might take a while to compute and push everything, and the current implementation might cap out lower than that, but the fundamental design scales well. |
| 12 | +- **Accommodates a variety of code hosts and review/merge processes.** Specifically, we don't want Batch Changes to only work for GitHub pull requests. (See [current support list](../../../batch_changes/index.md#supported-code-hosts-and-changeset-types).) |
| 13 | + |
| 14 | +## Comparison to other distributed systems |
| 15 | + |
| 16 | +Kubernetes is a distributed system with an API that many people are familiar with. Batch Changes is also a distributed system. All APIs for distributed systems need to handle a similar set of concerns around robustness, consistency, etc. Here's a comparison showing how these concerns are handled for a Kubernetes [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) and a batch change. In some cases, we've found Kubernetes to be a good source of inspiration for the Batch Changes API, but resembling Kubernetes is **not** an explicit goal. |
| 17 | + |
| 18 | + |
| 19 | +{/* <table> |
| 20 | + <tr> |
| 21 | + <td/> |
| 22 | + <th>Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Deployment</a></th> |
| 23 | + <th>Sourcegraph Batch Changes</th> |
| 24 | + </tr> |
| 25 | + <tr> |
| 26 | + <th>What underlying thing does this API manage?</th> |
| 27 | + <td>Pods running on many (possibly unreliable) nodes</td> |
| 28 | + <td>Branches and changesets on many repositories that can be rate-limited and externally modified (and our authorization can change)</td> |
| 29 | + </tr> |
| 30 | + <tr> |
| 31 | + <th>Spec YAML</th> |
| 32 | + <td> |
| 33 | + <pre><code><em># File: foo.Deployment.yaml</em> |
| 34 | +apiVersion: apps/v1 |
| 35 | +kind: Deployment |
| 36 | +metadata: |
| 37 | + name: nginx-deployment |
| 38 | +spec:<div style="padding:0.5rem;margin:0 -0.5rem;background-color:rgba(0,0,255,0.25)"><strong> # Evaluate this to enumerate instances of...</strong> |
| 39 | + replicas: 2</div> |
| 40 | +<div style="padding:0.5rem;margin:0 -0.5rem;background-color:rgba(255,0,255,0.25)"><strong> # ...this template.</strong> |
| 41 | + template: |
| 42 | + metadata: |
| 43 | + labels: |
| 44 | + app: nginx |
| 45 | + spec: |
| 46 | + containers: |
| 47 | + - name: nginx |
| 48 | + image: nginx:1.14.2 |
| 49 | + ports: |
| 50 | + - containerPort: 80</div></pre></code> |
| 51 | + </td> |
| 52 | + <td> |
| 53 | + <pre><code><em># File: hello-world.batch.yaml</em> |
| 54 | +name: hello-world |
| 55 | +description: Add Hello World to READMEs |
| 56 | +
|
| 57 | +<div style="padding:0.5rem;margin:0 -0.5rem;background-color:rgba(0,0,255,0.25)"><strong># Evaluate this to enumerate instances of...</strong> |
| 58 | +on: |
| 59 | + - repositoriesMatchingQuery: file:README.md |
| 60 | +
|
| 61 | +steps: |
| 62 | + - run: echo Hello | tee -a $(find -name '*.md') |
| 63 | + container: alpine:3 |
| 64 | +</div> |
| 65 | +<div style="padding:0.5rem;margin:0 -0.5rem;background-color:rgba(255,0,255,0.25)"><strong># ...this template.</strong> |
| 66 | +changesetTemplate: |
| 67 | + title: Hello World |
| 68 | + body: My first batch change! |
| 69 | + branch: hello-world |
| 70 | + commit: |
| 71 | + message: Append Hello to .md files |
| 72 | + published: false</pre></code> |
| 73 | + </td> |
| 74 | + </tr> |
| 75 | + <tr> |
| 76 | + <th>How desired state is computed</th> |
| 77 | + <td> |
| 78 | + <ol> |
| 79 | + <li>Evaluate <code>replicas</code>, etc. (blue) to determine pod count and other template inputs</li> |
| 80 | + <li>Instantiate <code>template</code> (pink) once for each pod to produce PodSpecs</li> |
| 81 | + </ol> |
| 82 | + </td> |
| 83 | + <td> |
| 84 | + <ol> |
| 85 | + <li>Evaluate <code>on</code>, <code>steps</code> (blue) to determine list of patches</li> |
| 86 | + <li>Instantiate <code>changesetTemplate</code> (purple) once for each patch to produce ChangesetSpecs |
| 87 | +</li> |
| 88 | + </ol> |
| 89 | + </td> |
| 90 | + </tr> |
| 91 | + <tr> |
| 92 | + <th>Desired state consists of...</th> |
| 93 | + <td> |
| 94 | + <ul> |
| 95 | + <li>DeploymentSpec file (the YAML above)</li> |
| 96 | + <li>List of PodSpecs (template instantiations)</li> |
| 97 | + </ul> |
| 98 | + </td> |
| 99 | + <td> |
| 100 | + <ul> |
| 101 | + <li>batch spec file (the YAML above)</li> |
| 102 | + <li>List of ChangesetSpecs (template instantiations)</li> |
| 103 | + </ul> |
| 104 | + </td> |
| 105 | + </tr> |
| 106 | + <tr> |
| 107 | + <th>Where is the desired state computed?</th> |
| 108 | + <td>The deployment controller (part of the Kubernetes cluster) consults the DeploymentSpec and continuously computes the desired state.</td> |
| 109 | + <td> |
| 110 | + <p>The <a href="https://github.com/sourcegraph/src-cli">Sourcegraph CLI</a> (running on your local machine, not on the Sourcegraph server) consults the batch spec and computes the desired state when you invoke <code>src batch apply</code>.</p> |
| 111 | + <p><strong>Difference vs. Kubernetes:</strong> A batch change's desired state is computed locally, not on the server. It requires executing arbitrary commands, which is not yet supported by the Sourcegraph server. See Batch Changes known issue "<a href="../../../batch_changes/explanations/introduction_to_batch_changes#server-execution">Batch Changes steps are run locally...</a>".</p> |
| 112 | + </td> |
| 113 | + </tr> |
| 114 | + <tr> |
| 115 | + <th>Reconciling desired state vs. actual state</th> |
| 116 | + <td>The "deployment controller" reconciles the resulting PodSpecs against the current actual PodSpecs (and does smart things like rolling deploy).</td> |
| 117 | + <td>The "Batch Changes controller" (i.e., our backend) reconciles the resulting ChangesetSpecs against the current actual changesets (and does smart things like gradual roll-out/publishing and auto-merging when checks pass).</td> |
| 118 | + </tr> |
| 119 | +</table> */} |
| 120 | + |
| 121 | +These docs explain more about Kubernetes' design: |
| 122 | + |
| 123 | +- [Kubernetes Object Management](https://kubernetes.io/docs/concepts/overview/working-with-objects/object-management/) |
| 124 | +- [Kubernetes Controllers](https://kubernetes.io/docs/concepts/architecture/controller/) |
| 125 | + - [Desired versus current state](https://kubernetes.io/docs/concepts/architecture/controller/#desired-vs-current) |
| 126 | +- [Kubernetes Architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/architecture.md) |
| 127 | +- [Kubernetes General Configuration Tips](https://kubernetes.io/docs/concepts/configuration/overview/#general-configuration-tips) |
| 128 | +- [Kubernetes Design and Development Explained](https://thenewstack.io/kubernetes-design-and-development-explained/) |
0 commit comments