|
| 1 | +# RFC 182 - Allow remote references to .taskcluster.yml files processed by Taskcluster-GitHub |
| 2 | +* Comments: [#182](https://github.com/taskcluster/taskcluster-rfcs/pull/182) |
| 3 | +* Proposed by: @bhearsum |
| 4 | + |
| 5 | +# Summary |
| 6 | + |
| 7 | +Allow `.taskcluster.yml` files in GitHub repositories to outsource most of their contents to a `.taskcluster.yml` stored elsewhere. |
| 8 | + |
| 9 | +## Motivation |
| 10 | + |
| 11 | +`.taskcluster.yml` files in GitHub repositories are responsible for creating tasks (or not) in response to various GitHub events. In some cases (eg: when a decision task is used to delegate most of this to a separate tool), these files end up looking almost entirely identical, and hundreds of lines of JSON-e ends up duplicated across many different repositories. This duplication is a significant maintenance burden, and easily results in unwanted differences. |
| 12 | + |
| 13 | +Allowing a `.taskcluster.yml` to reference a file stored elsewhere will allow for deduplication of the bulk of the content in these files, resolving the concerns noted above. |
| 14 | + |
| 15 | +# Details |
| 16 | + |
| 17 | +Two new variables will be introduced to `.taskcluster.yml` files. The first, `config-from`, will allow a reference to a different GitHub repository to be provided that points at a full-fledged `.taskcluster.yml` hosted elsewhere. This reference may be pinned to a specific revision, or to a tag or branch name. The second, `context`, will allow for variables to be defined that will be included with the JSON-e context when the referenced `.taskcluster.yml` is rendered. Here is an example of a possible concrete `.taskcluster.yml`: |
| 18 | + |
| 19 | +``` |
| 20 | +--- |
| 21 | +version: 1 |
| 22 | +config-from: github.com/taskcluster/taskgraph/data/taskcluster-yml-github.yml@main |
| 23 | +context: |
| 24 | + project-name: mozillavpn |
| 25 | + scopes: |
| 26 | + - secrets:get:project/mozillavpn/* |
| 27 | +``` |
| 28 | + |
| 29 | +When `config-from` is present in the `.taskcluster.yml`, the existing `tasks` top level key is not valid (they are mutually exclusive). Other top level keys, such as `policy` and `reporting` continue to be valid. If these keys are present in both the `config-from` `.taskcluster.yml` and the project repository `.taskcluster.yml`, the value in the latter will take precedence. |
| 30 | + |
| 31 | +The new `context` key is optional, and only permitted when `config-from` is present. |
| 32 | + |
| 33 | +In addition to the extra `context` that project repos may define (above), Taskcluster-GitHub will provide the following context as well: |
| 34 | +* `taskclusterYmlRepo` will be the full URL to the repo that contains the final `.taskcluster.yml`. In cases where `config-from` is not present this will be the URL of the project repository. In cases where it is present, it will be the reposisitory part of the `config-from` repository, as an `https` URL. |
| 35 | +* `taskclusterYmlRevision` will be the revision in the `taskclusterYmlRepo` of the `.taskcluster.yml` rendered by Taskcluster-GitHub. In cases where `config-from` is set, and a tag or branch reference has been used, it will be deferenced to a revision. |
| 36 | + |
| 37 | +This additional context is intended to allow tasks defined in `.taskcluster.yml` to republish as metadata, but it may have future uses as well. |
| 38 | + |
| 39 | +## Top level key precedence |
| 40 | + |
| 41 | +This change introduces two new top level keys, and introduces two possible sources for some existing ones. Here is a summary of what is allowed where, and the order of precedence for keys that are allowed in both places: |
| 42 | +| Key | Permitted in project repo? | Permitted in referenced .taskcluster.yml? | Precedence? | |
| 43 | +|-------------------|----------------------------|-------------------------------------------|------------------| |
| 44 | +| tasks | Y | Y | N/A(\*) | |
| 45 | +| policy | Y | Y | project repo | |
| 46 | +| reporting | Y | Y | project repo | |
| 47 | +| config-from | Y | N | N/A | |
| 48 | +| context | Y | N | N/A | |
| 49 | + |
| 50 | +(\*) In addition to the above rules, `tasks` and `config-from` are mutually exclusive: you must specify one or the other in a project repository `.taskcluster.yml`, but not both. |
| 51 | + |
| 52 | +## Building context for rendering |
| 53 | + |
| 54 | +With the introduction of additional sources for context, it's worth making explicit the order of precedence they have, which is: |
| 55 | +1) GitHub & Taskcluster-GitHub supplied context (`event`, `tasks_for`, etc.) (there should be no overlaps in the keys these provide - so they are treated as equal). |
| 56 | +2) Project supplied context |
| 57 | + |
| 58 | +Merging will be done shallowly - meaning that keys like `event` and `tasks_for` will be fully ignored if supplied by project context. |
| 59 | + |
| 60 | +The reasoning behind this is that GitHub and Taskcluster-GitHub supplied context are largely "facts": things like revision, repository, the event that triggered Taskcluster-GitHub, etc. Allowing overriding could both be confusing (eg: a project hardcoding `tasks_for` to `github-push` would cause pull requests to run unexpected things), and a possible security vulnerability (eg: overriding `event.pusher.email` could allow impersonation leading to privilege escalation). |
| 61 | + |
| 62 | +## Rendering process |
| 63 | + |
| 64 | +When Taskcluster-GitHub encounters a `.taskcluster.yml` such as the one above when processing an event for a repository, instead of directly rendering it with the context from the GitHub event, it will instead: |
| 65 | +1) Fetch the `.taskcluster.yml` referenced in `config-from` |
| 66 | +2) Build context per the above |
| 67 | +3) Render the fetched `.taskcluster.yml` with the combined context |
| 68 | +4) Proceed as usual, creating any Tasks resulting from the rendered `.taskcluster.yml` |
| 69 | + |
| 70 | +Note: Only a `.taskcluster.yml` defined directly in a project repository will have its `config-from` processed. There is no need to support recursion into multiple levels of references, and it is more likely to result in confusion than add any value. |
| 71 | + |
| 72 | +# Implementation |
| 73 | + |
| 74 | +<Once the RFC is decided, these links will provide readers a way to track the |
| 75 | +implementation through to completion, and to know if they are running a new |
| 76 | +enough version to take advantage of this change. It's fine to update this |
| 77 | +section using short PRs or pushing directly to master after the RFC is |
| 78 | +decided> |
| 79 | + |
| 80 | +* <link to tracker bug, issue, etc.> |
| 81 | +* <...> |
| 82 | +* Implemented in Taskcluster version ... |
0 commit comments