|
| 1 | +# Language Onboarding Guide |
| 2 | + |
| 3 | +This document provides a comprehensive guide for language teams to onboard their projects to the Librarian platform. It |
| 4 | +details the necessary steps and configurations required to develop a language-specific container that Librarian can |
| 5 | +delegate tasks to. |
| 6 | + |
| 7 | +## Core Concepts |
| 8 | + |
| 9 | +Before diving into the specifics, it's important to understand the key components of the Librarian ecosystem: |
| 10 | + |
| 11 | +* **Librarian:** The core orchestration tool that automates the generation, release, and maintenance of client |
| 12 | + libraries. |
| 13 | +* **Language-Specific Container:** A Docker container, built by each language team, that encapsulates the logic for |
| 14 | + generating, building, and releasing libraries for that specific language. Librarian interacts with this container by |
| 15 | + invoking different commands. |
| 16 | +* **`state.yaml`:** A manifest file within each language repository that defines the libraries managed by Librarian, |
| 17 | + their versions, and other essential metadata. |
| 18 | +* **`config.yaml`:** A configuration file that allows for repository-level customization of Librarian's behavior, such |
| 19 | + as specifying which files the container can access. |
| 20 | + |
| 21 | +## Configuration Files |
| 22 | + |
| 23 | +Librarian relies on two key configuration files to manage its operations: `state.yaml` and `config.yaml`. These files |
| 24 | +must be present in the `.librarian` directory at the root of the language repository. |
| 25 | + |
| 26 | +### `state.yaml` |
| 27 | + |
| 28 | +The `state.yaml` file is the primary manifest that informs Librarian about the libraries it is responsible for managing. |
| 29 | +It contains a comprehensive list of all libraries within the repository, along with their current state and |
| 30 | +configuration. |
| 31 | + |
| 32 | +For a detailed breakdown of all of the fields in the `state.yaml` file, please refer to [state-schema.md]. |
| 33 | + |
| 34 | +### `config.yaml` |
| 35 | + |
| 36 | +The `config.yaml` file is a handwritten configuration file that allows you to customize Librarian's behavior at the |
| 37 | +repository level. Its primary use is to define which files the language-specific container is allowed to access. |
| 38 | + |
| 39 | +Here is an example of a `config.yaml` file: |
| 40 | + |
| 41 | +```yaml |
| 42 | +# .librarian/config.yaml |
| 43 | + |
| 44 | +# A list of files that will be provided to the 'configure' and 'release-init' |
| 45 | +# container invocations. |
| 46 | +global_files_allowlist: |
| 47 | + # Allow the container to read and write the root go.work file during the |
| 48 | + # 'configure' step to add new modules. |
| 49 | + - path: "go.work" |
| 50 | + permissions: "read-write" |
| 51 | + |
| 52 | + # Allow the container to read a template. |
| 53 | + - path: "internal/README.md.template" |
| 54 | + permissions: "read-only" |
| 55 | + |
| 56 | + # Allow publishing the updated root README.md. |
| 57 | + - path: "README.md" |
| 58 | + permissions: "write-only" |
| 59 | +``` |
| 60 | +
|
| 61 | +## Container Contracts |
| 62 | +
|
| 63 | +Librarian orchestrates its workflows by making a series of invocations to a language-specific container. Each invocation |
| 64 | +corresponds to a specific command and is designed to perform a distinct task. For the container to function correctly, |
| 65 | +it must have a binary entrypoint that can accept the arguments passed by Librarian. |
| 66 | +
|
| 67 | +A successful container invocation is expected to exit with a code of `0`. Any non-zero exit code will be treated as an |
| 68 | +error and will halt the current workflow. If a container would like to send an error message back to librarian it can do |
| 69 | +so by including a field in the various response files outlined below. |
| 70 | + |
| 71 | +The following sections detail the contracts for each container command. |
| 72 | + |
| 73 | +### `configure` |
| 74 | + |
| 75 | +The `configure` command is invoked only during the onboarding of a new API. Its primary responsibility is to process |
| 76 | +the new API information and generate the necessary configuration for the library. |
| 77 | + |
| 78 | +The container is expected to produce up to two artifacts: |
| 79 | + |
| 80 | +* A `configure-response.json` file, which is derived from the `configure-request.json` and contains language-specific |
| 81 | + details. This response will be committed back to the `state.yaml` file by Librarian. |
| 82 | +* Any "side-configuration" files that the language may need for its libraries. These should be written to the `/input` mount, which corresponds to the `.librarian/generator-input` directory in the language repository. |
| 83 | + |
| 84 | +TODO: Global file edits |
| 85 | + |
| 86 | +**Contract:** |
| 87 | + |
| 88 | +| Context | Type | Description | |
| 89 | +| :----------- | :------------------ | :----------------------------------------------------------------------------- | |
| 90 | +| `/librarian` | Mount (Read/Write) | Contains `configure-request.json`. The container must process this and write back `configure-response.json`. | |
| 91 | +| `/input` | Mount (Read/Write) | The contents of the `.librarian/generator-input` directory. The container can add new language-specific configuration here. | |
| 92 | +| `/repo` | Mount (Read) | Contains only the files specified in the `global_files_allowlist` from `config.yaml`. | |
| 93 | +| `/source` | Mount (Read). | Contains the complete contents of the API definition repository (e.g., [googleapis/googleapis](https://github.com/googleapis/googleapis)). | |
| 94 | +| `/output` | Mount (Read/Write) | An output directory for writing any global file edits allowed by `global_files_allowlist`. | |
| 95 | +| `command` | Positional Argument | The value will always be `configure`. | |
| 96 | +| flags | Flags | Flags indicating the locations of the mounts: `--librarian`, `--input`, `--source`, `--repo`, `--output` | |
| 97 | + |
| 98 | +**Example `configure-request.json`:** |
| 99 | + |
| 100 | +*Note: There will be only one API with a `status` of `new`.* |
| 101 | + |
| 102 | +```json |
| 103 | +{ |
| 104 | + "libraries": [ |
| 105 | + { |
| 106 | + "id": "google-cloud-secretmanager", |
| 107 | + "apis": [ |
| 108 | + { |
| 109 | + "path": "google/cloud/secretmanager/v1", |
| 110 | + "service_config": "secretmanager_v1.yaml", |
| 111 | + "status": "new" |
| 112 | + } |
| 113 | + ], |
| 114 | + }, |
| 115 | + { |
| 116 | + "id": "google-cloud-pubsub-v1", |
| 117 | + "apis": [ |
| 118 | + { |
| 119 | + "path": "google/cloud/pubsub/v1", |
| 120 | + "service_config": "pubsub_v1.yaml", |
| 121 | + "status": "existing" |
| 122 | + } |
| 123 | + ], |
| 124 | + "source_roots": [ "pubsub" ] |
| 125 | + } |
| 126 | + ] |
| 127 | +} |
| 128 | +``` |
| 129 | + |
| 130 | +**Example `configure-response.json`:** |
| 131 | + |
| 132 | +*Note: Only the library with a `status` of `new` should be returned.* |
| 133 | + |
| 134 | +```json |
| 135 | +{ |
| 136 | + "id": "google-cloud-secretmanager", |
| 137 | + "apis": [ |
| 138 | + { |
| 139 | + "path": "google/cloud/secretmanager/v1", |
| 140 | + } |
| 141 | + ], |
| 142 | + "source_roots": [ "secretmanager" ], |
| 143 | + "preserve_regex": [ |
| 144 | + "secretmanager/subdir/handwritten-file.go" |
| 145 | + ], |
| 146 | + "remove_regex": [ |
| 147 | + "secretmanager/generated-dir" |
| 148 | + ], |
| 149 | + "version": "0.0.0", |
| 150 | + "tag_format": "{id}/v{version}", |
| 151 | + "error": "An optional field to share error context back to Librarian." |
| 152 | +} |
| 153 | +``` |
| 154 | + |
| 155 | +### `generate` |
| 156 | + |
| 157 | +The `generate` command is where the core work of code generation happens. The container is expected to generate the library code and write it to the `/output` mount, preserving the directory structure of the language repository. |
| 158 | + |
| 159 | +**Contract:** |
| 160 | + |
| 161 | +| Context | Type | Description | |
| 162 | +| :----------- | :------------------ | :------------------------------------------------------------------------------ | |
| 163 | +| `/librarian` | Mount (Read/Write) | Contains `generate-request.json`. Container can optionally write back a `generate-response.json`. | |
| 164 | +| `/input` | Mount (Read/Write) | The contents of the `.librarian/generator-input` directory. | |
| 165 | +| `/output` | Mount (Write) | The destination for the generated code. The output structure should match the target repository. | |
| 166 | +| `/source` | Mount (Read) | The complete contents of the API definition repository. (e.g. googlapis/googleapis) | |
| 167 | +| `command` | Positional Argument | The value will always be `generate`. | |
| 168 | +| flags | Flags | Flags indicating the locations of the mounts: `--librarian`, `--input`, `--output`, `--source` | |
| 169 | + |
| 170 | +**Example `generate-request.json`:** |
| 171 | + |
| 172 | +```json |
| 173 | +{ |
| 174 | + "id": "google-cloud-secretmanager", |
| 175 | + "apis": [ |
| 176 | + { |
| 177 | + "path": "google/cloud/secretmanager/v1", |
| 178 | + "service_config": "secretmanager_v1.yaml" |
| 179 | + } |
| 180 | + ], |
| 181 | + "source_paths": [ |
| 182 | + "secretmanager" |
| 183 | + ], |
| 184 | + "preserve_regex": [ |
| 185 | + "secretmanager/subdir/handwritten-file.go" |
| 186 | + ], |
| 187 | + "remove_regex": [ |
| 188 | + "secretmanager/generated-dir" |
| 189 | + ], |
| 190 | + "version": "0.0.0", |
| 191 | + "tag_format": "{id}/v{version}" |
| 192 | +} |
| 193 | +``` |
| 194 | + |
| 195 | +**Example `generate-response.json`:** |
| 196 | + |
| 197 | +```json |
| 198 | +{ |
| 199 | + "error": "An optional field to share error context back to Librarian." |
| 200 | +} |
| 201 | +``` |
| 202 | + |
| 203 | +After the `generate` container finishes, Librarian is responsible for copying the generated code to the language |
| 204 | +repository and handling any merging or deleting actions as defined in the library's state. |
| 205 | + |
| 206 | +### `build` |
| 207 | + |
| 208 | +The `build` command is responsible for building and testing the newly generated library to ensure its integrity. |
| 209 | + |
| 210 | +**Contract:** |
| 211 | + |
| 212 | +| Context | Type | Description | |
| 213 | +| :----------- | :------------------ | :------------------------------------------------------------------------------ | |
| 214 | +| `/librarian` | Mount (Read/Write) | Contains `build-request.json`. Container can optionally write back a `build-response.json`. | |
| 215 | +| `/repo` | Mount (Read/Write) | The entire language repository. This is a deep copy, so any changes made here will not affect the final generated code. | |
| 216 | +| `command` | Positional Argument | The value will always be `build`. | |
| 217 | +| flags. | Flags | Flags indicating the locations of the mounts: `--librarian`, `--repo` | |
| 218 | + |
| 219 | +**Example `build-request.json`:** |
| 220 | + |
| 221 | +```json |
| 222 | +{ |
| 223 | + "id": "google-cloud-secretmanager", |
| 224 | + "apis": [ |
| 225 | + { |
| 226 | + "path": "google/cloud/secretmanager/v1", |
| 227 | + "service_config": "secretmanager_v1.yaml" |
| 228 | + } |
| 229 | + ], |
| 230 | + "source_paths": [ |
| 231 | + "secretmanager" |
| 232 | + ], |
| 233 | + "preserve_regex": [ |
| 234 | + "secretmanager/subdir/handwritten-file.go" |
| 235 | + ], |
| 236 | + "remove_regex": [ |
| 237 | + "secretmanager/generated-dir" |
| 238 | + ], |
| 239 | + "version": "0.0.0", |
| 240 | + "tag_format": "{id}/v{version}" |
| 241 | +} |
| 242 | +``` |
| 243 | + |
| 244 | +**Example `build-response.json`:** |
| 245 | + |
| 246 | +```json |
| 247 | +{ |
| 248 | + "error": "An optional field to share error context back to Librarian." |
| 249 | +} |
| 250 | +``` |
| 251 | + |
| 252 | +### `release-init` |
| 253 | + |
| 254 | +The `release-init` command is the core of the release workflow. After Librarian determines the new version and collates |
| 255 | +the commits for a release, it invokes this container command to apply the necessary changes to the repository. |
| 256 | + |
| 257 | +The container command's primary responsibility is to update all required files with the new version and commit |
| 258 | +information. This includes, but is not limited to, updating `CHANGELOG.md` files, bumping version numbers in metadata |
| 259 | +files (e.g., `pom.xml`, `package.json`), and updating any global files that reference the libraries being released. |
| 260 | + |
| 261 | +**Contract:** |
| 262 | + |
| 263 | +| Context | Type | Description | |
| 264 | +| :----------- | :------------------ | :------------------------------------------------------------------------------ | |
| 265 | +| `/librarian` | Mount (Read/Write) | Contains `release-init-request.json`. Container writes back a `release-init-response.json`. | |
| 266 | +| `/repo` | Mount (Read/Write) | The entire language repository, allowing the container to make any necessary global edits. | |
| 267 | +| `/output` | Mount (Write) | Any files updated during the release phase should be moved to this directory, preserving their original paths. | |
| 268 | +| `command` | Positional Argument | The value will always be `release-init`. | |
| 269 | +| flags. | Flags | Flags indicating the locations of the mounts: `--librarian`, `--repo`, `--output` | |
| 270 | + |
| 271 | +**Example `release-init-request.json`:** |
| 272 | + |
| 273 | +```json |
| 274 | +{ |
| 275 | + "libraries": [ |
| 276 | + { |
| 277 | + "id": "google-cloud-secretmanager-v1", |
| 278 | + "version": "1.3.0", |
| 279 | + "changes": [ |
| 280 | + { |
| 281 | + "type": "feat", |
| 282 | + "subject": "add new UpdateRepository API", |
| 283 | + "body": "This adds the ability to update a repository's properties.", |
| 284 | + "piper_cl_number": "786353207", |
| 285 | + "source_commit_hash": "9461532e7d19c8d71709ec3b502e5d81340fb661" |
| 286 | + }, |
| 287 | + { |
| 288 | + "type": "docs", |
| 289 | + "subject": "fix typo in BranchRule comment", |
| 290 | + "body": "", |
| 291 | + "piper_cl_number": "786353207", |
| 292 | + "source_commit_hash": "9461532e7d19c8d71709ec3b502e5d81340fb661" |
| 293 | + } |
| 294 | + ], |
| 295 | + "apis": [ |
| 296 | + { |
| 297 | + "path": "google/cloud/secretmanager/v1" |
| 298 | + }, |
| 299 | + { |
| 300 | + "path": "google/cloud/secretmanager/v1beta" |
| 301 | + } |
| 302 | + ], |
| 303 | + "source_roots": [ |
| 304 | + "secretmanager", |
| 305 | + "other/location/secretmanager" |
| 306 | + ] |
| 307 | + } |
| 308 | + ] |
| 309 | +} |
| 310 | +``` |
| 311 | + |
| 312 | +**Example `release-init-response.json`:** |
| 313 | + |
| 314 | +```json |
| 315 | +{ |
| 316 | + "error": "An optional field to share error context back to Librarian." |
| 317 | +} |
| 318 | +``` |
| 319 | + |
| 320 | +[state-schema.md]: state-schema.md |
0 commit comments