Skip to content

Commit b519edd

Browse files
thomasqueirozbburaizugithub-actions[bot]
authored
docs(vrl): Automatically generate VRL function documentation (#24719)
* Add generation of cue files from vrl Function * Use __mock_return_values_for_tests to show mocked examples * Handle kind::any * Replace generate-component-docs with generate-docs * Bump VRL * chore(vrl): Bump vrl and add return_kind to functions * Enable all features for vector-vrl-functions * Add stubs and implement new params * Bump vrl and unstub pure * Bump vrl and unstub notices * Trim strings * chore(vrl): Bump VRL and implement category for functions * Use .category() * chore(vrl): bump VRL and use Parameter builder * Add .enum_variants to aggregate_vector_metrics * Add missing import * Add support for enum_variants * Add Raises block to remap rendering * Bump vrl * Fix clippy * Install protoc when compiling vdev * Add generated files * Format json using 4 spaces instead of 2 * Ignore VRL generated docs from cue fmt * Revert "Format json using 4 spaces instead of 2" This reverts commit ad2a14b. * Push newline at the end of generated json * Bump VRL * Add preserve_order to keep doc generation consistent * Render input * Regenerate docs * Use IndexMap to preserve enum ordering * Regenerate docs with enum ordering * Fix cue schema * Remove .log when rendering VRL examples' input * Fix clippy * Format cue * Add base64 encoded letters to spellcheck ignore * Add all unrecognized words to expect.txt * Update input to json value * Use branch VRL and use VRL generation code * Bump branch sha * Add generate-vrl-docs to check-docs * Add amplify.yml * Use __doc_paths * Bump version * Update licenses * Use . appRoot * Bump VRL * Generate docs to docs/generated * Generate docs * Rename vrl-docs to vector-vrl-docs * Generate Vector VRL and VRL docs using make generate-vrl-docs * Revert to prod amplify.yml * Add VECTOR_SHA override * Download latest vdev in amplify.yml * Install vdev from source * All in one compilation/copy * Add check_type_only: false * Compile debug instead of release * Format * Fix check-docs check * Revert changes to expect.txt * Update changes.yml * Add sticky comment workflow * Update docs/generated/README.md Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com> * Regenerate docs * Make check-generated-vrl-docs always run * Use needs.*.result and check for failure/cancelled * Revert "Regenerate docs" This reverts commit 896918e. * Use --vector-sha inside CI to ensure docs freshness * Revert "Use --vector-sha inside CI to ensure docs freshness" This reverts commit cc1d474. * Add --vector-sha to generate-vector-vrl-docs step * Revert "Add --vector-sha to generate-vector-vrl-docs step" This reverts commit 88c939e. * Use --vector-sha when deploying the website * Fix non CI make generate-vrl-docs * Move sparse_checkout_docs to vdev/src/utils/git.rs * Recompile vdev when there are changes to 'lib/vector-vrl/**' * chore(dev): move VRL-specific crates under lib/vector-vrl/ (#24854) * Update generated VRL docs * Format * Remove `_*_explainer`s * Fix generated docs filters * Add test doc change * Update generated VRL docs * Revert "Update generated VRL docs" This reverts commit 6e359f7. * Reapply "Update generated VRL docs" This reverts commit f6517da. * Delete amplify.yml * Revert "Reapply "Update generated VRL docs"" This reverts commit a445e03. * Update generated VRL docs * remove Hello from source code once and for all * Optimize setup action * Update generated VRL docs * Fix potential head_ref injection * Update generated VRL docs * GHA trigger * Fail check when out of date * Update generated VRL docs * ci-test: add foo-bar to find_enrichment_table_records example * ci-test: make docs incorrect * Update generated VRL docs * Update vdev cache keys * Update generated VRL docs * Revert mock changes * Update generated VRL docs * GHA trigger * Update warning comment --------- Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
1 parent 2debb99 commit b519edd

File tree

240 files changed

+1263
-11380
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

240 files changed

+1263
-11380
lines changed

‎.github/actions/install-vdev/action.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,8 @@ runs:
2020
uses: actions/cache@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
2121
with:
2222
path: ~/.cargo/bin/vdev
23-
key: ${{ runner.os }}-vdev-${{ hashFiles('vdev/**', 'Cargo.toml', 'Cargo.lock') }}
23+
# WARNING: this key need to be in sync with the key in .github/actions/setup/action.yml
24+
key: ${{ runner.os }}-vdev-${{ hashFiles('vdev/**', 'lib/vector-vrl/**', 'Cargo.toml', 'Cargo.lock') }}
2425
restore-keys: |
2526
${{ runner.os }}-vdev-
2627
@@ -36,7 +37,8 @@ runs:
3637
uses: actions/cache/save@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
3738
with:
3839
path: ~/.cargo/bin/vdev
39-
key: ${{ runner.os }}-vdev-${{ hashFiles('vdev/**', 'Cargo.toml', 'Cargo.lock') }}
40+
# WARNING: this key need to be in sync with the key in .github/actions/setup/action.yml
41+
key: ${{ runner.os }}-vdev-${{ hashFiles('vdev/**', 'lib/vector-vrl/**', 'Cargo.toml', 'Cargo.lock') }}
4042

4143
- name: Set VDEV environment variable
4244
shell: bash

‎.github/actions/setup/action.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,8 @@ runs:
9090
uses: actions/cache@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
9191
with:
9292
path: ~/.cargo/bin/vdev
93-
key: ${{ runner.os }}-vdev-${{ hashFiles('vdev/**', 'Cargo.toml', 'Cargo.lock') }}
93+
# WARNING: this key need to be in sync with the key in .github/actions/install-vdev/action.yml
94+
key: ${{ runner.os }}-vdev-${{ hashFiles('vdev/**', 'lib/vector-vrl/**', 'Cargo.toml', 'Cargo.lock') }}
9495
restore-keys: |
9596
${{ runner.os }}-vdev-
9697
lookup-only: true
@@ -211,7 +212,7 @@ runs:
211212
EOF
212213
213214
- name: Install protoc
214-
if: ${{ inputs.protoc == 'true' }}
215+
if: ${{ inputs.protoc == 'true' || env.VDEV_NEEDS_COMPILE == 'true' }}
215216
shell: bash
216217
run: |
217218
echo "Installing protoc"

‎.github/workflows/changes.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -227,7 +227,11 @@ jobs:
227227
component_docs:
228228
- 'scripts/generate-component-docs.rb'
229229
- "vdev/**"
230-
- 'website/cue/**/base/**.cue'
230+
- 'website/cue/**/*.cue'
231+
- 'docs/generated/**'
232+
# If changes to the VRL sha is made the combined generated cue file will change which
233+
# may cause issues
234+
- 'Cargo.lock'
231235
- ".github/workflows/changes.yml"
232236
markdown:
233237
- '**/**.md'

‎.github/workflows/check_generated_vrl_docs.yml‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
name: Check Generated VRL Docs Freshness
2+
3+
on:
4+
pull_request:
5+
merge_group:
6+
types: [checks_requested]
7+
push:
8+
branches:
9+
- master
10+
11+
concurrency:
12+
group: ${{ github.workflow }}-${{ github.event.number || github.sha }}
13+
cancel-in-progress: true
14+
15+
env:
16+
COMMIT_MESSAGE: "Update generated VRL docs"
17+
COMMIT_AUTHOR: "github-actions[bot]"
18+
19+
permissions:
20+
contents: read
21+
22+
jobs:
23+
changes:
24+
runs-on: ubuntu-latest
25+
permissions:
26+
contents: read
27+
outputs:
28+
docs: ${{ steps.filter.outputs.docs }}
29+
steps:
30+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
31+
- uses: dorny/paths-filter@v3
32+
id: filter
33+
with:
34+
filters: |
35+
docs:
36+
- "lib/vector-vrl/**"
37+
- "vdev/**"
38+
- "Cargo.lock"
39+
- ".github/workflows/check_generated_vrl_docs.yml"
40+
41+
run-check-generated-vrl-docs:
42+
needs: changes
43+
if: needs.changes.outputs.docs == 'true'
44+
runs-on: ubuntu-24.04-8core
45+
permissions:
46+
contents: write
47+
steps:
48+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
49+
with:
50+
ref: ${{ github.event.pull_request.head.sha || github.sha }}
51+
52+
- uses: ./.github/actions/setup
53+
with:
54+
vdev: true
55+
mold: false
56+
cargo-cache: false
57+
58+
- name: Regenerate VRL docs
59+
run: make generate-vector-vrl-docs
60+
61+
- name: Check for changes
62+
id: check
63+
run: |
64+
git add docs/generated/
65+
if git diff --cached --quiet; then
66+
echo "changed=false" >> "$GITHUB_OUTPUT"
67+
else
68+
echo "changed=true" >> "$GITHUB_OUTPUT"
69+
fi
70+
71+
- name: Check last commit
72+
if: steps.check.outputs.changed == 'true'
73+
id: last-commit
74+
run: |
75+
MSG=$(git log -1 --pretty=%s)
76+
AUTHOR=$(git log -1 --pretty=%an)
77+
if [ "$MSG" = "$COMMIT_MESSAGE" ] && [ "$AUTHOR" = "$COMMIT_AUTHOR" ]; then
78+
echo "is-auto=true" >> "$GITHUB_OUTPUT"
79+
else
80+
echo "is-auto=false" >> "$GITHUB_OUTPUT"
81+
fi
82+
83+
- name: Commit and push
84+
if: >
85+
steps.check.outputs.changed == 'true'
86+
&& steps.last-commit.outputs.is-auto != 'true'
87+
&& github.event_name == 'pull_request'
88+
&& github.event.pull_request.head.repo.full_name == github.repository
89+
id: push
90+
continue-on-error: true
91+
env:
92+
HEAD_REF: ${{ github.head_ref }}
93+
run: |
94+
git config user.name "github-actions[bot]"
95+
git config user.email "github-actions[bot]@users.noreply.github.com"
96+
git commit -m "$COMMIT_MESSAGE"
97+
git push origin HEAD:refs/heads/$HEAD_REF
98+
99+
- name: Save PR number for comment workflow
100+
if: >
101+
steps.check.outputs.changed == 'true'
102+
&& steps.last-commit.outputs.is-auto != 'true'
103+
&& github.event_name == 'pull_request'
104+
&& steps.push.outcome != 'success'
105+
run: |
106+
mkdir -p /tmp/docs-check
107+
echo "${{ github.event.pull_request.number }}" > /tmp/docs-check/pr-number
108+
109+
- name: Upload PR metadata
110+
if: >
111+
steps.check.outputs.changed == 'true'
112+
&& steps.last-commit.outputs.is-auto != 'true'
113+
&& github.event_name == 'pull_request'
114+
&& steps.push.outcome != 'success'
115+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
116+
with:
117+
name: vrl-docs-check-pr
118+
path: /tmp/docs-check/pr-number
119+
120+
- name: Fail if docs are out of date
121+
if: steps.check.outputs.changed == 'true'
122+
run: |
123+
echo "docs/generated/ is out of date. Regenerate with: make generate-vector-vrl-docs"
124+
exit 1
125+
126+
check-generated-vrl-docs:
127+
if: always()
128+
runs-on: ubuntu-latest
129+
needs: run-check-generated-vrl-docs
130+
steps:
131+
- run: |
132+
if [[ "${{ contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') }}" == "true" ]]; then
133+
echo "One or more jobs failed or were cancelled"
134+
exit 1
135+
fi

‎.github/workflows/check_generated_vrl_docs_comment.yml‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
name: Comment on PR (Generated VRL Docs)
2+
3+
on:
4+
workflow_run:
5+
workflows: ["Check Generated VRL Docs Freshness"]
6+
types:
7+
- completed
8+
9+
permissions:
10+
contents: read
11+
12+
jobs:
13+
comment:
14+
runs-on: ubuntu-latest
15+
if: >
16+
github.event.workflow_run.conclusion == 'failure'
17+
&& github.event.workflow_run.event == 'pull_request'
18+
permissions:
19+
pull-requests: write
20+
steps:
21+
- name: Download PR metadata
22+
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
23+
with:
24+
name: vrl-docs-check-pr
25+
run-id: ${{ github.event.workflow_run.id }}
26+
github-token: ${{ github.token }}
27+
28+
- name: Comment on PR
29+
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
30+
with:
31+
script: |
32+
const fs = require('fs');
33+
const prNumber = parseInt(fs.readFileSync('pr-number', 'utf8').trim(), 10);
34+
if (isNaN(prNumber)) {
35+
core.setFailed('Invalid PR number');
36+
return;
37+
}
38+
39+
const marker = '<!-- generated-vrl-docs-check -->';
40+
const body = marker + '\nThe `docs/generated/` folder is out of date but I was unable to push the fix automatically.\n\nPlease run the following and commit the result:\n\n```\nmake generate-vector-vrl-docs\n```';
41+
42+
const { data: comments } = await github.rest.issues.listComments({
43+
owner: context.repo.owner,
44+
repo: context.repo.repo,
45+
issue_number: prNumber,
46+
});
47+
const existing = comments.find(c => c.body.startsWith(marker));
48+
if (existing) {
49+
await github.rest.issues.updateComment({
50+
owner: context.repo.owner,
51+
repo: context.repo.repo,
52+
comment_id: existing.id,
53+
body,
54+
});
55+
} else {
56+
await github.rest.issues.createComment({
57+
owner: context.repo.owner,
58+
repo: context.repo.repo,
59+
issue_number: prNumber,
60+
body,
61+
});
62+
}

‎.github/workflows/test.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -150,8 +150,8 @@ jobs:
150150
markdownlint: true
151151
- run: make check-markdown
152152

153-
check-component-docs:
154-
name: Check Component Docs
153+
check-generated-docs:
154+
name: Check Generated Docs
155155
runs-on: ubuntu-24.04-8core
156156
if: ${{ needs.changes.outputs.source == 'true' || needs.changes.outputs.component_docs == 'true' || needs.changes.outputs.test-yml == 'true' }}
157157
needs: changes
@@ -163,7 +163,7 @@ jobs:
163163
protoc: true
164164
cue: true
165165
libsasl2: true
166-
- run: make check-component-docs
166+
- run: make check-generated-docs
167167

168168
check-rust-docs:
169169
name: Check Rust Docs
@@ -221,7 +221,7 @@ jobs:
221221
- check-licenses
222222
- check-docs
223223
- check-markdown
224-
- check-component-docs
224+
- check-generated-docs
225225
- check-rust-docs
226226
- test-vrl
227227
- build-vrl-playground

‎CONTRIBUTING.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,7 @@ echo "Running pre-push checks..."
141141
make check-licenses
142142
make check-fmt
143143
make check-clippy
144-
make check-component-docs
144+
make check-generated-docs
145145

146146
# Some other checks that in our experience rarely fail on PRs.
147147
make check-deny
@@ -292,7 +292,7 @@ cargo vdev check events
292292
cargo vdev check licenses
293293
# Vector's documentation for each component is generated from the comments attached to the Component structs and members.
294294
# Running this ensures that the generated docs are up to date.
295-
make check-component-docs
295+
make check-generated-docs
296296
# Generate the code documentation for the Vector project.
297297
# Run this to ensure the docs can be generated without errors (warnings are acceptable at the minute).
298298
cd rust-doc && make docs

‎Cargo.lock‎

Lines changed: 14 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎LICENSE-3rdparty.csv‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -635,6 +635,7 @@ regex-automata,https://github.com/rust-lang/regex/tree/master/regex-automata,MIT
635635
regex-filtered,https://github.com/ua-parser/uap-rust,BSD-3-Clause,The regex-filtered Authors
636636
regex-lite,https://github.com/rust-lang/regex,MIT OR Apache-2.0,"The Rust Project Developers, Andrew Gallant <jamslam@gmail.com>"
637637
regex-syntax,https://github.com/rust-lang/regex/tree/master/regex-syntax,MIT OR Apache-2.0,"The Rust Project Developers, Andrew Gallant <jamslam@gmail.com>"
638+
relative-path,https://github.com/udoprog/relative-path,MIT OR Apache-2.0,John-John Tedro <udoprog@tedro.se>
638639
rend,https://github.com/djkoloski/rend,MIT,David Koloski <djkoloski@gmail.com>
639640
reqwest,https://github.com/seanmonstar/reqwest,MIT OR Apache-2.0,Sean McArthur <sean@seanmonstar.com>
640641
reqwest-middleware,https://github.com/TrueLayer/reqwest-middleware,MIT OR Apache-2.0,Rodrigo Gryzinski <rodrigo.gryzinski@truelayer.com>

0 commit comments

Comments
 (0)