Skip to content

Run config_docs CI check on PRs to change auto generated docs #17046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

Run config_docs CI check on PRs to change auto generated docs #17046

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6011676
Run generated docs checks on changes to md files too
alamb Aug 5, 2025
a2e40e9
fix
alamb Aug 5, 2025
dd21cf3
fix
alamb Aug 5, 2025
1f87c33
Update .github/workflows/docs_generated.yaml
alamb Aug 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions .github/workflows/docs_generated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

name: Generated Docs

concurrency:
group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
cancel-in-progress: true

on:
push:
paths:
- ".github/**"
- "datafusion/**"
- "dev/update_config_docs.sh"
- "dev/update_function_docs.sh"
- "docs/source/user-guide/configs.md"
- "docs/source/user-guide/sql/aggregate_functions.md"
- "docs/source/user-guide/sql/scalar_functions.md"
- "docs/source/user-guide/sql/window_functions.md"
pull_request:
paths:
- ".github/**"
- "datafusion/**"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cargo.toml, cargo.lock, rust-toolchain.toml, ?

- "dev/update_config_docs.sh"
- "dev/update_function_docs.sh"
- "docs/source/user-guide/configs.md"
- "docs/source/user-guide/sql/aggregate_functions.md"
- "docs/source/user-guide/sql/scalar_functions.md"
- "docs/source/user-guide/sql/window_functions.md"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Easy to miss adding a new document here.
Can we run this workflow on any changes to the user guide?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these are the only files that are auto generated (and thus the only .md files that can be affected by the change in the rust code). I was trying to preserve the property that most changes to the docs did not require all the rust tests to be run

# manual trigger
# https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow
workflow_dispatch:

jobs:
generated-docs-check:
name: check configs.md and ***_functions.md is up-to-date
runs-on: ubuntu-latest
container:
image: amd64/rust
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think actions/checkout is controlled by GitHub. I believe it's safe to use tag reference on this one and the setup-node as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copyied from the existing jobs

They were pinned to shas in this PR by @gopidesupavan i think as an added level of protection

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@findepi yes agree thought its managed from github tags will not protect us from any attacks if something wrong or anyone manipulates the pinned tag, commit sha is preffered way :) and its recommended way from ASF policy aswell https://infra.apache.org/github-actions-policy.html

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i am well aware of 3rd party github actions being an "awesome" (easy) attack vector when they are not pinned to a sha. However, i think GH recommends this for 3rd party actions, not github-owned actions

from ASF page linked above

You MAY use all actions internal to the apache/, github/ and actions/* namespaces without restrictions.

You MUST pin all external actions to the specific git hash (SHA1) of

I interpret it as saying "actions./*" is "internal" and you don't need to pin to a sha.

While we MAY pin also internal actions, it can be viewed as unnecessary hassle when updating to a newer version.

with:
submodules: true
fetch-depth: 1
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
rust-version: stable
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: "20"
- name: Check if configs.md has been modified
run: |
# If you encounter an error, run './dev/update_config_docs.sh' and commit
./dev/update_config_docs.sh
git diff --exit-code
- name: Check if any of the ***_functions.md has been modified
run: |
# If you encounter an error, run './dev/update_function_docs.sh' and commit
./dev/update_function_docs.sh
git diff --exit-code
28 changes: 0 additions & 28 deletions .github/workflows/rust.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -702,34 +702,6 @@ jobs:
- name: Check Cargo.toml formatting
run: taplo format --check

config-docs-check:
name: check configs.md and ***_functions.md is up-to-date
needs: linux-build-lib
runs-on: ubuntu-latest
container:
image: amd64/rust
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
submodules: true
fetch-depth: 1
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
rust-version: stable
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: "20"
- name: Check if configs.md has been modified
run: |
# If you encounter an error, run './dev/update_config_docs.sh' and commit
./dev/update_config_docs.sh
git diff --exit-code
- name: Check if any of the ***_functions.md has been modified
run: |
# If you encounter an error, run './dev/update_function_docs.sh' and commit
./dev/update_function_docs.sh
git diff --exit-code

# Verify MSRV for the crates which are directly used by other projects:
# - datafusion
Expand Down