Skip to content

Commit 6bf5e98

Browse files
2010YOUY012010YOUY01
authored
dev: Add a script to auto fix all lint violations (#19560)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> Part of #19227 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> See issue for details. The existing script `./dev/rust_lint.sh` do checks for all non-functional tests include formater/clippy checks. Some check tools support auto fix options, so this PR add an option to the lint scripts to perform auto-fixes. Now `./dev/rust_lint.sh --write --allow-dirty` can perform auto-fixes for all linter etc. violations ``` yongting@Yongtings-MacBook-Pro-2 ~/C/datafusion (auto-fix)> ./dev/rust_lint.sh --help Usage: ./dev/rust_lint.sh [--write] [--allow-dirty] Runs the local Rust lint suite similar to CI. --write Run formatters, clippy and other non-functional checks in best-effort write/fix mode (requires a clean git worktree, no uncommitted changes; some checks are test-only and ignore this flag). --allow-dirty Allow `--write` to run even when the git worktree has uncommitted changes. ``` ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Adds `[--write] [--allow-dirty]` flag to `rust_lint.sh` to support auto fixes - `rust_lint.sh` consists of several sub-scripts like `rust_fmt.sh`, they're all extended with auto-fix feature through `--write` flag, and the `rust_lint.sh` is optionally calling them with an additional flag for auto fixes. - Clean up `rust_lint.sh` ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes, commit 8c99417 intentionally introduced one violation for each available lint check, and the auto-fix command is able to fix all of them. The test may not be comprehensive, but it provides a reasonable starting point. We can begin using this script now and iterate on it if we discover cases where the auto-fix does not behave correctly. ## Are there any user-facing changes? No <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
1 parent 1d19c52 commit 6bf5e98

File tree

9 files changed

+450
-72
lines changed

9 files changed

+450
-72
lines changed

‎ci/scripts/doc_prettier_check.sh‎

Lines changed: 56 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -17,41 +17,68 @@
1717
# specific language governing permissions and limitations
1818
# under the License.
1919

20-
SCRIPT_PATH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
21-
22-
MODE="--check"
23-
ACTION="Checking"
24-
if [ $# -gt 0 ]; then
25-
if [ "$1" = "--write" ]; then
26-
MODE="--write"
27-
ACTION="Formatting"
28-
else
29-
echo "Usage: $0 [--write]" >&2
30-
exit 1
31-
fi
20+
set -euo pipefail
21+
22+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
23+
SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
24+
PRETTIER_VERSION="2.7.1"
25+
PRETTIER_TARGETS=(
26+
'{datafusion,datafusion-cli,datafusion-examples,dev,docs}/**/*.md'
27+
'!datafusion/CHANGELOG.md'
28+
README.md
29+
CONTRIBUTING.md
30+
)
31+
32+
source "${SCRIPT_DIR}/utils/git.sh"
33+
34+
MODE="check"
35+
ALLOW_DIRTY=0
36+
37+
usage() {
38+
cat >&2 <<EOF
39+
Usage: $SCRIPT_NAME [--write] [--allow-dirty]
40+
41+
Runs prettier@${PRETTIER_VERSION} over markdown docs.
42+
--write Run with \`--write\` to format files (requires a clean git worktree, no uncommitted changes).
43+
--allow-dirty Allow \`--write\` to run even when the git worktree has uncommitted changes.
44+
EOF
45+
exit 1
46+
}
47+
48+
while [[ $# -gt 0 ]]; do
49+
case "$1" in
50+
--write)
51+
MODE="write"
52+
;;
53+
--allow-dirty)
54+
ALLOW_DIRTY=1
55+
;;
56+
-h|--help)
57+
usage
58+
;;
59+
*)
60+
usage
61+
;;
62+
esac
63+
shift
64+
done
65+
66+
if [[ "$MODE" == "write" && $ALLOW_DIRTY -eq 0 ]]; then
67+
require_clean_work_tree "$SCRIPT_NAME" || exit 1
3268
fi
3369

34-
echo "$SCRIPT_PATH: $ACTION documents with prettier"
70+
echo "[${SCRIPT_NAME}] prettier@${PRETTIER_VERSION} ${MODE}"
3571

3672
# Ensure `npx` is available
3773
if ! command -v npx >/dev/null 2>&1; then
3874
echo "npx is required to run the prettier check. Install Node.js (e.g., brew install node) and re-run." >&2
3975
exit 1
4076
fi
41-
42-
# Ignore subproject CHANGELOG.md because it is machine generated
43-
npx prettier@2.7.1 $MODE \
44-
'{datafusion,datafusion-cli,datafusion-examples,dev,docs}/**/*.md' \
45-
'!datafusion/CHANGELOG.md' \
46-
README.md \
47-
CONTRIBUTING.md
48-
status=$?
49-
50-
if [ $status -ne 0 ]; then
51-
if [ "$MODE" = "--check" ]; then
52-
echo "Prettier check failed. Re-run with --write (e.g., ./ci/scripts/doc_prettier_check.sh --write) to format files, commit the changes, and re-run the check." >&2
53-
else
54-
echo "Prettier format failed. Files may have been modified; commit any changes and re-run." >&2
55-
fi
56-
exit $status
77+
78+
PRETTIER_MODE=(--check)
79+
if [[ "$MODE" == "write" ]]; then
80+
PRETTIER_MODE=(--write)
5781
fi
82+
83+
# Ignore subproject CHANGELOG.md because it is machine generated
84+
npx "prettier@${PRETTIER_VERSION}" "${PRETTIER_MODE[@]}" "${PRETTIER_TARGETS[@]}"

‎ci/scripts/license_header.sh‎

Lines changed: 59 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,62 @@
1717
# specific language governing permissions and limitations
1818
# under the License.
1919

20-
# Check Apache license header
21-
set -ex
22-
hawkeye check --config licenserc.toml
20+
set -euo pipefail
21+
22+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
23+
SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
24+
25+
source "${SCRIPT_DIR}/utils/git.sh"
26+
27+
MODE="check"
28+
ALLOW_DIRTY=0
29+
HAWKEYE_CONFIG="licenserc.toml"
30+
31+
usage() {
32+
cat >&2 <<EOF
33+
Usage: $SCRIPT_NAME [--write] [--allow-dirty]
34+
35+
Checks Apache license headers with \`hawkeye check --config $HAWKEYE_CONFIG\`.
36+
--write Run \`hawkeye format --config $HAWKEYE_CONFIG\` to auto-add/fix headers (requires a clean git worktree, no uncommitted changes).
37+
--allow-dirty Allow \`--write\` to run even when the git worktree has uncommitted changes.
38+
EOF
39+
exit 1
40+
}
41+
42+
while [[ $# -gt 0 ]]; do
43+
case "$1" in
44+
--write)
45+
MODE="write"
46+
;;
47+
--allow-dirty)
48+
ALLOW_DIRTY=1
49+
;;
50+
-h|--help)
51+
usage
52+
;;
53+
*)
54+
usage
55+
;;
56+
esac
57+
shift
58+
done
59+
60+
if [[ "$MODE" == "write" && $ALLOW_DIRTY -eq 0 ]]; then
61+
require_clean_work_tree "$SCRIPT_NAME" || exit 1
62+
fi
63+
64+
if [[ "$MODE" == "write" ]]; then
65+
echo "[${SCRIPT_NAME}] \`hawkeye format --config ${HAWKEYE_CONFIG}\`"
66+
if ! hawkeye format --config "${HAWKEYE_CONFIG}"; then
67+
status=$?
68+
# hawkeye returns exit code 1 when it applies fixes; treat that as success.
69+
if [[ $status -eq 1 ]]; then
70+
echo "[${SCRIPT_NAME}] hawkeye format applied fixes (exit 1 treated as success)"
71+
else
72+
exit $status
73+
fi
74+
fi
75+
else
76+
echo "[${SCRIPT_NAME}] \`hawkeye check --config ${HAWKEYE_CONFIG}\`"
77+
hawkeye check --config "${HAWKEYE_CONFIG}"
78+
fi

‎ci/scripts/rust_clippy.sh‎

Lines changed: 57 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,5 +17,60 @@
1717
# specific language governing permissions and limitations
1818
# under the License.
1919

20-
set -ex
21-
cargo clippy --all-targets --workspace --features avro,integration-tests,extended_tests -- -D warnings
20+
set -euo pipefail
21+
22+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
23+
SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
24+
CLIPPY_FEATURES="avro,integration-tests,extended_tests"
25+
CLIPPY_ARGS=(--all-targets --workspace --features "$CLIPPY_FEATURES")
26+
CLIPPY_LINT_ARGS=(-- -D warnings)
27+
28+
source "${SCRIPT_DIR}/utils/git.sh"
29+
30+
MODE="check"
31+
ALLOW_DIRTY=0
32+
33+
usage() {
34+
cat >&2 <<EOF
35+
Usage: $SCRIPT_NAME [--write] [--allow-dirty]
36+
37+
Runs \`cargo clippy\` to lint.
38+
--write Run \`cargo clippy --fix\` to apply fixes for clippy lints (requires a clean git worktree, no uncommitted changes).
39+
--allow-dirty Allow \`--write\` to run even when the git worktree has uncommitted or staged changes.
40+
EOF
41+
exit 1
42+
}
43+
44+
while [[ $# -gt 0 ]]; do
45+
case "$1" in
46+
--write)
47+
MODE="write"
48+
;;
49+
--allow-dirty)
50+
ALLOW_DIRTY=1
51+
;;
52+
-h|--help)
53+
usage
54+
;;
55+
*)
56+
usage
57+
;;
58+
esac
59+
shift
60+
done
61+
62+
if [[ "$MODE" == "write" && $ALLOW_DIRTY -eq 0 ]]; then
63+
require_clean_work_tree "$SCRIPT_NAME" || exit 1
64+
fi
65+
66+
CLIPPY_CMD=(cargo clippy)
67+
if [[ "$MODE" == "write" ]]; then
68+
CLIPPY_CMD+=(--fix)
69+
if [[ $ALLOW_DIRTY -eq 1 ]]; then
70+
CLIPPY_CMD+=(--allow-dirty --allow-staged)
71+
fi
72+
fi
73+
CLIPPY_CMD+=("${CLIPPY_ARGS[@]}" "${CLIPPY_LINT_ARGS[@]}")
74+
75+
echo "[${SCRIPT_NAME}] \`${CLIPPY_CMD[*]}\`"
76+
"${CLIPPY_CMD[@]}"

‎ci/scripts/rust_docs.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
# specific language governing permissions and limitations
1818
# under the License.
1919

20+
# Note: cargo doc does not support an auto-fix mode; this script runs the check-only build.
2021
set -ex
2122
export RUSTDOCFLAGS="-D warnings"
2223
cargo doc --document-private-items --no-deps --workspace

‎ci/scripts/rust_fmt.sh‎

Lines changed: 49 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,5 +17,52 @@
1717
# specific language governing permissions and limitations
1818
# under the License.
1919

20-
set -ex
21-
cargo fmt --all -- --check
20+
set -euo pipefail
21+
22+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
23+
SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
24+
source "${SCRIPT_DIR}/utils/git.sh"
25+
26+
MODE="check"
27+
ALLOW_DIRTY=0
28+
29+
usage() {
30+
cat >&2 <<EOF
31+
Usage: $0 [--write] [--allow-dirty]
32+
33+
Runs \`cargo fmt --all -- --check\` by default to verify Rust formatting.
34+
--write Run \`cargo fmt --all\` to auto-fix formatting (requires a clean git worktree, no uncommitted changes).
35+
--allow-dirty Allow \`--write\` to run even when the git worktree has uncommitted changes.
36+
EOF
37+
exit 1
38+
}
39+
40+
while [[ $# -gt 0 ]]; do
41+
case "$1" in
42+
--write)
43+
MODE="write"
44+
;;
45+
--allow-dirty)
46+
ALLOW_DIRTY=1
47+
;;
48+
-h|--help)
49+
usage
50+
;;
51+
*)
52+
usage
53+
;;
54+
esac
55+
shift
56+
done
57+
58+
if [[ "$MODE" == "write" && $ALLOW_DIRTY -eq 0 ]]; then
59+
require_clean_work_tree "$SCRIPT_NAME" || exit 1
60+
fi
61+
62+
if [[ "$MODE" == "write" ]]; then
63+
echo "[${SCRIPT_NAME}] \`cargo fmt --all\`"
64+
cargo fmt --all
65+
else
66+
echo "[${SCRIPT_NAME}] \`cargo fmt --all -- --check\`"
67+
cargo fmt --all -- --check
68+
fi

‎ci/scripts/rust_toml_fmt.sh‎

Lines changed: 50 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,53 @@
1717
# specific language governing permissions and limitations
1818
# under the License.
1919

20-
# Run `taplo format` with flag `--check` in dry run to check formatting
21-
# without overwritng the file. If any error occur, you may want to
22-
# rerun `taplo format` to fix the formatting automatically.
23-
set -ex
24-
taplo format --check
20+
set -euo pipefail
21+
22+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
23+
SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
24+
25+
source "${SCRIPT_DIR}/utils/git.sh"
26+
27+
MODE="check"
28+
ALLOW_DIRTY=0
29+
30+
usage() {
31+
cat >&2 <<EOF
32+
Usage: $0 [--write] [--allow-dirty]
33+
34+
Runs \`taplo format --check\` by default to verify TOML formatting.
35+
--write Run \`taplo format\` to auto-fix formatting (best-effort; requires a clean git worktree, no uncommitted changes).
36+
--allow-dirty Allow \`--write\` to run even when the git worktree has uncommitted changes.
37+
EOF
38+
exit 1
39+
}
40+
41+
while [[ $# -gt 0 ]]; do
42+
case "$1" in
43+
--write)
44+
MODE="write"
45+
;;
46+
--allow-dirty)
47+
ALLOW_DIRTY=1
48+
;;
49+
-h|--help)
50+
usage
51+
;;
52+
*)
53+
usage
54+
;;
55+
esac
56+
shift
57+
done
58+
59+
if [[ "$MODE" == "write" && $ALLOW_DIRTY -eq 0 ]]; then
60+
require_clean_work_tree "$SCRIPT_NAME" || exit 1
61+
fi
62+
63+
if [[ "$MODE" == "write" ]]; then
64+
echo "[${SCRIPT_NAME}] \`taplo format\`"
65+
taplo format
66+
else
67+
echo "[${SCRIPT_NAME}] \`taplo format --check\`"
68+
taplo format --check
69+
fi

0 commit comments

Comments
 (0)