Skip to content

Commit f8edbcc

Browse files
NaveenaiduNaveenaidu
authored
Merge pull request ceph#63136 from Naveenaidu/wip-naveen-config-diff-scripts
script: add configuration diff tool
2 parents 0790938 + dacabbb commit f8edbcc

File tree

6 files changed

+939
-0
lines changed

6 files changed

+939
-0
lines changed

‎.github/labeler.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -345,3 +345,6 @@ script:
345345
- src/script/**
346346
- admin/**
347347
- doc/scripts/**
348+
349+
config-change:
350+
- src/common/options/**

‎.github/workflows/diff-ceph-config.yml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
name: Check ceph config changes
2+
on:
3+
pull_request_target:
4+
types:
5+
- opened
6+
- synchronize
7+
- edited
8+
- reopened
9+
10+
# The following permissions are needed to write a comment to repo
11+
permissions:
12+
issues: write
13+
contents: read
14+
pull-requests: write
15+
16+
jobs:
17+
pull_request:
18+
runs-on: ubuntu-latest
19+
steps:
20+
- name: checkout ceph.git
21+
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 #v4.2.2
22+
with:
23+
path: ceph
24+
sparse-checkout: |
25+
src/script
26+
src/common/options
27+
.github/workflows
28+
29+
- name: Setup Python
30+
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 #v5.6.0
31+
with:
32+
python-version: '3.13'
33+
34+
- name: Install python packages
35+
run: |
36+
pip3 install -r ./src/script/config-diff/requirements.txt
37+
working-directory: ceph
38+
39+
- name: execute config diff tool
40+
id: diff_tool
41+
env:
42+
REF_REPO: ${{ github.event.pull_request.base.repo.clone_url }}
43+
REF_BRANCH: ${{ github.event.pull_request.base.ref }}
44+
REMOTE_REPO: ${{ github.event.pull_request.head.repo.clone_url }}
45+
REMOTE_BRANCH: ${{ github.event.pull_request.head.ref }}
46+
run: |
47+
{
48+
echo 'DIFF_JSON<<EOF'
49+
python3 ./src/script/config-diff/config_diff.py diff-branch-remote-repo --ref-branch $REF_BRANCH --remote-repo $REMOTE_REPO --cmp-branch $REMOTE_BRANCH --format=posix-diff --skip-clone
50+
echo EOF
51+
} >> "$GITHUB_OUTPUT"
52+
working-directory: ceph
53+
54+
- name: Post output as a comment
55+
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea #v7.0.1
56+
env:
57+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
58+
DIFF_JSON_OUTPUT: ${{ steps.diff_tool.outputs.DIFF_JSON }}
59+
with:
60+
script: |
61+
const configDiff = process.env.DIFF_JSON_OUTPUT;
62+
const postComment = require('./ceph/.github/workflows/scripts/config-diff-post-comment.js');
63+
postComment({ github, context, core, configDiff });

‎.github/workflows/scripts/config-diff-post-comment.js‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
module.exports = async ({ github, context, core, configDiff }) => {
2+
try {
3+
// Do not create comment if there are no configuration changes
4+
if (!configDiff) {
5+
console.log("No changes detected. Skipping comment creation.");
6+
return;
7+
}
8+
9+
const commentBody = `
10+
### Config Diff Tool Output
11+
12+
\`\`\`diff
13+
14+
${configDiff}
15+
16+
\`\`\`
17+
18+
19+
The above configuration changes are found in the PR. Please update the relevant release documentation if necessary.
20+
`;
21+
22+
core.summary.addRaw(commentBody);
23+
await core.summary.write()
24+
25+
const { owner, repo } = context.repo;
26+
const issueNumber = context.payload.pull_request.number;
27+
28+
// List all files in the pull request
29+
core.info("Fetching list of files changed in the pull request...");
30+
const files = await github.paginate(
31+
github.rest.pulls.listFiles,
32+
{
33+
owner,
34+
repo,
35+
pull_number: issueNumber,
36+
per_page: 100,
37+
}
38+
);
39+
40+
// Annotate YAML files
41+
files.forEach(file => {
42+
// Only annotate the `yaml.in` files present in `src/common/options` folder
43+
if (file.filename.endsWith(".yaml.in") && file.filename.startsWith("src/common/options/")) {
44+
core.info(`Annotating file: ${file.filename}`);
45+
core.notice(
46+
`Configuration changes detected in ${file.filename}. Please update the relevant release documentation if necessary.`,
47+
{
48+
title: "Configuration Change Detected",
49+
file: file.filename,
50+
startLine: 1,
51+
endLine: 1,
52+
}
53+
);
54+
}
55+
});
56+
57+
58+
// List all the comments
59+
const comments = await github.paginate(
60+
github.rest.issues.listComments, {
61+
owner,
62+
repo,
63+
issue_number: issueNumber,
64+
per_page: 100,
65+
}
66+
);
67+
68+
const existingComment = comments.find(comment => comment.body.includes("### Config Diff Tool Output"));
69+
70+
if (existingComment) {
71+
core.info("A config diff comment already exists, deleting it...");
72+
} else {
73+
core.info("Creating a new config diff comment...");
74+
// Create a new comment
75+
await github.rest.issues.createComment({
76+
issue_number: issueNumber,
77+
owner,
78+
repo,
79+
body: commentBody,
80+
});
81+
82+
}
83+
84+
// Set the status as FAILED if any configuration changes are detected
85+
core.setFailed("Configuration Changes Detected, Update release documents - if necessary");
86+
} catch (error) {
87+
core.setFailed(error.message);
88+
}
89+
}

‎src/script/config-diff/README.md‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
# Ceph Config Diff Tool
2+
3+
This program is a Python-based tool designed to compare the configuration options of Ceph by cloning the repository and analyzing the files present in the `src/common/options` directory. It supports three modes of operation: `diff-branch`, `diff-tag`, and `diff-branch-remote-repo`.
4+
5+
Note: This tool **does not** compare the configuration changes occuring on a running Ceph cluster.
6+
7+
## Features
8+
9+
- **Compare Branches**: Compare configuration options between two branches in the same repository.
10+
- **Compare Tags**: Compare configuration options between two tags in the same repository.
11+
- **Compare Branches Across Repositories**: Compare configuration options between branches in different repositories.
12+
13+
## Usage
14+
15+
Run the program using the following command:
16+
17+
```bash
18+
python3 config_diff.py <mode> [options]
19+
```
20+
21+
### Modes
22+
23+
1. **`diff-branch`**: Compare configuration options between two branches in the same Ceph repository.
24+
```bash
25+
python3 config_diff.py diff-branch --ref-branch <branch1> --cmp-branch <branch2> [--ref-repo <repo-url>] [--skip-clone] [--format <output-format>]
26+
```
27+
28+
- `--ref-branch`: The reference branch to compare against.
29+
- `--cmp-branch`: The branch to compare.
30+
- `--ref-repo`: (Optional) The repository URL. Defaults to the Ceph upstream repository.
31+
- `--skip-clone`: (Optional) Skips cloning repositories for diff. **Note**: When using this flag, the script must be run from a valid Ceph upstream repository or a forked repository that has access to the branches present in the upstream repository or already contains those branches.
32+
- `--format`: (Optional) Specify the output format for the configuration diff. Options are `json` or `posix-diff`. Default is `json`.
33+
34+
35+
2. **`diff-tag`**: Compare configuration options between two tags in the same Ceph repository.
36+
```bash
37+
python3 config_diff.py diff-tag --ref-tag <tag1> --cmp-tag <tag2> [--ref-repo <repo-url>] [--posix-diff]
38+
```
39+
40+
- `--ref-tag`: The reference tag to compare against.
41+
- `--cmp-tag`: The tag to compare.
42+
- `--ref-repo`: (Optional) The repository URL. Defaults to the Ceph upstream repository.
43+
- `--skip-clone`: (Optional) Skips cloning repositories for diff. **Note**: When using this flag, the script must be run from a valid Ceph upstream repository or a forked repository that has access to the branches present in the upstream repository or already contains those branches.
44+
- `--format`: (Optional) Specify the output format for the configuration diff. Options are `json` or `posix-diff`. Default is `json`.
45+
46+
3. **`diff-branch-remote-repo`**: Compare configuration options between branches in different repositories.
47+
```bash
48+
python3 config_diff.py diff-branch-remote-repo --ref-branch <branch1> --cmp-branch <branch2> --remote-repo <repo-url> [--ref-repo <repo-url>] [--posix-diff]
49+
```
50+
51+
- `--ref-branch`: The reference branch to compare against.
52+
- `--cmp-branch`: The branch to compare.
53+
- `--remote-repo`: The remote repository URL for the branch to compare.
54+
- `--ref-repo`: (Optional) The repository URL for the reference branch. Defaults to the Ceph upstream repository.
55+
- `--skip-clone`: (Optional) Skips cloning repositories for diff. **Note**: When using this flag, the script must be run from a valid Ceph upstream repository or a forked repository that has access to the branches present in the upstream repository or already contains those branches.
56+
- `--format`: (Optional) Specify the output format for the configuration diff. Options are `json` or `posix-diff`. Default is `json`.
57+
58+
### Example Commands
59+
60+
1. Compare two branches in the same repository:
61+
```bash
62+
python3 config_diff.py diff-branch --ref-branch main --cmp-branch feature-branch
63+
```
64+
65+
The above command checks how the configuration options present in the branch
66+
`feature-branch` has changed from the `main` branch
67+
68+
2. Compare two tags in the same repository:
69+
```bash
70+
python3 config_diff.py diff-tag --ref-tag v1.0.0 --cmp-tag v1.1.0
71+
```
72+
The above command checks how the configuration options present in the tag
73+
`v1.1.0` has changed from the `v1.0.0` branch
74+
75+
3. Compare branches across repositories:
76+
```bash
77+
python3 config_diff.py diff-branch-remote-repo --ref-branch main --cmp-branch feature-branch --remote-repo https://github.com/username/ceph
78+
```
79+
80+
81+
The above command checks how the configuration options present in the
82+
`feature-branch` of the remote repository `https://github.com/username/ceph`
83+
has changed from the `main` branch of the ceph upstream repo. This command is
84+
used by the `diff-ceph-config.yml` GitHub Action to check for any
85+
configuration changes on the any PR's that are raised.
86+
87+
### Important Notes for `--skip-clone` Flag
88+
89+
- When using the `--skip-clone` flag, the script assumes it is being run from a valid Ceph upstream repository or a forked repository.
90+
- If using a forked repository, ensure that the forked repository has access to the branches present in the upstream repository or already contains those branches.
91+
92+
93+
## Output
94+
95+
The program generates a JSON output containing the following structure:
96+
97+
```json
98+
{
99+
"added": {
100+
"daemon1": ["config1", "config2"]
101+
},
102+
"deleted": {
103+
"daemon2": ["config3"]
104+
},
105+
"modified": {
106+
"daemon3": {
107+
"config4": {
108+
"key1": {
109+
"before": "old_value",
110+
"after": "new_value"
111+
}
112+
}
113+
}
114+
}
115+
}
116+
```
117+
118+
- **`added`**: Configuration options added in the comparing version.
119+
- **`deleted`**: Configuration options removed in the comparing version.
120+
- **`modified`**: Configuration options modified between the two versions.
121+
122+
## Example:
123+
124+
### diff in JSON
125+
126+
```json
127+
{
128+
"added": {
129+
"mgr.yaml.in": [
130+
"mgr_client_bytes_test"
131+
]
132+
},
133+
"deleted": {
134+
"osd.yaml.in": [
135+
"osd_scrub_load_threshold"
136+
]
137+
},
138+
"modified": {
139+
"global.yaml.in": {
140+
"mon_debug_no_require_tentacle": {
141+
"default": {
142+
"before": false,
143+
"after": true
144+
}
145+
}
146+
}
147+
}
148+
}
149+
```
150+
151+
### POSIX like diff:
152+
153+
```diff
154+
+ added: bluefs_wal_envelope_mode
155+
- removed: rgw_d4n_l1_write_open_flags
156+
! changed: mon_nvmeofgw_skip_failovers_interval: old: 16
157+
! changed: mon_nvmeofgw_skip_failovers_interval: new: 32
158+
```

0 commit comments

Comments
 (0)