-
Notifications
You must be signed in to change notification settings - Fork 9
feat(linters): Add cli tool for running clang-tidy in parallel. #17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
LinZhihao-723
wants to merge
11
commits into
y-scope:main
Choose a base branch
from
LinZhihao-723:clang-tidy-parallel-script
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 4 commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
9f8fefe
Initial commit for cli
LinZhihao-723 7a6002c
Let's use uv
LinZhihao-723 5b12b0c
Polish readme
LinZhihao-723 d74ac85
Update readme
LinZhihao-723 cf407b9
Renaming + readme update
LinZhihao-723 27fcb24
Project rename
LinZhihao-723 a6ca5f3
Rename symbols
LinZhihao-723 48083bd
Merge main
LinZhihao-723 817fdef
Remove build dir
LinZhihao-723 62edc76
Lock clang-tidy version to 19
LinZhihao-723 548ff02
Update the cli script.
LinZhihao-723 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| # Environment | ||
| venv*/ | ||
| __pycache__ | ||
|
|
||
| # Packaging | ||
| *.egg-info | ||
|
|
||
| # Dev | ||
| .mypy_cache | ||
| .ruff_cache |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| # yscope-clang-tidy-utils | ||
|
|
||
| This project is a CLI scripts for running [clang-tidy][clang-tidy-home] checks. | ||
|
|
||
| ## Requirements | ||
| - [uv] | ||
|
|
||
| ## Installation | ||
| To install the tool system-wide (permanently), run: | ||
| ```shell | ||
| uv tool install . | ||
| ``` | ||
|
|
||
| Or, installing the tool in a virtual environment: | ||
| ```shell | ||
| uv venv | ||
| uv pip install . | ||
| ``` | ||
| Chech [here] for uv's virtual environment behaviour. | ||
|
|
||
| ## Usage | ||
| ```shell | ||
| yscope-clang-tidy-utils [-h] [-j NUM_JOBS] FILE [FILE ...] [-- CLANG-TIDY-ARGS ...] | ||
| ``` | ||
| Note: | ||
| - By default, the number of jobs will be set to the number of cores in the running environment. | ||
| - Anything after `--` will be considered as clang-tidy arguments and will be directly passed into | ||
| clang-tidy. | ||
|
|
||
| ## Development: | ||
| Run linting tools with the following commands: | ||
| ```shell | ||
| uv tool run mypy src | ||
| uv tool run docformatter -i src | ||
| uv tool run black src | ||
| uv tool run ruff check --fix src | ||
| ``` | ||
|
|
||
| [clang-tidy-home]: https://clang.llvm.org/extra/clang-tidy/ | ||
| [uv]: https://docs.astral.sh/uv/getting-started/installation/ | ||
| [uv-venv]: https://docs.astral.sh/uv/pip/compatibility/#virtual-environments-by-default |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| [project] | ||
| name = "yscope-clang-tidy-utils" | ||
| version = "0.0.1" | ||
| authors = [ | ||
| { name="zhihao lin", email="[email protected]" }, | ||
| ] | ||
| requires-python = ">=3.10" | ||
| dependencies = [ | ||
| "clang-tidy >= 19.1.0", | ||
| ] | ||
|
|
||
| [dependency-groups] | ||
| dev = [ | ||
| "black >= 24.10.0", | ||
| "docformatter >= 1.7.5", | ||
| "mypy >= 1.14.1", | ||
| "ruff >= 0.9.2", | ||
| ] | ||
|
|
||
| [project.scripts] | ||
| yscope-clang-tidy-utils = "yscope_clang_tidy_utils.cli:main" | ||
|
|
||
| [tool.black] | ||
| line-length = 100 | ||
| color = true | ||
| preview = true | ||
|
|
||
| [tool.docformatter] | ||
| make-summary-multi-line = true | ||
| pre-summary-newline = true | ||
| recursive = true | ||
| wrap-summaries = 100 | ||
| wrap-descriptions = 100 | ||
|
|
||
| [tool.mypy] | ||
| explicit_package_bases = true | ||
| strict = true | ||
| pretty = true | ||
|
|
||
| [tool.ruff] | ||
| line-length = 100 | ||
|
|
||
| [tool.ruff.lint] | ||
| select = ["E", "I", "F"] | ||
| isort.order-by-type = false | ||
175 changes: 175 additions & 0 deletions
175
cli/clang-tidy-utils/src/yscope_clang_tidy_utils/cli.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,175 @@ | ||
| import argparse | ||
| import asyncio | ||
| import dataclasses | ||
| import multiprocessing | ||
| import subprocess | ||
| import sys | ||
| from typing import List, Optional | ||
|
|
||
|
|
||
| @dataclasses.dataclass | ||
| class ClangTidyResult: | ||
| """ | ||
| Class that represents clang-tidy's execution results. | ||
| """ | ||
|
|
||
| file_name: str | ||
| ret_code: int | ||
| stdout: str | ||
| stderr: str | ||
|
|
||
|
|
||
| def create_clang_tidy_task_arg_list(file: str, clang_tidy_args: List[str]) -> List[str]: | ||
| """ | ||
| :param file: The file to check. | ||
| :param clang_tidy_args: The clang-tidy cli arguments. | ||
| :return: A list of arguments to run clang-tidy to check the given file with the given args. | ||
| """ | ||
| args: List[str] = ["clang-tidy", file] | ||
| args.extend(clang_tidy_args) | ||
| return args | ||
|
|
||
|
|
||
| async def execute_clang_tidy_task(file: str, clang_tidy_args: List[str]) -> ClangTidyResult: | ||
| """ | ||
| Executes a single clang-tidy task by checking one file using a process managed by asyncio. | ||
|
|
||
| :param file: The file to check. | ||
| :param clang_tidy_args: The clang-tidy cli arguments. | ||
| :return: Execution results represented by an instance of `ClangTidyResult`. | ||
| """ | ||
| task_args: List[str] = create_clang_tidy_task_arg_list(file, clang_tidy_args) | ||
| try: | ||
| process = await asyncio.create_subprocess_exec( | ||
| *task_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE | ||
| ) | ||
| stdout, stderr = await process.communicate() | ||
| except asyncio.CancelledError: | ||
| process.terminate() | ||
| await process.wait() | ||
| raise | ||
|
|
||
| assert process.returncode is not None | ||
| return ClangTidyResult( | ||
| file, | ||
| process.returncode, | ||
| stdout.decode("UTF-8"), | ||
| stderr.decode("UTF-8"), | ||
| ) | ||
|
|
||
|
|
||
| async def execute_clang_tidy_task_with_sem( | ||
| sem: asyncio.Semaphore, file: str, clang_tidy_args: List[str] | ||
| ) -> ClangTidyResult: | ||
| """ | ||
| Wrapper of `execute_clang_tidy_task` with a global semaphore for concurrency control. | ||
|
|
||
| :param sem: The global semaphore for concurrency control. | ||
| :param file: The file to check. | ||
| :param clang_tidy_args: The clang-tidy cli arguments. | ||
| :return: Forwards `execute_clang_tidy_task`'s return values. | ||
| """ | ||
| async with sem: | ||
| return await execute_clang_tidy_task(file, clang_tidy_args) | ||
|
|
||
|
|
||
| async def clang_tidy_parallel_execution_entry( | ||
| num_jobs: int, | ||
| files: List[str], | ||
| clang_tidy_args: List[str], | ||
| ) -> int: | ||
| """ | ||
| Async entry for running clang-tidy checks in parallel. | ||
|
|
||
| :param num_jobs: The maximum number of jobs allowed to run in parallel. | ||
| :param files: The list of files to check. :clang_tidy_args: The clang-tidy cli arguments. | ||
| """ | ||
| sem: asyncio.Semaphore = asyncio.Semaphore(num_jobs) | ||
| tasks: List[asyncio.Task[ClangTidyResult]] = [ | ||
| asyncio.create_task(execute_clang_tidy_task_with_sem(sem, file, clang_tidy_args)) | ||
| for file in files | ||
| ] | ||
| num_total_files: int = len(files) | ||
|
|
||
| ret_code: int = 0 | ||
| try: | ||
| for idx, clang_tidy_task in enumerate(asyncio.as_completed(tasks)): | ||
| result: ClangTidyResult = await clang_tidy_task | ||
| if 0 != result.ret_code: | ||
| ret_code = 1 | ||
| print(f"[{idx + 1}/{num_total_files}]: {result.file_name}") | ||
| print(result.stdout) | ||
| print(result.stderr) | ||
| except asyncio.CancelledError as e: | ||
| print(f"\nAll tasks cancelled: {e}") | ||
| for task in tasks: | ||
| task.cancel() | ||
|
|
||
| return ret_code | ||
|
|
||
|
|
||
| def main() -> None: | ||
| parser: argparse.ArgumentParser = argparse.ArgumentParser( | ||
| description="yscope-clang-tidy-utils cli options.", | ||
| ) | ||
|
|
||
| parser.add_argument( | ||
| "-j", | ||
| "--num-jobs", | ||
| type=int, | ||
| help="Number of jobs to run for parallel processing.", | ||
| ) | ||
|
|
||
| parser.add_argument( | ||
| "input_files", | ||
| metavar="FILE", | ||
| type=str, | ||
| nargs="+", | ||
| help="Input files to process.", | ||
| ) | ||
|
|
||
| default_parser_usage: str = parser.format_usage() | ||
| if default_parser_usage.endswith("\n"): | ||
| default_parser_usage = default_parser_usage[:-1] | ||
| usage_prefix: str = "usage: " | ||
| if default_parser_usage.startswith(usage_prefix): | ||
| default_parser_usage = default_parser_usage[len(usage_prefix) :] | ||
| usage: str = default_parser_usage + " [-- CLANG-TIDY-ARGS ...]" | ||
| parser.usage = usage | ||
|
|
||
| args: List[str] = sys.argv[1:] | ||
| delimiter_idx: Optional[int] = None | ||
| try: | ||
| delimiter_idx = args.index("--") | ||
| except ValueError: | ||
| pass | ||
|
|
||
| cli_args: List[str] = args | ||
| clang_tidy_args: List[str] = [] | ||
| if delimiter_idx is not None: | ||
| cli_args = args[:delimiter_idx] | ||
| clang_tidy_args = args[delimiter_idx + 1 :] | ||
|
|
||
| parsed_cli_args: argparse.Namespace = parser.parse_args(cli_args) | ||
|
|
||
| num_jobs: int | ||
| if parsed_cli_args.num_jobs is not None: | ||
| num_jobs = parsed_cli_args.num_jobs | ||
| else: | ||
| num_jobs = multiprocessing.cpu_count() | ||
|
|
||
| ret_code: int = 0 | ||
| try: | ||
| ret_code = asyncio.run( | ||
| clang_tidy_parallel_execution_entry( | ||
| num_jobs, parsed_cli_args.input_files, clang_tidy_args | ||
| ) | ||
| ) | ||
| except KeyboardInterrupt: | ||
| pass | ||
|
|
||
| exit(ret_code) | ||
|
|
||
|
|
||
| if "__main__" == __name__: | ||
| main() |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We may drop
yscope-prefixThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't have a strong preference, but if we ever do upload this to pypi it probably makes more sense to have the prefix. If we never upload it to pypi the name won't ever really matter right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I prefer to drop the prefix for now since it's likely this will be an internal tool for the foreseeable future.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed.