-
Notifications
You must be signed in to change notification settings - Fork 2.8k
[GHA] Switch code style checks to use Docker environment #32342
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
gkoscins
wants to merge
23
commits into
openvinotoolkit:master
Choose a base branch
from
gkoscins:CVS-171726-GHA-Switch-code-style-checks-to-use-Docker-environment
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+173
−57
Open
Changes from all commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
3fb09ec
Switch code style checks to use Docker environment
gkoscins 42241e1
update docker_tag with my PR ID
gkoscins 8c1ba13
Add clang-format-18 to the docker image
gkoscins 631cc37
Add install shellcheck to docker image
gkoscins d2ca574
update docker files with newest version of git
gkoscins f5dcd55
use path with action checkout
gkoscins 72dba73
Merge branch 'master' into CVS-171726-GHA-Switch-code-style-checks-to…
gkoscins c32e419
Merge branch 'master' into CVS-171726-GHA-Switch-code-style-checks-to…
gkoscins 5e08453
delete path from action checkout
gkoscins 41ddbf4
try to add git safe directory manually
gkoscins 6fef3ab
Merge branch 'master' into CVS-171726-GHA-Switch-code-style-checks-to…
gkoscins ba7284d
install python requirements for code style directly in docker image
gkoscins 931c085
update ubuntu24 code style dockerfile to contain python3-dev
gkoscins 7af6356
use python3-setuptools
gkoscins 406fd51
Merge branch 'master' into CVS-171726-GHA-Switch-code-style-checks-to…
gkoscins b0e7959
Merge branch 'master' into CVS-171726-GHA-Switch-code-style-checks-to…
gkoscins b7ac4eb
Merge branch 'master' into CVS-171726-GHA-Switch-code-style-checks-to…
gkoscins ae3c063
Merge branch 'master' into CVS-171726-GHA-Switch-code-style-checks-to…
gkoscins c6f5666
try to avoid extra step by adding env variables
gkoscins ed91951
try using different user to run the checkout
gkoscins c4523d7
use different user to run checkout on every code style job
gkoscins 61aa182
Merge branch 'master' into CVS-171726-GHA-Switch-code-style-checks-to…
gkoscins 6f442b5
add comments
gkoscins File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,4 @@ | ||
* | ||
!install_build_dependencies.sh | ||
!scripts/install_dependencies/install_openvino_dependencies.sh | ||
!cmake/developer_package/ncc_naming_style/requirements_dev.txt |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1 @@ | ||
pr-31878 | ||
pr-32342 |
40 changes: 40 additions & 0 deletions
40
.github/dockerfiles/ov_test/ubuntu_22_04_x64_code_style/Dockerfile
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
ARG REGISTRY="docker.io" | ||
FROM ${REGISTRY}/library/ubuntu:22.04 | ||
|
||
USER root | ||
|
||
# APT configuration | ||
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \ | ||
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \ | ||
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \ | ||
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf | ||
|
||
ENV DEBIAN_FRONTEND="noninteractive" \ | ||
TZ="Europe/London" | ||
|
||
RUN apt-get update && \ | ||
apt-get install software-properties-common && \ | ||
add-apt-repository --yes --no-update ppa:git-core/ppa && \ | ||
apt-get update && \ | ||
apt-get install \ | ||
curl \ | ||
git \ | ||
ca-certificates \ | ||
cmake \ | ||
wget \ | ||
build-essential \ | ||
shellcheck \ | ||
clang-14 \ | ||
libclang-14-dev \ | ||
&& \ | ||
rm -rf /var/lib/apt/lists/* | ||
|
||
ENV PIP_VERSION="24.0" | ||
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \ | ||
python3 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \ | ||
rm -f get-pip.py | ||
|
||
ADD cmake/developer_package/ncc_naming_style/requirements_dev.txt /requirements_dev.txt | ||
|
||
RUN pip3 install --no-cache-dir -r /requirements_dev.txt && \ | ||
rm -f /requirements_dev.txt |
38 changes: 38 additions & 0 deletions
38
.github/dockerfiles/ov_test/ubuntu_24_04_x64_code_style/Dockerfile
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
ARG REGISTRY="docker.io" | ||
FROM ${REGISTRY}/library/ubuntu:24.04 | ||
|
||
USER root | ||
|
||
# APT configuration | ||
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \ | ||
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \ | ||
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \ | ||
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf | ||
|
||
ENV DEBIAN_FRONTEND="noninteractive" \ | ||
TZ="Europe/London" | ||
|
||
RUN apt-get update && \ | ||
apt-get install software-properties-common && \ | ||
add-apt-repository --yes --no-update ppa:git-core/ppa && \ | ||
apt-get update && \ | ||
apt-get install \ | ||
curl \ | ||
git \ | ||
ca-certificates \ | ||
wget \ | ||
cmake \ | ||
lsb-release \ | ||
gnupg \ | ||
build-essential \ | ||
clang-format-18 \ | ||
shellcheck \ | ||
python3-dev \ | ||
python3-setuptools \ | ||
python3-pip \ | ||
&& \ | ||
wget https://apt.llvm.org/llvm.sh && \ | ||
chmod +x llvm.sh && \ | ||
./llvm.sh 18 && \ | ||
apt-get --assume-yes install binutils-aarch64-linux-gnu gcc-aarch64-linux-gnu g++-aarch64-linux-gnu scons gcc-riscv64-linux-gnu g++-riscv64-linux-gnu binutils-riscv64-linux-gnu && \ | ||
rm -rf /var/lib/apt/lists/* |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -8,23 +8,75 @@ concurrency: | |
permissions: read-all | ||
|
||
jobs: | ||
Smart_CI: | ||
runs-on: ubuntu-latest | ||
outputs: | ||
affected_components: "${{ steps.smart_ci.outputs.affected_components }}" | ||
changed_components: "${{ steps.smart_ci.outputs.changed_components }}" | ||
skip_workflow: "${{ steps.smart_ci.outputs.skip_workflow }}" | ||
steps: | ||
- name: checkout action | ||
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
timeout-minutes: 15 | ||
with: | ||
sparse-checkout: .github/actions/smart-ci | ||
|
||
- name: Get affected components | ||
id: smart_ci | ||
uses: ./.github/actions/smart-ci | ||
with: | ||
repository: ${{ github.repository }} | ||
pr: ${{ github.event.number }} | ||
commit_sha: ${{ github.sha }} | ||
ref_name: ${{ github.ref_name }} | ||
component_pattern: "category: (.*)" | ||
repo_token: ${{ secrets.GITHUB_TOKEN }} | ||
skip_when_only_listed_labels_set: 'docs' | ||
skip_when_only_listed_files_changed: '*.md,*.rst,*.png,*.jpg,*.svg' | ||
|
||
- name: Show affected components | ||
run: | | ||
echo "${{ toJSON(steps.smart_ci.outputs.affected_components) }}" | ||
shell: bash | ||
|
||
Docker: | ||
needs: Smart_CI | ||
runs-on: aks-linux-4-cores-16gb-docker-build | ||
container: | ||
image: openvinogithubactions.azurecr.io/docker_build:0.2 | ||
volumes: | ||
- /mount:/mount | ||
outputs: | ||
images: "${{ steps.handle_docker.outputs.images }}" | ||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
timeout-minutes: 15 | ||
|
||
- uses: ./.github/actions/handle_docker | ||
id: handle_docker | ||
with: | ||
images: | | ||
ov_test/ubuntu_24_04_x64_code_style | ||
ov_test/ubuntu_22_04_x64_code_style | ||
registry: 'openvinogithubactions.azurecr.io' | ||
dockerfiles_root_dir: '.github/dockerfiles' | ||
changed_components: ${{ needs.smart_ci.outputs.changed_components }} | ||
|
||
clang-format: | ||
runs-on: ubuntu-24.04 | ||
if: ${{ github.repository_owner == 'openvinotoolkit' }} | ||
permissions: | ||
pull-requests: write | ||
needs: Docker | ||
runs-on: aks-linux-4-cores-16gb | ||
container: | ||
image: ${{ fromJSON(needs.docker.outputs.images).ov_test.ubuntu_24_04_x64_code_style }} | ||
volumes: | ||
- /mount:/mount | ||
options: --user 1001 # We use standard github-actions user inside container to avoid permission issues after checkout | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we move it inside the docker image configuration? |
||
steps: | ||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
timeout-minutes: 15 | ||
with: | ||
submodules: 'true' | ||
|
||
- name: Install clang-format-18 | ||
run: | | ||
wget https://apt.llvm.org/llvm.sh | ||
chmod +x llvm.sh | ||
sudo ./llvm.sh 18 | ||
|
||
# Run cmake with extra options to cover as much source code as possible: | ||
# - -DENABLE_PROFILING_ITT=ON -DSELECTIVE_BUILD=COLLECT to enable codestyle check for ITT collector | ||
# - -DENABLE_SNIPPETS_LIBXSMM_TPP to cover snippets TPP adaptation | ||
|
@@ -44,23 +96,19 @@ jobs: | |
filter_mode: nofilter | ||
|
||
clang-format-aarch64: | ||
runs-on: ubuntu-24.04 | ||
if: ${{ github.repository_owner == 'openvinotoolkit' }} | ||
permissions: | ||
pull-requests: write | ||
needs: Docker | ||
runs-on: aks-linux-4-cores-16gb | ||
container: | ||
image: ${{ fromJSON(needs.docker.outputs.images).ov_test.ubuntu_24_04_x64_code_style }} | ||
volumes: | ||
- /mount:/mount | ||
options: --user 1001 # We use standard github-actions user inside container to avoid permission issues after checkout | ||
steps: | ||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
timeout-minutes: 15 | ||
with: | ||
submodules: 'true' | ||
|
||
- name: Install clang-format-18 and cross-compilation dependencies | ||
run: | | ||
wget https://apt.llvm.org/llvm.sh | ||
chmod +x llvm.sh | ||
sudo ./llvm.sh 18 | ||
sudo apt update | ||
sudo apt --assume-yes install binutils-aarch64-linux-gnu gcc-aarch64-linux-gnu g++-aarch64-linux-gnu scons | ||
|
||
# Run cmake with extra options to cover as much source code as possible: | ||
# - -DENABLE_PROFILING_ITT=ON -DSELECTIVE_BUILD=COLLECT to enable codestyle check for ITT collector | ||
# - -DENABLE_SNIPPETS_LIBXSMM_TPP to cover snippets TPP adaptation | ||
|
@@ -80,25 +128,19 @@ jobs: | |
filter_mode: nofilter | ||
|
||
clang-format-riscv64: | ||
runs-on: ubuntu-24.04 | ||
if: ${{ github.repository_owner == 'openvinotoolkit' }} | ||
permissions: | ||
pull-requests: write | ||
needs: Docker | ||
runs-on: aks-linux-4-cores-16gb | ||
container: | ||
image: ${{ fromJSON(needs.docker.outputs.images).ov_test.ubuntu_24_04_x64_code_style }} | ||
volumes: | ||
- /mount:/mount | ||
options: --user 1001 # We use standard github-actions user inside container to avoid permission issues after checkout | ||
steps: | ||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
timeout-minutes: 15 | ||
with: | ||
submodules: 'true' | ||
|
||
- name: Install clang-format-18 and cross-compilation dependencies | ||
run: | | ||
wget https://apt.llvm.org/llvm.sh | ||
chmod +x llvm.sh | ||
sudo ./llvm.sh 18 | ||
sudo apt update | ||
sudo apt --assume-yes install gcc-riscv64-linux-gnu g++-riscv64-linux-gnu binutils-riscv64-linux-gnu | ||
|
||
# Run cmake with extra options to cover as much source code as possible: | ||
# - -DENABLE_PROFILING_ITT=ON -DSELECTIVE_BUILD=COLLECT to enable codestyle check for ITT collector | ||
- name: CMake configure | ||
run: cmake -DENABLE_CLANG_FORMAT=ON -DENABLE_TESTS=ON -DENABLE_PROFILING_ITT=ON -DSELECTIVE_BUILD=COLLECT -DCMAKE_TOOLCHAIN_FILE=cmake/toolchains/riscv64.linux.toolchain.cmake -B build_riscv64 | ||
|
||
|
@@ -115,21 +157,19 @@ jobs: | |
filter_mode: nofilter | ||
|
||
ShellCheck: | ||
runs-on: ubuntu-22.04 | ||
if: ${{ github.repository_owner == 'openvinotoolkit' }} | ||
permissions: | ||
pull-requests: write | ||
needs: Docker | ||
runs-on: aks-linux-4-cores-16gb | ||
container: | ||
image: ${{ fromJSON(needs.docker.outputs.images).ov_test.ubuntu_22_04_x64_code_style }} | ||
volumes: | ||
- /mount:/mount | ||
options: --user 1001 # We use standard github-actions user inside container to avoid permission issues after checkout | ||
steps: | ||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
timeout-minutes: 15 | ||
with: | ||
submodules: 'true' | ||
|
||
- name: Install ShellCheck | ||
run: | | ||
sudo apt update | ||
sudo apt --assume-yes install shellcheck | ||
|
||
- name: CMake configure | ||
run: cmake -B build | ||
|
||
|
@@ -150,23 +190,19 @@ jobs: | |
"./temp/*" | ||
|
||
NamingConventionCheck: | ||
runs-on: ubuntu-22.04 | ||
if: ${{ github.repository_owner == 'openvinotoolkit' }} | ||
needs: Docker | ||
runs-on: aks-linux-4-cores-16gb | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why not the medium runner? |
||
container: | ||
image: ${{ fromJSON(needs.docker.outputs.images).ov_test.ubuntu_22_04_x64_code_style }} | ||
volumes: | ||
- /mount:/mount | ||
options: --user 1001 # We use standard github-actions user inside container to avoid permission issues after checkout | ||
steps: | ||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
timeout-minutes: 15 | ||
with: | ||
submodules: 'true' | ||
|
||
- name: Install Clang dependency | ||
run: | | ||
sudo apt update | ||
sudo apt --assume-yes remove clang-7 clang-8 clang-9 clang-10 clang-11 clang-12 clang-13 clang-15 | ||
sudo apt --assume-yes install clang-14 libclang-14-dev | ||
|
||
- name: Install Python-based dependencies | ||
run: python3 -m pip install -r cmake/developer_package/ncc_naming_style/requirements_dev.txt | ||
|
||
- name: CMake configure | ||
run: cmake -B build | ||
|
||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally we install Python requirements in runtime (since we need to ensure the freshest state of requirements, including those that don't include direct requirement file change, e.g. when a new version of a non-pinned module is released on PyPi). If we're installing it inside the docker - only direct file change will cause image rebuild, which is not enough
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do we need this step then?