Skip to content

Commit f1a25a4

Browse files
fulmicotonfulmicoton
authored
Refactoring of the CI. (#3676)
The previous CI was strangely relying on workflows to split the CI and have it run in parallel. A lot of the work was made several times: e.g. installing rust stable + nightly, installing packages, starting our builder image etc. Overall it was done x6 times. It was strongly overkill considering some of the steps run in (<1s). After this refactoring we run in two "jobs" with a different set of steps. - Lint runs clippy + all of the smaller jobs (cargo fmt, cargo deny, checking the license) - Test runs the unit test. In addition, we stop relying on the builder image and the associated confusion. For instance, the rust stable installed in the CI workflow was actually not used. The quickwit-builder image one was taken because of the rust-toolchain.toml. After the PR, we run on stock ubuntu-latest directly. We then rely on rustup to install the right version of cargo/rust as define in the rust-toolchain. Updating the rust version does not require any extra changes. cargo deny / cargo nextest need to be build unfortunately. That step is however cached.
1 parent adfff7f commit f1a25a4

File tree

3 files changed

+85
-47
lines changed

3 files changed

+85
-47
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 82 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ env:
1616
RUST_BACKTRACE: 1
1717
RUSTFLAGS: -Dwarnings
1818
RUSTDOCFLAGS: -Dwarnings -Arustdoc::private_intra_doc_links
19-
TEST_DATABASE_URL: postgres://quickwit-dev:quickwit-dev@postgres:5432/quickwit-metastore-dev
19+
TEST_DATABASE_URL: postgres://quickwit-dev:quickwit-dev@localhost:5432/quickwit-metastore-dev
2020

2121
# Ensures that we cancel running jobs for the same PR / same workflow.
2222
concurrency:
@@ -25,32 +25,8 @@ concurrency:
2525

2626
jobs:
2727
tests:
28-
name: ${{ matrix.task.name }} (${{ matrix.os }})
29-
runs-on: ${{ matrix.os }}
30-
strategy:
31-
fail-fast: false
32-
matrix:
33-
os: [ubuntu-latest]
34-
task:
35-
- name: cargo clippy
36-
command: cargo clippy --workspace --tests --all-features
37-
cache: true
38-
- name: cargo deny
39-
command: cargo deny check licenses
40-
cache: false
41-
- name: cargo doc
42-
command: cargo doc --no-deps --all-features --document-private-items
43-
cache: true
44-
- name: cargo nextest
45-
command: cargo nextest run --features=postgres --profile ci --retries 1
46-
cache: true
47-
- name: License headers check
48-
command: bash scripts/check_license_headers.sh
49-
cache: false
50-
- name: rustfmt
51-
command: cargo +nightly fmt --all -- --check
52-
cache: false
53-
container: public.ecr.aws/l6o9a3f9/quickwit-builder:latest
28+
name: Unit tests
29+
runs-on: "ubuntu-latest"
5430
services:
5531
# PostgreSQL service container
5632
postgres:
@@ -69,35 +45,95 @@ jobs:
6945
--health-retries 5
7046
steps:
7147
- uses: actions/checkout@v3
48+
- name: Install Ubuntu packages
49+
run: sudo apt-get -y install protobuf-compiler python3 python3-pip
7250
- uses: dorny/paths-filter@v2
73-
id: modified-files
51+
id: modified
7452
with:
7553
filters: |
76-
qw-src:
77-
- quickwit/**
78-
ui-src:
79-
- quickwit/quickwit-ui/**
80-
- name: Setup nightly Rust Toolchain
54+
rust_src:
55+
- quickwit/**.rs
56+
- quickwit/**.toml
57+
- quickwit/**.proto
58+
# The following step is just meant to install rustup actually.
59+
# The next one installs the correct toolchain.
60+
- name: Install rustup
8161
uses: actions-rs/toolchain@v1
62+
if: steps.modified.outputs.rust_src == 'true'
8263
with:
83-
toolchain: nightly
84-
components: rustfmt, clippy
64+
toolchain: 1.70
65+
components: ""
8566
- name: Setup stable Rust Toolchain
67+
if: steps.modified.outputs.rust_src == 'true'
68+
run: rustup show
69+
working-directory: ./quickwit
70+
- name: Setup cache
71+
uses: Swatinem/rust-cache@v2
72+
if: steps.modified.outputs.rust_src == 'true'
73+
with:
74+
workspaces: "./quickwit -> target"
75+
- name: Install nextest
76+
if: steps.modified.outputs.rust_src == 'true'
77+
uses: taiki-e/cache-cargo-install-action@v1
78+
with:
79+
tool: cargo-nextest
80+
- name: cargo nextest
81+
if: steps.modified.outputs.rust_src == 'true'
82+
run: cargo nextest run --features=postgres --profile ci --retries 1
83+
working-directory: ./quickwit
84+
lints:
85+
name: Lints
86+
runs-on: "ubuntu-latest"
87+
steps:
88+
- uses: actions/checkout@v3
89+
- uses: dorny/paths-filter@v2
90+
id: modified
91+
with:
92+
filters: |
93+
rust_src:
94+
- quickwit/**.rs
95+
- quickwit/**.toml
96+
- quickwit/**.proto
97+
- name: Install Ubuntu packages
98+
if: steps.modified.outputs.rust_src == 'true'
99+
run: sudo apt-get -y install protobuf-compiler python3 python3-pip
100+
- name: Setup nightly Rust Toolchain (for rustfmt)
86101
uses: actions-rs/toolchain@v1
102+
if: steps.modified.outputs.rust_src == 'true'
87103
with:
88-
toolchain: stable
89-
override: true
90-
components: rustfmt, clippy
104+
toolchain: nightly
105+
components: rustfmt
106+
- name: Setup stable Rust Toolchain
107+
if: steps.modified.outputs.rust_src == 'true'
108+
run: rustup show
109+
working-directory: ./quickwit
91110
- name: Setup cache
92-
if: matrix.task.cache == true
111+
if: steps.modified.outputs.rust_src == 'true'
93112
uses: Swatinem/rust-cache@v2
94113
with:
95-
key: "v1-${{ matrix.task.name }}"
96114
workspaces: "./quickwit -> target"
97-
- name: Install nextest
98-
if: matrix.task.name == 'cargo nextest'
99-
uses: taiki-e/install-action@nextest
100-
- name: ${{ matrix.task.name }}
101-
if: steps.modified-files.outputs.qw-src == 'true' && steps.modified-files.outputs.ui-src == 'false'
102-
run: ${{ matrix.task.command }}
115+
- name: Install cargo deny
116+
if: steps.modified.outputs.rust_src == 'true'
117+
uses: taiki-e/cache-cargo-install-action@v1
118+
with:
119+
tool: cargo-deny
120+
- name: cargo clippy
121+
if: steps.modified.outputs.rust_src == 'true'
122+
run: cargo clippy --workspace --tests --all-features
123+
working-directory: ./quickwit
124+
- name: cargo deny
125+
if: steps.modified.outputs.rust_src == 'true'
126+
run: cargo deny check licenses
127+
working-directory: ./quickwit
128+
- name: cargo doc
129+
if: steps.modified.outputs.rust_src == 'true'
130+
run: cargo doc
131+
working-directory: ./quickwit
132+
- name: License headers check
133+
if: always()
134+
run: bash scripts/check_license_headers.sh
135+
working-directory: ./quickwit
136+
- name: rustfmt
137+
if: steps.modified.outputs.rust_src == 'true'
138+
run: cargo +nightly fmt --all -- --check
103139
working-directory: ./quickwit

‎quickwit/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
[workspace]
22
resolver = "2"
3-
rust-version = "1.70"
43
members = [
54
"quickwit-actors",
65
"quickwit-aws",
@@ -414,3 +413,4 @@ sasl2-sys = { git = "https://github.com/quickwit-oss/rust-sasl/", rev = "daca921
414413
#tracing-log = { git = "https://github.com/trinity-1686a/tracing.git", rev = "6806cac3" }
415414
#tracing-opentelemetry = { git = "https://github.com/trinity-1686a/tracing.git", rev = "6806cac3" }
416415
#tracing-subscriber = { git = "https://github.com/trinity-1686a/tracing.git", rev = "6806cac3" }
416+

‎quickwit/rust-toolchain.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,4 @@
11
[toolchain]
22
channel = "1.70"
3+
components = ["cargo", "clippy", "rustfmt", "rust-docs"]
4+

0 commit comments

Comments
 (0)