Skip to content

add benchmarks using pytest-benchmark and codspeed #3562

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

add benchmarks using pytest-benchmark and codspeed #3562

wants to merge 8 commits into from

Conversation

@d-v-b
Copy link
Contributor

@d-v-b d-v-b commented Oct 30, 2025

since #3554 was an unpopular direction I'm going instead with codspeed + pytest-benchmark. Opening as a draft because I haven't looked into how codspeed works at all, but I'd like people to weigh in on whether these initial benchmarks make sense. Naturally we can add more specific ones later, but I figured just some bulk array read / write workloads would be a good start.

@github-actions github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Oct 30, 2025
@github-actions github-actions bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Oct 30, 2025
@d-v-b d-v-b marked this pull request as ready for review October 30, 2025 13:46
@d-v-b
Copy link
Contributor Author

d-v-b commented Oct 30, 2025

@zarr-developers/steering-council I don't have permission to register this repo with codspeed. I submitted a request to register it, could someone approve it?

@d-v-b d-v-b requested review from dcherian and jhamman and removed request for jhamman October 30, 2025 14:37
@d-v-b d-v-b mentioned this pull request Oct 30, 2025
Open
@normanrz
Copy link
Member

@zarr-developers/steering-council I don't have permission to register this repo with codspeed. I submitted a request to register it, could someone approve it?

done

@d-v-b
Copy link
Contributor Author

d-v-b commented Oct 30, 2025

does anyone have opinions about benchmarks? feel free to suggest something concrete. Otherwise, I think we should take this as-is and deal with later benchmarks (like partial shard read / writes) in a subsequent pr

@codspeed-hq
Copy link

codspeed-hq bot commented Oct 30, 2025

CodSpeed Performance Report

Congrats! CodSpeed is installed 🎉

🆕 80 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

dcherian
dcherian reviewed Oct 30, 2025
View reviewed changes
.github/workflows/codspeed.yml
uses: CodSpeedHQ/action@v4
with:
mode: instrumentation
run: hatch run test.py3.11-1.26-minimal:run-benchmark
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we test the latest instead? seems more appropriate...

@dcherian
Copy link
Contributor

feel free to suggest something concrete

indexing please. that'll exercise the codec pipeline too.

a peakmem metric would be good to track also, if possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dcherian dcherian dcherian left review comments

@jhamman jhamman Awaiting requested review from jhamman

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@d-v-b @normanrz @dcherian