Single-threaded performance

[bkomuves]

As I understand, most (?) of the benchmarks are multi-threaded.

However, I think it would be very useful to see single-threaded performance too. There are several reasons for that:

• most people probably have no idea how many threads / cpu cores / hyperthreading / etc those amazon instances have
• different libraries have different levels of support for parallel processing
• some algorithms are inherently less or more parallelizable (MSM and FFT are both interesting examples)
• you may want to build something custom on the top of these libraries, which will have different parallelization properties than the algorithms implemented in those libraries
• easier to compare with other benchmarks, and extrapolate some proxy estimates for machines not present in zkalc
• i personally think single-threaded performance is a more intuitive measure
• but even if you disagree with that, it's certainly a useful measure

So a feature request: the ability to switch between single-threaded and multi-threaded benchmarks (and maybe also present some information about the number of physical / logical cores in the machines used for testing)

[mmaker]

Hi @bkomuves!

I am in favor of creating an advanced pane where users can select more operations and also select single vs multi-core operations. I would be happy to review a PR that addresses this issue, if you want to take a stab at it.
Unfortunately, I don't have capacity for implementing it myself at the moment.

Replying more pedantically to your points:
I think people can have an idea of how many cpu cores etc those amazon instances have easily, they can just hover on the machine name

I also think that the number of threads/the language support/parallelism of an algorithm/metric of comparison are not part of the core scope of zkalc. Zkalc's core business is answering the question of how much will it roughly take to run [public-key protocol] in [modern machine]? with a simple interface that non-experts can use. Towards this goal, I don't think having single-threaded performance as the default is a good idea, because nobody today will use single-threading in production. That said, I agree that it'd be really nice to have an interface for advanced users like you.

Thoughts?

[bkomuves]

I politely disagree.

I always use single-threaded performance when estimating the speeds of the different algorithms (and implementations), because that's what's actually comparable, and you can always just multiply by some ad-hoc parallel multipliers, say 6x-10x for a modern laptop, or 15-30x for a modern server (assuming very good parallelization), or whatever. But the point is that these multipliers change with the hardware.

Not all ZK applications are supposed to run on the latest bleeding edge hardware. And it's simple to estimate the multiplier for your target machine, but pretty much impossible to estimate it for whatever machines these benchmarks where run on, without getting access to the exact same machine and running some benchmarks on your own, which kind of defeats the purpose.

I don't think this service is useful for "non-experts" at all. I however think it can be very useful for experts.

Oh I somehow missed the hovering machine description! Still, to reiterate, I personally have zero idea what EC2 m6g.8xlarge etc means, and the only part of that description I can decipher is the number of cores (which arguably is the most important part) and size of the memory. And I would bet that their meaning 1) changes in time; 2) will become obscurity in a few years as Amazon replaces their server fleet. While single-threaded speeds change much more slowly these days, as we are close to the limits of what the current technology can achieve.

Unfortunately I don't have the capacity either , but maybe when finally I get access to more man-power I can try and delegate.

Thanks.

[mmaker]

Just replying to your paragraph:

Not all ZK applications are supposed to run on the latest bleeding edge hardware.

because I think my view is already in the previous message (in short: I am ok with an advanced interface).

About that, and trying to constructive, we try to support the widest range of hardware and CPU possible, including the cheapest available

and (as mentioned in the about page) we are very welcoming for people to submit their new measurements :)

If you are able to find some man-power I'm happy to have a more extended discussion.

[burdges]

An easy approximation: Ed25519/Ristretto results are already single threaded, which makes them incomparable here usually. If anything else, then use the smallest machine and multiply the results by 8.

This is probably not the approximation you'd like for real zk protocols, especialy not if comparing with bulletproofs. It's likely okay for basic questions though, like: How mush could BLS signatures cost vs Ed25519 or EC VRF? Answer there: BLS verifiers cost almost 200 x the cost of Ed25519. You could reduce this to 42 x for gossip messages by sending the EC VRF proof of the BLS signature.

[alinush]

I also think single-threaded numbers would be preferred to multi-threaded ones.

[mmaker]

@alinush PR are welcome!