Add evp_hash perftool

[andrewkdinh]

Adds a CLI tool that computes hashes using the specified algorithm. Prints out the average time per hash computation after 5 seconds.

Features:

• Uses the EVP API by default, but this tool can also use the corresponding deprecated API's
• Currently supported algorithms: [SHA1, SHA224, SHA256, SHA384, SHA512]
• Configurable number of times to update digest. Pass 1 for one-shot
• Configurable thread count

Example:

$ ./evp_hash -h
Usage: evp_hash [-h] [-x] [-t] [-u update-times] [-a algorithm] thread-count
-h - print this help output
-x - use deprecated API instead of EVP API
-t - terse output
-u update-times - times to update digest. 1 for one-shot. By default, do one-shot
-a algorithm - One of: [SHA1, SHA224, SHA256, SHA384, SHA512]. By default, use SHA1

$ ./evp_hash -u 10 -a SHA512 15 # 10 update iterations, SHA512 hash algorithm, 15 threads
Average time per hash: 12.430825us

$ ./evp_hash -u 10 -a SHA512 -x 15 # now use deprecated API's
Average time per hash: 11.912479us

Fixes openssl/project#1681

[bob-beck]

Don't print averages.. print "I did XXXXXXX operagtions in XXXXX seconds"

[bob-beck]

Don't print averages.. print "I did XXXXXXX operagtions in XXXXX seconds"

or even "did XXXXXX operations with api in time. basically, don't "distill" the output. print out exactly what you're doing.

[bob-beck]

Don't print averages.. print "I did XXXXXXX operagtions in XXXXX seconds"

or even "did XXXXXX operations with api in time. basically, don't "distill" the output. print out exactly what you're doing.

github mangled my comment. I hate it

Do XXXX operaions of type foo using the bar api in XXXX seconds.

[Sashan]

Don't print averages.. print "I did XXXXXXX operagtions in XXXXX seconds"

But I think we might need to keep printing of averages for -t option, which is used by @quarckster to obtain numbers for Grafana (in case the tool is going to be used to gather data for Grafana).

[andrewkdinh]

Do XXXX operaions of type foo using the bar api in XXXX seconds.

@bob-beck How the test works is that it runs threads for 5 seconds and averages the result.

So if there's multiple threads, maybe something like this?

$ ./evp_hash 10 SHA512 3 # 10 update iterations, 3 threads
Runtime: 5 seconds | Update iterations: 10
Thread 1: XXXX SHA512 hash operations using the EVP API
Thread 2: XXXX SHA512 hash operations using the EVP API
Thread 3: XXXX SHA512 hash operations using the EVP API
Average time per hash: 12.430825us

@Sashan and yes, I'll keep the -t option for Grafana

[quarckster]

That's correct. -t flag is useful for automation, please implement it for each test.

[bob-beck]

Do XXXX operaions of type foo using the bar api in XXXX seconds.

@bob-beck How the test works is that it runs threads for 5 seconds and averages the result.

So if there's multiple threads, maybe something like this?

$ ./evp_hash 10 SHA512 3 # 10 update iterations, 3 threads
Runtime: 5 seconds | Update iterations: 10
Thread 1: XXXX SHA512 hash operations using the EVP API
Thread 2: XXXX SHA512 hash operations using the EVP API
Thread 3: XXXX SHA512 hash operations using the EVP API
Average time per hash: 12.430825us

@Sashan and yes, I'll keep the -t option for Grafana

sure, and total at the end (after stopping timing)

[bob-beck]

Do XXXX operaions of type foo using the bar api in XXXX seconds.

@bob-beck How the test works is that it runs threads for 5 seconds and averages the result.
So if there's multiple threads, maybe something like this?

$ ./evp_hash 10 SHA512 3 # 10 update iterations, 3 threads
Runtime: 5 seconds | Update iterations: 10
Thread 1: XXXX SHA512 hash operations using the EVP API
Thread 2: XXXX SHA512 hash operations using the EVP API
Thread 3: XXXX SHA512 hash operations using the EVP API
Average time per hash: 12.430825us

@Sashan and yes, I'll keep the -t option for Grafana

sure, and total at the end (after stopping timing)

Is there any reason why you don't know beforehand?

I.e. You tell it to do 100000 operations. if it splits it up evenly, meh.

[jogme]

How the test works is that it runs threads for 5 seconds and averages the result.

Just a note: that's not 100% true, because the tests are run until we pass the 5s margin and that might be by any amount time is needed to get to the while cycle condition

[npajkovsky]

Do XXXX operaions of type foo using the bar api in XXXX seconds.

@bob-beck How the test works is that it runs threads for 5 seconds and averages the result.
So if there's multiple threads, maybe something like this?

$ ./evp_hash 10 SHA512 3 # 10 update iterations, 3 threads
Runtime: 5 seconds | Update iterations: 10
Thread 1: XXXX SHA512 hash operations using the EVP API
Thread 2: XXXX SHA512 hash operations using the EVP API
Thread 3: XXXX SHA512 hash operations using the EVP API
Average time per hash: 12.430825us

@Sashan and yes, I'll keep the -t option for Grafana

sure, and total at the end (after stopping timing)

I would not introduce yet another output for threaded operations. @esyr had some good ideas in his #50.

[bob-beck]

How the test works is that it runs threads for 5 seconds and averages the result.

Just a note: that's not 100% true, because the tests are run until we pass the 5s margin and that might be by any amount time is needed to get to the while cycle condition

5 seconds seems like a very short time. it would certainly be better to have the ability to specify the number of operations to perform.

This makes it much easier to compare impact on different types of machines, etc. because you can do the same amount of work and measuer the time difference.

[jogme]

How the test works is that it runs threads for 5 seconds and averages the result.

Just a note: that's not 100% true, because the tests are run until we pass the 5s margin and that might be by any amount time is needed to get to the while cycle condition

5 seconds seems like a very short time. it would certainly be better to have the ability to specify the number of operations to perform.

This makes it much easier to compare impact on different types of machines, etc. because you can do the same amount of work and measuer the time difference.

We had it like that, then there was a decision to rework to this model for some reason

[npajkovsky]

How the test works is that it runs threads for 5 seconds and averages the result.

Just a note: that's not 100% true, because the tests are run until we pass the 5s margin and that might be by any amount time is needed to get to the while cycle condition

5 seconds seems like a very short time. it would certainly be better to have the ability to specify the number of operations to perform.
This makes it much easier to compare impact on different types of machines, etc. because you can do the same amount of work and measuer the time difference.

We had it like that, then there was a decision to rework to this model for some reason

#29

[andrewkdinh]

I've incorporated everybody's feedback but haven't changed the output of the program yet.

Since all the other perftools have the same output Average time per operation: XXXXus, maybe we can add a -v verbose option for either output like @esyr or @bob-beck suggests.

[andrewkdinh]

I've updated the program to work slightly differently. @bob-beck wanted an option to compare performance of EVP API when you had the option to have shared data between threads or not.

$ ./evp_hash -h
Usage: evp_hash [-h] [-t] [-o operation] [-u update-times] [-a algorithm] thread-count
-h - print this help output
-t - terse output
-o operation - mode of operation. One of [deprecated, evp_isolated, evp_shared] (default: evp_shared)
-u update-times - times to update digest. 1 for one-shot (default: 1)
-a algorithm - One of: [SHA1, SHA224, SHA256, SHA384, SHA512] (default: SHA1)
thread-count - number of threads

$ ./evp_hash -o deprecated -a SHA256 1000 # deprecated API, SHA256 hash algorithm, 1000 threads
Average time per hash: 48.245043us

$ ./evp_hash -o evp_shared -a SHA256 1000 # EVP API, shared
Average time per hash: 53.653964us

$ ./evp_hash -o evp_isolated -a SHA256 1000 # EVP API, isolated
Average time per hash: 485.848308us

[Sashan]

looks good to me. thanks.

[Sashan]

merged in as 3eb9e61