Gather statistics for multiple runs under same job id

[alfer-edb]

Hi Steve,

Is there a functionallity to run lets say a sequence of vus ( eg. 1, 2,4, ..., n) and gather all stats referring to a single job id? I played around it a bit and it didn't seem like i could obtain anything like that, might be a good idea to implement?

I was also wondering how stats would work if we had autopilot in place.

Thanks
Alessandro

[sm-shaw]

Alessandro,

You may need to elaborate a bit more for me to completely understand the question.
So to run a sequence of multiple VUs yes you have autopilot as you have mentioned or if using the CLI you can use a simple for loop to run for any sequence you wish in a script. In this case each run e.g. for 1VU, 2VUs 4VUs etc is a single run and all of the output for a particular run is stored under a job id. You can use the web service to view all of the output for each of these runs https://www.hammerdb.com/docs/ch10s02.html. At the moment, there is no functionality to compare the output between job ids.

[alfer-edb]

Let me make an example.
Suppose we want to test a system with a variable number of virtual users (either by autopilot or a tcl loop) and instead of getting individual metrics per vu we wanted to collect all the metrics, each related to a specific vu in order to build a NOPM/TPM chart that displays a curve. Would that be possible?

The current implementation shows this:

hammerdb>job 65B3E10460FD03E213739313 result
[
  "65B3E10460FD03E213739313",
  "2024-01-26 16:42:44",
  "2 Active Virtual Users configured",
  "TEST RESULT : System achieved 15374 NOPM from 35723 PostgreSQL TPM"
]

The output i have in mind would be something like this:

hammerdb>job 123456789 result
[
vu: 1,
<datetime>,
tpm: xx,
nopm: yy

vu: 2
<datetime>,
tpm: xxx,
nopm: yyy

vu: n
<datetime>,
tpm: xxxx,
nopm: yyyy
]

With this format it would be simple to generate a chart (NOPM/VU) with a variable number of virtual users.
Likewise for timing metrics, but i guess in that case we would need to make a change in the json structure to make each vu parent of all procedures.

For the CLI, we could just set a job id before the vucreate loop to indicate a single parent job container for all the subsequent runs.

Let me know if I wasn't clear enough :)

[sm-shaw]

Currently, there is something similar in that if you type "jobs result" with no jobid it will print out all of the jobids, the date, the active vus and the test result:

...
[
  "65B234A360FC03E283931393",
  "2024-01-25 02:14:59",
  "1 Active Virtual Users configured",
  "TEST RESULT : System achieved 21738 NOPM from 50490 MariaDB TPM"
]

[
  "65B2364F60FC03E253032353",
  "2024-01-25 02:22:07",
  "2 Active Virtual Users configured",
  "TEST RESULT : System achieved 44680 NOPM from 103743 MariaDB TPM"
]

[
  "65B237FB60FC03E223239353",
  "2024-01-25 02:29:15",
  "4 Active Virtual Users configured",
  "TEST RESULT : System achieved 88956 NOPM from 206542 MariaDB TPM"
]

[
  "65B239A860FC03E273137373",
  "2024-01-25 02:36:24",
  "8 Active Virtual Users configured",
  "TEST RESULT : System achieved 170542 NOPM from 396349 MariaDB TPM"
]

[
  "65B23B5A60FC03E243332353",
  "2024-01-25 02:43:38",
  "12 Active Virtual Users configured",
  "TEST RESULT : System achieved 237702 NOPM from 552718 MariaDB TPM"
]
...

So what you are proposing is adding a parent jobid to group specific jobs so then jobs result "parentid" would print out only a subset of these results rather than all of them. It would also then be a logical extension that a chart could be generated for these grouped jobs under a parent id showing VUs on the x-axis and NOPM on the y-axis. At the moment this must be done manually by copying the data such as above into a spreadsheet.

It is certainly do-able and an Issue could be raised as a suggested enhancement and of course contributions are always welcome as comparing job data has been suggested previously, however it is not something done as yet.

[alfer-edb]

Thank you for the workaround, I'll get some script working to extract datapoints and build the chart.

I'll be happy to help integrating this enanchement, I'll take a look at the code and see what i can do ;)

[sm-shaw]

Excellent. I don't think it should be too difficult. It would probably need a new SQLite table to map jobids to a parent. This would then provide the jobids that could be queried with the existing jobs result command to collect all of the NOPM values for the parent. Then it use Apache e-charts to generate the graphs for the current web-service https://www.hammerdb.com/blog/uncategorized/hammerdb-v4-8-new-features-viewing-your-benchmark-results-with-the-hammerdb-web-service/ and the line graph for the transaction counter is probably the most appropriate as a guide for building a graph of the NOPM values.
This will be a useful addition as for now the step is manual to run "jobs result" to collect all ids and then filter the ones you need into a spreadsheet to produce this type of performance profile graph.