benchmark tool for Riak KV

[richamishra006]

Hi, can someone please suggest me some benchmarking tool for Riak to generate and perform load test on the machine.

[martinsumner]

The develop-3.0 branch of basho_bench is the main one used for pre-release testing of Riak - https://github.com/basho/basho_bench/tree/develop-3.0 with this configuration.

There is a alternative branch which is used for testing the same thing via the HTTP API.

[richamishra006]

Thanks for your reply, I installed rebar3, but when running make all command, it is giving error

root@68d36c17d1d9:/basho_bench# make all
/basho_bench/rebar get-deps
make: /basho_bench/rebar: Command not found
make: *** [Makefile:22: deps] Error 127

Please let me know if I am missing anything

[richamishra006]

ok i resolved that error by updating Makefile and replacing rebar with rebar3

[richamishra006]

i have installed erlang 25 and rebar3 but getting this error when running make all

root@68d36c17d1d9:/basho_bench# make all
/basho_bench/rebar3 get-deps
=ERROR REPORT==== 21-Jun-2023::15:30:10.069489 ===
beam/beam_load.c(551): Error loading function rebar3:run_aux/2: op put_tuple u x:
  please re-compile this module with an Erlang/OTP 25 compiler


escript: exception error: undefined function rebar3:main/1
  in function  escript:run/2 (escript.erl, line 750)
  in call from escript:start/1 (escript.erl, line 277)
  in call from init:start_em/1 
  in call from init:do_boot/3 
make: *** [Makefile:22: deps] Error 127

[martinsumner]

Try with OTP 22.

Run ./rebar3 escriptize from the basho_bench folder.

That should generate a basho_bench executable in _build/default/bin which can then be used like this nohup _build/default/bin/basho_bench examples/riakc_nhs_general.config & (or whatever config file), which will start generating output in tests/current/. You will need to update the config file to include your ip addresses (not the use of commas not periods in the addresses).

Sorry, the readme instructions are lagging behind the changes made. There's a bit of fiddling normally required to get this working.

The generation of charts using R is probably broken, you may need to do your own work on the csv outputs to chart any results.

[richamishra006]

Thankyou for the response, it is working with otp 22,now probably one last doubt, I have three riak nodes riak0.local, riak1.local and riak2.local
in what all files I need to update this name or I can update with IP address as well, but there are lots of file in examples directory,
basically the end goal is to generate load, and how about if I want to increase the load, is there any parameter which I need to update for increasing the load on cluster

[richamishra006]

hey @martinsumner could you please help me out, I am just about to finish my load test

[martinsumner]

For basho_bench you have a configuration file which you can set-up to control your test. You can pick one of the examples as a starting point, a good starting point might be the riakc_nhs_general.config which is what is used for riak release testing.

Here is an annotated version of that config file to explain what it means:

{mode, max}.

{duration, 1920}.

{report_interval, 10}.

{node_name, testnode1}.

{concurrent, 100}.

The first few elements define the throughput for the test.

• The mode in this case is {mode, max} - hit the cluster as hard as you can, each worker will try a new piece of work once the last has been completed.
• The {concurrent, 100} means have a 100 workers each generating and sending requests concurrently.
• The {duration, 1920} just sets the test to run for 1920 minutes.

So in this case you would increase the throughput by increasing the {concurrent, 100} value.

{driver, basho_bench_driver_nhs}.

{record_bucket, "recordBucket"}.
{document_bucket, "documentBucket"}.
{record_sync, "one"}.
{document_sync, "backend"}.
{node_confirms, 2}.

{postcode_indexcount, 6}.

%% Ignored by alwaysget and unique operations
{key_generator, {eightytwenty_int, 100000000}}.

{value_generator, {semi_compressible, 10000, 2000, 10, 0.1}}.

%% For alwaysget operations what is:
%% - the maximum number of keys per worker (max number of keys = this * concurrent)
%% - whether the inserts should be in key_order
{alwaysget, {2000000, 700000, skew_order}}.
{unique, {6000, key_order}}.

• the next key but of configuration is to define the driver (the erlang file which defines the individual test commands). In this case this is the basho_bench_driver_nhs.
• all the rest of this section is the configuration for that driver, things like object size (mean variation), size of the key space, how many indexes a record will have, bucket names to generate keys in. There's a lot going on here.

{pb_ips, [{127,0,0,1}]}.
{http_ips, [{127,0,0,1}]}.

• This is where the IP addresses should go. You can point at a single address (i.e. a load-balancer), or a list of addresses. I think only IP addresses will work, and they need to be comma separated inside a tuple e.g. [{192, 168, 3, 1}, {192, 168, 3, 2}]. Normally the pb IPs and http IPs will be the same list.

{operations, [{alwaysget_pb, 620}, {alwaysget_updatewith2i, 130}, 
                {put_unique, 90}, {get_unique, 130}, {delete_unique, 25},
                {postcodequery_http, 2}, {dobquery_http, 3}]}.

• This defines the distribution of operations. In this case there are 1000 operations in total, so 62% of operations will be of the form alwayget_pb (this is an operation that fetches an object that has been added as part of the test - so never gets a not found - using the PB API). {postcodequery_http, 2} - means that 0.2% of test requests will be for a HTTP 2i query of the postcode index.

[martinsumner]

This isn't easy. There's a lot going on in this particular config file to generate various test scenarios. this particular test scenario runs in an upload mode until a certain threshold is reached, and then switches to a load which has more GETs than PUTs once the database is of sufficient size to be worth testing.

There are much simpler test configs available - https://github.com/basho/basho_bench/blob/mas-nhs-httponly/examples/riakc_pb.config is a good example. The simpler test scenarios tend to give unrealistic tests - e.g. most of the test runs with against a small database, with lots of not_found responses, and there's no testing of 2i etc.

[martinsumner]

econnrefused normally means either there is no lustener on the TCP port, or some sort of firewall is blocking it. On the riak node 172.22.0.212 if you do netstat -an | grep LISTEN | grep 8087 is it listening on that TCP port? Can you telnet to that port/IP from the basho_bench server?

I'm not sure why riakc_pb.config would work though. this has an info message just before it connects - may be worth confirming the details being reported in the console log when it hits this log:

https://github.com/basho/basho_bench/blob/develop-3.0/src/basho_bench_driver_riakc_pb.erl#L130

[richamishra006]

yes i got to know that riak node is getting down again and again, when checked the logs, I found this error

Supervisor riak_core_sup had child riak_core_vnode_manager started with riak_core_vnode_manager:start_link() at <0.298.0> exit with reason {{function_clause,[{riak_kv_vnode,terminate,[{bad_return_value,{stop,{{badmatch,{error,{{badmatch,{error,{{badmatch,{error,emfile}},[{leveled_pmanifest,open_manifest,1,[{file,"/root/riak/rel/pkg/out/riak-3.0.10-OTP22.3/_build/default/lib/leveled/src/leveled_pmanifest.erl"},{line,128}]},{leveled_penciller,start_from_file,1,[{file,"/root/riak/rel/pkg/out/riak-3.0.10-OTP22.3/_build/default/lib/leveled/src/leveled_penciller.erl"},{line,1231}]},{gen_server,init_it,2,[{file,"gen_server.erl"},...]},...]}}},...}}},...}}},...],...},...]},...} in context child_terminated

I increased the limit and fs.file-max but still getting this error

[martinsumner]

This looks like a standard ulimit issue. Guidance for setting ulimit here

[richamishra006]

Thankyou @martinsumner , i updated the limits by following the doc you shared, but still the service is dropping again and again. So I removed the ring and decided to test with a single node itself(where I am not facing issues), I used the riakc_nhs_general.config file and ran the test.
Also I installed R using this command sudo apt-get install r-base as mentioned in this doc https://docs.riak.com/riak/kv/latest/using/performance/benchmarking/index.html but when I am running priv/summary.r -i tests/current the summary.png is not getting created. and I am getting this as the output

root@application-node01:~/basho_bench# priv/summary.r -i tests/current
[1] "plyr"
Loading required package: plyr
[1] "grid"
Loading required package: grid
[1] "getopt"
Loading required package: getopt
[1] "proto"
Loading required package: proto
[1] "ggplot2"
Loading required package: ggplot2
[1] 0
[1] -Inf
Warning message:
In max(summary$elapsed) : no non-missing arguments to max; returning -Inf
Error: No latency information available to analyze in tests/current
Execution halted

can you please help me in generating the graph for this test

[martinsumner]

I can't help I'm afraid. Personally, I load results into a spreadsheet and then manipulate and chart them there. This made it a easier (for me) to chart comparisons between different runs of basho_bench etc and tidy up the presentation.

I'm not sufficiently familiar with R/ggplot to troubleshoot this code.

[richamishra006]

can you please explain me the way you are doing it, I mean which file you are uploading to spreadheet, it will be really helpful for me