Skip to content

Commit 145c02b

Browse files
yossigoyossigo
authored
Merge pull request #100 from filipecosta90/hdr.hist
Enable full percentile spectrum analysis on latency.
2 parents fbff305 + 9dc3690 commit 145c02b

31 files changed

+5080
-184
lines changed

‎.gitignore‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,11 @@ aclocal.m4
22
autom4te.cache
33
config.guess
44
config.h.in
5+
config.h.in~
56
config.h
67
config.sub
78
configure
9+
compile
810
depcomp
911
install-sh
1012
ltmain.sh
@@ -19,6 +21,11 @@ config.log
1921
config.status
2022
libtool
2123
stamp-h1
22-
.idea
23-
.vscode
24+
*.DS_Store
25+
.vscode/*
26+
.idea/*
27+
28+
# memtier outputs
29+
*.hgrm
30+
*.txt
2431
__pycache__

‎Makefile.am‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,11 @@ memtier_benchmark_SOURCES = \
4040
obj_gen.cpp obj_gen.h \
4141
item.cpp item.h \
4242
file_io.cpp file_io.h \
43-
config_types.cpp config_types.h
43+
config_types.cpp config_types.h \
44+
deps/hdr_histogram/hdr_histogram_log.c deps/hdr_histogram/hdr_histogram_log.h deps/hdr_histogram/byteorder.h \
45+
deps/hdr_histogram/hdr_histogram.c deps/hdr_histogram/hdr_histogram.h \
46+
deps/hdr_histogram/hdr_time.c deps/hdr_histogram/hdr_time.h deps/hdr_histogram/hdr_encoding.c deps/hdr_histogram/hdr_encoding.h
47+
4448
memtier_benchmark_LDADD = \
4549
$(LIBEVENT_LIBS) \
4650
$(LIBEVENT_OPENSSL_LIBS) \

‎README.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,3 +156,24 @@ Also, the ratio and the key generator is per client (and not connection).
156156
In this case, setting the ratio to 1:1 does not guarantee 100% hits because
157157
the keys spread to different connections/nodes.
158158

159+
### Full latency spectrum analysis
160+
161+
For distributions that are non-normal, such as the latency, many “basic rules” of normally distributed statistics are violated. Instead of computing just the mean, which tries to express the whole distribution in a single result, we can use a sampling of the distribution at intervals -- percentiles, which tell you how many requests actually would experience that delay.
162+
163+
164+
When used for normally distributed data, the samples are usually taken at regular intervals. However, since the data does not obey to a normal distribution it would be very expensive to keep equally spaced intervals of latency records while enabling large value ranges. We can apply algorithms that can calculate a good approximation of percentiles at minimal CPU and memory cost, such as [t-digest](https://github.com/tdunning/t-digest) or [HdrHistogram](https://github.com/HdrHistogram/HdrHistogram_c). On memtier_benchmark we’ve decided to use the HdrHistogram due to its low memory footprint, high precision, zero allocation during the benchmark and constant access time.
165+
166+
167+
By default Memtier will output the 50th, 99th, and 99.9th percentiles. They are the latency thresholds at which 50%, 99%, and 99.9% of commands are faster than that particular presented value.
168+
To output different percentiles you should use the --print-percentiles option followed by the comma separated list of values ( example: `--print-percentiles 90,99,99.9,99.99` ).
169+
170+
#### Saving the full latency spectrum
171+
To save the full latencies you should use the --hdr-file-prefix option followed by the prefix name you wish the filenames to have.
172+
Each distinct command will be saved into two different files - one in .txt (textual format) and another in .hgrm (HistogramLogProcessor format).
173+
The textual format can be hard to analyze solely, but you can use an [online formatter](http://hdrhistogram.github.io/HdrHistogram/plotFiles.html) to generate visual histograms from it. The .hgrm format will be later added as input to Redislabs [mbdirector](https://github.com/redislabs/mbdirector) to enable visualization of time-domain results.
174+
175+
Sample Visual Feel of the full latency spectrum using an [online formatter](http://hdrhistogram.github.io/HdrHistogram/plotFiles.html):
176+
![alt text][sample_visual_histogram]
177+
178+
179+
[sample_visual_histogram]: ./docs/sample_visual_histogram.png "Sample Full Latency Spectrum Histogram"

‎client.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@ bool client::setup_client(benchmark_config *config, abstract_protocol *protocol,
5656
{
5757
m_config = config;
5858
assert(m_config != NULL);
59+
unsigned long long total_num_of_clients = config->clients*config->threads;
5960

6061
// create main connection
6162
shard_connection* conn = new shard_connection(m_connections.size(), this, m_config, m_event_base, protocol);
@@ -74,7 +75,6 @@ bool client::setup_client(benchmark_config *config, abstract_protocol *protocol,
7475
// Parallel key-pattern determined according to the first command
7576
if ((config->arbitrary_commands->is_defined() && config->arbitrary_commands->at(0).key_pattern == 'P') ||
7677
(config->key_pattern[key_pattern_set]=='P')) {
77-
unsigned long long total_num_of_clients = config->clients*config->threads;
7878
unsigned long long client_index = config->next_client_idx % total_num_of_clients;
7979

8080
unsigned long long range = (config->key_maximum - config->key_minimum)/total_num_of_clients + 1;

‎client.h‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -61,17 +61,17 @@ class client : public connections_manager {
6161
object_generator* m_obj_gen;
6262
run_stats m_stats;
6363

64-
unsigned long long m_reqs_processed; // requests processed (responses received)
65-
unsigned long long m_reqs_generated; // requests generated (wait for responses)
66-
unsigned int m_set_ratio_count; // number of sets counter (overlaps on ratio)
67-
unsigned int m_get_ratio_count; // number of gets counter (overlaps on ratio)
64+
unsigned long long m_reqs_processed; // requests processed (responses received)
65+
unsigned long long m_reqs_generated; // requests generated (wait for responses)
66+
unsigned int m_set_ratio_count; // number of sets counter (overlaps on ratio)
67+
unsigned int m_get_ratio_count; // number of gets counter (overlaps on ratio)
6868
unsigned int m_arbitrary_command_ratio_count; // number of arbitrary commands counter (overlaps on ratio)
69-
unsigned int m_executed_command_index; // current arbitrary command executed
69+
unsigned int m_executed_command_index; // current arbitrary command executed
7070

71-
unsigned long long m_tot_set_ops; // Total number of SET ops
72-
unsigned long long m_tot_wait_ops; // Total number of WAIT ops
71+
unsigned long long m_tot_set_ops; // Total number of SET ops
72+
unsigned long long m_tot_wait_ops; // Total number of WAIT ops
7373

74-
keylist *m_keylist; // used to construct multi commands
74+
keylist *m_keylist; // used to construct multi commands
7575

7676
public:
7777
client(client_group* group);

‎config_types.cpp‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@
3535
#include <netdb.h>
3636

3737
#include <string>
38+
#include <iostream>
3839
#include <stdexcept>
3940
#include <climits>
4041
#include <algorithm>
@@ -87,6 +88,34 @@ config_ratio::config_ratio(const char *ratio_str) :
8788
}
8889
}
8990

91+
config_quantiles::config_quantiles(){
92+
93+
}
94+
95+
config_quantiles::config_quantiles(const char *str)
96+
{
97+
assert(str != NULL);
98+
99+
do {
100+
float quantile;
101+
char *p = NULL;
102+
quantile = strtof(str, &p);
103+
if (!p || (*p != ',' && *p != '\0')) {
104+
quantile_list.clear();
105+
return;
106+
}
107+
str = p;
108+
if (*str) str++;
109+
quantile_list.push_back(quantile);
110+
} while (*str);
111+
}
112+
113+
bool config_quantiles::is_defined(void)
114+
{
115+
return quantile_list.size() > 0;
116+
}
117+
118+
90119
config_weight_list::config_weight_list() :
91120
next_size_weight(0)
92121
{

‎config_types.h‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,15 @@ struct config_ratio {
4646
bool is_defined(void) { return (a > 0 || b > 0); }
4747
};
4848

49+
struct config_quantiles {
50+
std::vector<float> quantile_list;
51+
config_quantiles();
52+
config_quantiles(const char *ratio_str);
53+
bool is_defined(void);
54+
inline std::vector<float>::iterator begin() { return quantile_list.begin(); }
55+
inline std::vector<float>::iterator end() { return quantile_list.end(); }
56+
};
57+
4958
struct config_weight_list {
5059
struct weight_item {
5160
unsigned int size;

‎deps/hdr_histogram/COPYING.txt‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
Creative Commons Legal Code
2+
3+
CC0 1.0 Universal
4+
5+
CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
6+
LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
7+
ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
8+
INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
9+
REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
10+
PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
11+
THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
12+
HEREUNDER.
13+
14+
Statement of Purpose
15+
16+
The laws of most jurisdictions throughout the world automatically confer
17+
exclusive Copyright and Related Rights (defined below) upon the creator
18+
and subsequent owner(s) (each and all, an "owner") of an original work of
19+
authorship and/or a database (each, a "Work").
20+
21+
Certain owners wish to permanently relinquish those rights to a Work for
22+
the purpose of contributing to a commons of creative, cultural and
23+
scientific works ("Commons") that the public can reliably and without fear
24+
of later claims of infringement build upon, modify, incorporate in other
25+
works, reuse and redistribute as freely as possible in any form whatsoever
26+
and for any purposes, including without limitation commercial purposes.
27+
These owners may contribute to the Commons to promote the ideal of a free
28+
culture and the further production of creative, cultural and scientific
29+
works, or to gain reputation or greater distribution for their Work in
30+
part through the use and efforts of others.
31+
32+
For these and/or other purposes and motivations, and without any
33+
expectation of additional consideration or compensation, the person
34+
associating CC0 with a Work (the "Affirmer"), to the extent that he or she
35+
is an owner of Copyright and Related Rights in the Work, voluntarily
36+
elects to apply CC0 to the Work and publicly distribute the Work under its
37+
terms, with knowledge of his or her Copyright and Related Rights in the
38+
Work and the meaning and intended legal effect of CC0 on those rights.
39+
40+
1. Copyright and Related Rights. A Work made available under CC0 may be
41+
protected by copyright and related or neighboring rights ("Copyright and
42+
Related Rights"). Copyright and Related Rights include, but are not
43+
limited to, the following:
44+
45+
i. the right to reproduce, adapt, distribute, perform, display,
46+
communicate, and translate a Work;
47+
ii. moral rights retained by the original author(s) and/or performer(s);
48+
iii. publicity and privacy rights pertaining to a person's image or
49+
likeness depicted in a Work;
50+
iv. rights protecting against unfair competition in regards to a Work,
51+
subject to the limitations in paragraph 4(a), below;
52+
v. rights protecting the extraction, dissemination, use and reuse of data
53+
in a Work;
54+
vi. database rights (such as those arising under Directive 96/9/EC of the
55+
European Parliament and of the Council of 11 March 1996 on the legal
56+
protection of databases, and under any national implementation
57+
thereof, including any amended or successor version of such
58+
directive); and
59+
vii. other similar, equivalent or corresponding rights throughout the
60+
world based on applicable law or treaty, and any national
61+
implementations thereof.
62+
63+
2. Waiver. To the greatest extent permitted by, but not in contravention
64+
of, applicable law, Affirmer hereby overtly, fully, permanently,
65+
irrevocably and unconditionally waives, abandons, and surrenders all of
66+
Affirmer's Copyright and Related Rights and associated claims and causes
67+
of action, whether now known or unknown (including existing as well as
68+
future claims and causes of action), in the Work (i) in all territories
69+
worldwide, (ii) for the maximum duration provided by applicable law or
70+
treaty (including future time extensions), (iii) in any current or future
71+
medium and for any number of copies, and (iv) for any purpose whatsoever,
72+
including without limitation commercial, advertising or promotional
73+
purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
74+
member of the public at large and to the detriment of Affirmer's heirs and
75+
successors, fully intending that such Waiver shall not be subject to
76+
revocation, rescission, cancellation, termination, or any other legal or
77+
equitable action to disrupt the quiet enjoyment of the Work by the public
78+
as contemplated by Affirmer's express Statement of Purpose.
79+
80+
3. Public License Fallback. Should any part of the Waiver for any reason
81+
be judged legally invalid or ineffective under applicable law, then the
82+
Waiver shall be preserved to the maximum extent permitted taking into
83+
account Affirmer's express Statement of Purpose. In addition, to the
84+
extent the Waiver is so judged Affirmer hereby grants to each affected
85+
person a royalty-free, non transferable, non sublicensable, non exclusive,
86+
irrevocable and unconditional license to exercise Affirmer's Copyright and
87+
Related Rights in the Work (i) in all territories worldwide, (ii) for the
88+
maximum duration provided by applicable law or treaty (including future
89+
time extensions), (iii) in any current or future medium and for any number
90+
of copies, and (iv) for any purpose whatsoever, including without
91+
limitation commercial, advertising or promotional purposes (the
92+
"License"). The License shall be deemed effective as of the date CC0 was
93+
applied by Affirmer to the Work. Should any part of the License for any
94+
reason be judged legally invalid or ineffective under applicable law, such
95+
partial invalidity or ineffectiveness shall not invalidate the remainder
96+
of the License, and in such case Affirmer hereby affirms that he or she
97+
will not (i) exercise any of his or her remaining Copyright and Related
98+
Rights in the Work or (ii) assert any associated claims and causes of
99+
action with respect to the Work, in either case contrary to Affirmer's
100+
express Statement of Purpose.
101+
102+
4. Limitations and Disclaimers.
103+
104+
a. No trademark or patent rights held by Affirmer are waived, abandoned,
105+
surrendered, licensed or otherwise affected by this document.
106+
b. Affirmer offers the Work as-is and makes no representations or
107+
warranties of any kind concerning the Work, express, implied,
108+
statutory or otherwise, including without limitation warranties of
109+
title, merchantability, fitness for a particular purpose, non
110+
infringement, or the absence of latent or other defects, accuracy, or
111+
the present or absence of errors, whether or not discoverable, all to
112+
the greatest extent permissible under applicable law.
113+
c. Affirmer disclaims responsibility for clearing rights of other persons
114+
that may apply to the Work or any use thereof, including without
115+
limitation any person's Copyright and Related Rights in the Work.
116+
Further, Affirmer disclaims responsibility for obtaining any necessary
117+
consents, permissions or other rights required for any use of the
118+
Work.
119+
d. Affirmer understands and acknowledges that Creative Commons is not a
120+
party to this document and has no duty or obligation with respect to
121+
this CC0 or use of the Work.

‎deps/hdr_histogram/LICENSE.txt‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
The code in this repository code was Written by Gil Tene, Michael Barker,
2+
and Matt Warren, and released to the public domain, as explained at
3+
http://creativecommons.org/publicdomain/zero/1.0/
4+
5+
For users of this code who wish to consume it under the "BSD" license
6+
rather than under the public domain or CC0 contribution text mentioned
7+
above, the code found under this directory is *also* provided under the
8+
following license (commonly referred to as the BSD 2-Clause License). This
9+
license does not detract from the above stated release of the code into
10+
the public domain, and simply represents an additional license granted by
11+
the Author.
12+
13+
-----------------------------------------------------------------------------
14+
** Beginning of "BSD 2-Clause License" text. **
15+
16+
Copyright (c) 2012, 2013, 2014 Gil Tene
17+
Copyright (c) 2014 Michael Barker
18+
Copyright (c) 2014 Matt Warren
19+
All rights reserved.
20+
21+
Redistribution and use in source and binary forms, with or without
22+
modification, are permitted provided that the following conditions are met:
23+
24+
1. Redistributions of source code must retain the above copyright notice,
25+
this list of conditions and the following disclaimer.
26+
27+
2. Redistributions in binary form must reproduce the above copyright notice,
28+
this list of conditions and the following disclaimer in the documentation
29+
and/or other materials provided with the distribution.
30+
31+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
32+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
33+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
34+
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
35+
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
36+
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
37+
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
38+
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
39+
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
40+
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
41+
THE POSSIBILITY OF SUCH DAMAGE.

‎deps/hdr_histogram/README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
HdrHistogram_c v0.11.0
2+
3+
----------------------------------------------
4+
5+
This port contains a subset of the 'C' version of High Dynamic Range (HDR) Histogram available at [github.com/HdrHistogram/HdrHistogram_c](https://github.com/HdrHistogram/HdrHistogram_c).
6+
7+
8+
The code present on `hdr_histogram.c`, `hdr_histogram.h`, and `hdr_atomic.c` was Written by Gil Tene, Michael Barker,
9+
and Matt Warren, and released to the public domain, as explained at
10+
http://creativecommons.org/publicdomain/zero/1.0/.

0 commit comments

Comments
 (0)