Skip to content

Commit 75ae214

Browse files
author
Frederick Ross
committed
Merge branch 'develop' into fross/review-fixes
2 parents d8e4c96 + 44dab31 commit 75ae214

File tree

3 files changed

+276
-840
lines changed

3 files changed

+276
-840
lines changed

examples/export/README.md

Lines changed: 0 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -29,28 +29,5 @@ the most friendly/intuitive format.
2929

3030
## Notes
3131

32-
* The "time chunking" algorithm tries to put as many events, up to the
33-
limit specified in a "bucket". We start out by breaking the index into buckets
34-
of 86400 seconds, or one day. If the number of events in this bucket is more
35-
than our limit, we split the day into 24 buckets of one hour each. If any of the
36-
hour buckets contain more events than our limit, the hour is split into 60
37-
buckets of one minute each. If any of the minute buckets contain more events
38-
than our limit, the minute is split into 60 buckets of one second each. A second
39-
bucket is the smallest granular size.
40-
41-
The code has a downsample map:
42-
43-
{ 86400 : 3600, 3600 : 60, 60 : 1 }
44-
45-
This maps the current "bucket length in seconds" to "next bucket length in
46-
seconds" if the current bucket contains more events than our limit.
47-
48-
As such, it is important that the initial starttime begins on a day boundary
49-
(i.e. 12:00:00 AM).
50-
51-
* The goal of export.py is NOT to optimize the number of requests to splunk,
52-
rather to optimize the size of the return request from splunk so that in the
53-
cases of very large indices, robustness and restart are paramount.
54-
5532
* When using csv or json output formats, sideband messages are not included. If
5633
you wish to capture sideband messages, the xml format should be used.

0 commit comments

Comments
 (0)