|
1 | 1 | # Disk buffering |
2 | 2 |
|
3 | | -This module provides exporters that store telemetry data in files which can be |
4 | | -sent later on demand. A high level description of how it works is that there are two separate |
5 | | -processes in place, one for writing data in disk, and one for reading/exporting the previously |
6 | | -stored data. |
| 3 | +This module provides an abstraction |
| 4 | +named [SignalStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/SignalStorage.java), |
| 5 | +as well as default implementations for each signal type that allow writing signals to disk and |
| 6 | +reading them later. |
7 | 7 |
|
8 | | -* Each exporter stores the received data automatically in disk right after it's received from its |
9 | | - processor. |
10 | | -* The reading of the data back from disk and exporting process has to be done manually. At |
11 | | - the moment there's no automatic mechanism to do so. There's more information on how it can be |
12 | | - achieved, under [Reading data](#reading-data). |
| 8 | +For a more detailed information on how the whole process works, take a look at |
| 9 | +the [DESIGN.md](DESIGN.md) file. |
13 | 10 |
|
14 | | -> For a more detailed information on how the whole process works, take a look at |
15 | | -> the [DESIGN.md](DESIGN.md) file. |
| 11 | +## Default implementation usage |
16 | 12 |
|
17 | | -## Configuration |
| 13 | +The default implementations are the following: |
18 | 14 |
|
19 | | -The configurable parameters are provided **per exporter**, the available ones are: |
| 15 | +* [FileSpanStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileSpanStorage.java) |
| 16 | +* [FileLogRecordStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileLogRecordStorage.java) |
| 17 | +* [FileMetricStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileMetricStorage.java) |
20 | 18 |
|
21 | | -* Max file size, defaults to 1MB. |
22 | | -* Max folder size, defaults to 10MB. All files are stored in a single folder per-signal, therefore |
23 | | - if all 3 types of signals are stored, the total amount of space from disk to be taken by default |
24 | | - would be of 30MB. |
25 | | -* Max age for file writing, defaults to 30 seconds. |
26 | | -* Min age for file reading, defaults to 33 seconds. It must be greater that the max age for file |
27 | | - writing. |
28 | | -* Max age for file reading, defaults to 18 hours. After that time passes, the file will be |
29 | | - considered stale and will be removed when new files are created. No more data will be read from a |
30 | | - file past this time. |
31 | | - |
32 | | -## Usage |
| 19 | +### Set up |
33 | 20 |
|
34 | | -### Storing data |
| 21 | +We need to create a signal storage object per signal type to start writing signals to disk. Each |
| 22 | +`File*Storage` implementation has a `create()` function that receives: |
| 23 | + |
| 24 | +* A File directory to store the signal files. Note that each signal storage object must have a |
| 25 | + dedicated directory to work properly. |
| 26 | +* (Optional) a configuration object. |
35 | 27 |
|
36 | | -In order to use it, you need to wrap your own exporter with a new instance of |
37 | | -the ones provided in here: |
| 28 | +The available configuration parameters are the following: |
38 | 29 |
|
39 | | -* For a LogRecordExporter, it must be wrapped within |
40 | | - a [LogRecordToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/LogRecordToDiskExporter.java). |
41 | | -* For a MetricExporter, it must be wrapped within |
42 | | - a [MetricToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/MetricToDiskExporter.java). |
43 | | -* For a SpanExporter, it must be wrapped within |
44 | | - a [SpanToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/SpanToDiskExporter.java). |
| 30 | +* Max file size, defaults to 1MB. |
| 31 | +* Max folder size, defaults to 10MB. |
| 32 | +* Max age for file writing. It sets the time window where a file can get signals appended to it. |
| 33 | + Defaults to 30 seconds. |
| 34 | +* Min age for file reading. It sets the time to wait before starting to read from a file after |
| 35 | + its creation. Defaults to 33 seconds. It must be greater that the max age for file writing. |
| 36 | +* Max age for file reading. After that time passes, the file will be considered stale and will be |
| 37 | + removed when new files are created. No more data will be read from a file past this time. Defaults |
| 38 | + to 18 hours. |
45 | 39 |
|
46 | | -Each wrapper will need the following when instantiating them: |
| 40 | +```java |
| 41 | +// Root dir |
| 42 | +File rootDir = new File("/some/root"); |
47 | 43 |
|
48 | | -* The exporter to be wrapped. |
49 | | -* A File instance of the root directory where all the data is going to be written. The same root dir |
50 | | - can be used for all the wrappers, since each will create their own folder inside it. |
51 | | -* An instance |
52 | | - of [StorageConfiguration](src/main/java/io/opentelemetry/contrib/disk/buffering/config/StorageConfiguration.java) |
53 | | - with the desired parameters. You can create one with default values by |
54 | | - calling `StorageConfiguration.getDefault()`. |
| 44 | +// Setting up span storage |
| 45 | +SignalStorage.Span spanStorage = FileSpanStorage.create(new File(rootDir, "spans")); |
55 | 46 |
|
56 | | -After wrapping your exporters, you must register the wrapper as the exporter you'll use. It will |
57 | | -take care of always storing the data it receives. |
| 47 | +// Setting up metric storage |
| 48 | +SignalStorage.Metric metricStorage = FileMetricStorage.create(new File(rootDir, "metrics")); |
58 | 49 |
|
59 | | -#### Set up example for spans |
| 50 | +// Setting up log storage |
| 51 | +SignalStorage.LogRecord logStorage = FileLogRecordStorage.create(new File(rootDir, "logs")); |
| 52 | +``` |
60 | 53 |
|
61 | | -### Writing data |
| 54 | +### Storing data |
62 | 55 |
|
63 | | -The data is written in the disk by "ToDisk" exporters, these are exporters that serialize and store the data as received by their processors. If for some reason |
64 | | -the "ToDisk" cannot store data in the disk, they'll delegate the data to their wrapped exporter. |
| 56 | +While you could manually call your `SignalStorage.write(items)` function, disk buffering |
| 57 | +provides convenience exporters that you can use in your OpenTelemetry's instance, so |
| 58 | +that all signals are automatically stored as they are created. |
65 | 59 |
|
66 | | -```java |
67 | | -// Creating the SpanExporter of our choice. |
68 | | -SpanExporter mySpanExporter = OtlpGrpcSpanExporter.getDefault(); |
| 60 | +* For a span storage, use |
| 61 | + a [SpanToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/SpanToDiskExporter.java). |
| 62 | +* For a log storage, use |
| 63 | + a [LogRecordToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/LogRecordToDiskExporter.java). |
| 64 | +* For a metric storage, use |
| 65 | + a [MetricToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/MetricToDiskExporter.java). |
69 | 66 |
|
70 | | -// Wrapping our exporter with its "ToDisk" exporter. |
71 | | -SpanToDiskExporter toDiskExporter = SpanToDiskExporter.create(mySpanExporter, StorageConfiguration.getDefault(new File("/my/signals/cache/dir"))); |
| 67 | +Each will wrap a signal storage for its respective signal type, as well as an optional callback |
| 68 | +to notify when it succeeds, fails, and gets shutdown. |
72 | 69 |
|
73 | | - // Registering the disk exporter within our OpenTelemetry instance. |
74 | | -SdkTracerProvider myTraceProvider = SdkTracerProvider.builder() |
75 | | - .addSpanProcessor(SimpleSpanProcessor.create(toDiskExporter)) |
| 70 | +```java |
| 71 | +// Setting up span to disk exporter |
| 72 | +SpanToDiskExporter spanToDiskExporter = |
| 73 | + SpanToDiskExporter.builder(spanStorage).setExporterCallback(spanCallback).build(); |
| 74 | +// Setting up metric to disk |
| 75 | +MetricToDiskExporter metricToDiskExporter = |
| 76 | + MetricToDiskExporter.builder(metricStorage).setExporterCallback(metricCallback).build(); |
| 77 | +// Setting up log to disk exporter |
| 78 | +LogRecordToDiskExporter logToDiskExporter = |
| 79 | + LogRecordToDiskExporter.builder(logStorage).setExporterCallback(logCallback).build(); |
| 80 | + |
| 81 | +// Using exporters in your OpenTelemetry instance. |
| 82 | +OpenTelemetry openTelemetry = |
| 83 | + OpenTelemetrySdk.builder() |
| 84 | + // Using span to disk exporter |
| 85 | + .setTracerProvider( |
| 86 | + SdkTracerProvider.builder() |
| 87 | + .addSpanProcessor(BatchSpanProcessor.builder(spanToDiskExporter).build()) |
| 88 | + .build()) |
| 89 | + // Using log to disk exporter |
| 90 | + .setLoggerProvider( |
| 91 | + SdkLoggerProvider.builder() |
| 92 | + .addLogRecordProcessor( |
| 93 | + BatchLogRecordProcessor.builder(logToDiskExporter).build()) |
| 94 | + .build()) |
| 95 | + // Using metric to disk exporter |
| 96 | + .setMeterProvider( |
| 97 | + SdkMeterProvider.builder() |
| 98 | + .registerMetricReader(PeriodicMetricReader.create(metricToDiskExporter)) |
| 99 | + .build()) |
76 | 100 | .build(); |
77 | | -OpenTelemetrySdk.builder() |
78 | | - .setTracerProvider(myTraceProvider) |
79 | | - .buildAndRegisterGlobal(); |
80 | | - |
81 | 101 | ``` |
82 | 102 |
|
| 103 | +Now when creating signals using your `OpenTelemetry` instance, those will get stored in disk. |
| 104 | + |
83 | 105 | ### Reading data |
84 | 106 |
|
85 | | -In order to read data, we need to create "FromDisk" exporters, which read data from the disk, parse it and delegate it |
86 | | -to their wrapped exporters. |
| 107 | +In order to read data, we can iterate through our signal storage objects and then forward them to |
| 108 | +a network exporter, as shown in the example for spans below. |
87 | 109 |
|
88 | 110 | ```java |
89 | | -try { |
90 | | - SpanFromDiskExporter fromDiskExporter = SpanFromDiskExporter.create(memorySpanExporter, storageConfig); |
91 | | - if(fromDiskExporter.exportStoredBatch(1, TimeUnit.SECONDS)) { |
92 | | - // A batch was successfully exported and removed from disk. You can call this method for as long as it keeps returning true. |
93 | | - } else { |
94 | | - // Either there was no data in the disk or the wrapped exporter returned CompletableResultCode.ofFailure(). |
95 | | - } |
96 | | -} catch (IOException e) { |
97 | | - // Something unexpected happened. |
| 111 | +// Example of reading an exporting spans from disk |
| 112 | +OtlpHttpSpanExporter networkExporter; |
| 113 | +Iterator<Collection<SpanData>> spanCollections = spanStorage.iterator(); |
| 114 | +while(spanCollections.hasNext()){ |
| 115 | + networkExporter.export(spanCollections.next()); |
98 | 116 | } |
99 | 117 | ``` |
100 | 118 |
|
| 119 | +The `File*Storage` iterators delete the previously returned collection when `next()` is called, |
| 120 | +assuming that if the next collection is requested is because the previous one was successfully |
| 121 | +consumed. |
| 122 | + |
101 | 123 | Both the writing and reading processes can run in parallel and they don't overlap |
102 | 124 | because each is supposed to happen in different files. We ensure that reader and writer don't |
103 | | -accidentally meet in the same file by using the configurable parameters. These parameters set non-overlapping time frames for each action to be done on a single file at a time. On top of that, there's a mechanism in |
104 | | -place to avoid overlapping on edge cases where the time frames ended but the resources haven't been |
105 | | -released. For that mechanism to work properly, this tool assumes that both the reading and the |
106 | | -writing actions are executed within the same application process. |
| 125 | +accidentally meet in the same file by using the configurable parameters. These parameters set |
| 126 | +non-overlapping time frames for each action to be done on a single file at a time. On top of that, |
| 127 | +there's a mechanism in place to avoid overlapping on edge cases where the time frames ended but the |
| 128 | +resources haven't been released. For that mechanism to work properly, this tool assumes that both |
| 129 | +the reading and the writing actions are executed within the same application process. |
107 | 130 |
|
108 | 131 | ## Component owners |
109 | 132 |
|
|
![otelbot[bot]](https://avatars.githubusercontent.com/in/1130599?v=4&size=40){kind=avatar}
0 commit comments