You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/design/4940-reliable-loki-pipelines.md
+8-12Lines changed: 8 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -32,15 +32,15 @@ loki.process "logs" {
32
32
loki.write "loki" {}
33
33
```
34
34
35
-
`loki.source.file` will tail all files from targets and compete to send on the channel exposed by `loki.process`. Only one entry will be processed by each stage configured in `loki.process`. If a reload happens or if alloy is shutting down logs could be silently dropped.
35
+
`loki.source.file` will tail all files from targets and compete to send on the channel exposed by `loki.process`. Only one entry will be processed by each stage configured in `loki.process`. If a reload happens or if Alloy is shutting down, logs could be silently dropped.
36
36
37
37
## Proposal 0: Do nothing
38
38
This architecture works in most cases, it will be hard to use slow components such as `secretfilter` because a lot of the time it's too slow.
39
39
It's also hard to use Alloy as a gateway for loki pipelines with e.g. `loki.source.api` due to the limitations listed above.
40
40
41
41
## Proposal 1: Chain function calls
42
42
43
-
Loki pipelines are the only one using channels for passing data between components. Prometheus, Pyroscope and otelcol are all using this pattern where each component just calls functions on the next.
43
+
Loki pipelines are the only ones using channels for passing data between components. Prometheus, Pyroscope and otelcol are all using this pattern where each component just calls functions on the next.
44
44
45
45
They all have slightly different interfaces but basically work the same. Each component exports its own interface like Appender for Prometheus or Consumer for Otel.
46
46
@@ -52,7 +52,7 @@ type Consumer interface {
52
52
}
53
53
```
54
54
55
-
Adopting this pattern for loki pipelines would change it from a channelbased pipeline to a functionbased pipeline. This would give us two things:
55
+
Adopting this pattern for loki pipelines would change it from a channel-based pipeline to a function-based pipeline. This would give us two things:
56
56
1. Increased throughput because several sources such as many files or http requests can now call the next component in the pipeline at the same time.
57
57
2. A way to return signals back to the source so we can handle things like giving a proper error response or determine if the position file should be updated.
58
58
@@ -65,7 +65,7 @@ Pros:
65
65
* A way to signal back to the source
66
66
Cons:
67
67
* We need to rewrite all loki components with this new pattern and make them safe to call in parallel.
68
-
* We go from an iteratorlike pipeline to passing slices around. Every component would have to iterate over this slice and we need to make sure it's safe to mutate because of fanout.
68
+
* We go from an iterator-like pipeline to passing slices around. Every component would have to iterate over this slice and we need to make sure it's safe to mutate because of fan-out.
69
69
70
70
## Proposal 2: Appendable
71
71
@@ -88,14 +88,12 @@ type Appender interface {
88
88
This approach would, like Proposal 1, solve the issues listed above with a function-based pipeline, but the pipeline would still be iterator-like (one entry at a time).
89
89
90
90
### How it works
91
-
92
-
Sources would obtain an `Appender` from the next component in the pipeline, then:
93
-
1. Call `Append(entry)` for each entry in a batch
94
-
2. Call `Commit()` to finalize the batch, or `Rollback()` to discard it
95
-
3. Handle errors at any step
91
+
Source components would:
92
+
Obtain an `Appender` that can fan-out to all downstream components, then call `Append` for each entry.
93
+
If every call to `Append` is successful, `Commit` should be called; otherwise `Rollback`.
96
94
97
95
Processing components would:
98
-
Implement `Appendable` to return an `Appender` that runs processing for each entry and forwards it to next component.
96
+
Implement `Appendable` to return an `Appender` that runs processing for each entry and fan-out to all downstream components.
99
97
100
98
Sink components would:
101
99
Implement `Appendable` to return an `Appender` that buffers entries until either `Commit` or `Rollback` is called.
@@ -153,5 +151,3 @@ The following components need to be updated with this new interface and we need
153
151
**Sink components** (need to implement `Consumer`):
0 commit comments