Skip to content

Commit b79877f

Browse files
committed
Add section about transition to new pipeline code
1 parent e209358 commit b79877f

File tree

1 file changed

+14
-0
lines changed

1 file changed

+14
-0
lines changed

docs/design/4940-reliable-loki-pipelines.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,17 @@ Two options to handle this:
119119
* When using `loki.source.file`, we would retry the same batch again.
120120
* Configuration option `min_success` - Only retry if we don't succeed on at least the configured number of destinations.
121121

122+
### Transition from current pipeline to either Proposal 1 or Proposal 2
123+
Changing the way loki pipeline works is a big effort and will affect all loki components.
124+
125+
We have a couple of options how to do this:
126+
1. Build tag
127+
* We build out the new pipeline under a build tag. This way we could build custom Alloy image using this new pipeline and test it out internally before we commit it to an official release.
128+
2. New argument
129+
* We could add additional argument to components in addition to `forward_to`. This new argument would be using the new pipeline code. This argument would be protected by experimental flag and we would remove it once we are confident in the new code and remove the current pipeline.
130+
3. Replace pipeline directly
131+
* We could replace the pipeline directly without any fallback mechanism. This should be doable over several PRs where we first only replace the communication between components, e.g. in loki.source.file we would still have the [main loop](https://github.com/grafana/alloy/blob/main/internal/component/loki/source/file/file.go#L229-L247) reading from channel and send one entry at a time with this new pipeline between components. Then we could work component by component and remove most of channel usage.
132+
122133
### Affected components
123134

124135
The following components need to be updated with this new interface and we need to make sure they are concurrency safe:
@@ -140,6 +151,9 @@ The following components need to be updated with this new interface and we need
140151
- `loki.source.azure_event_hubs`
141152
- `loki.source.aws_firehose`
142153
- `loki.source.windowsevent`
154+
- `database_observability.mysql`
155+
- `database_observability.postgres`
156+
- `faro.receiver`
143157

144158
**Processing components** (need to implement `Consumer` and forward to next):
145159
- `loki.process`

0 commit comments

Comments
 (0)