I use vector+kafka+vector+loki to write the service log to cloki, but there are some problems

[ktpktr0]

Use vector1 to collect file logs to Kafka; Use Vector2 to send logs from Kafka to cloki. But I have some questions:

1. When I have multiple Vector2, can I set the number of partitions consumed by each vector, such as the consumer of logstash_ Same configuration as threads

2. I don't know which step is better to use transform to parse logs on vector1 or Vector2

3. I see that transform can use VRL and grok at the same time. Are there any performance differences between them

4. It is recommended that VRL provide more log examples of programming languages or services when parsing logs, such as Java, golang, nginx, redis, /var/log, and so on

[jszwedko]

Hi @ktpktr0 !

Use vector1 to collect file logs to Kafka; Use Vector2 to send logs from Kafka to cloki. But I have some questions:

1. When I have multiple Vector2, can I set the number of partitions consumed by each vector, such as the consumer of logstash_ Same configuration as threads

My understanding is that the Kafka broker determines which partitions are assigned to which consumers. It should fairly distribute them so if you have, say, 8 partitions and 2 Vector instances consuming, each Vector instance will consume 4 partitions.

You can find many available Kafka configuration options on https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md which maps to librdkafka_options on the kafka source which should hopefully support any Kafka consumer options you wish to tweak.

2. I don't know which step is better to use transform to parse logs on vector1 or Vector2

Either one would work, but it depends on several trade-offs:

• Do the hosts vector1 is running on have sufficient resources to do the parsing? I'm guessing the vector2 instances are probably easier to scale out horizontally as needed
• Do you want to reject data if it cannot parse before you even put it into Kafka?

In general, we recommend doing most of the parsing and aggregation in the "aggregator" layer of this sort of architecture; in your case vector2.

3. I see that transform can use VRL and grok at the same time. Are there any performance differences between them

I'm not sure I completely understand this question. You can do grok parsing from VRL. We aim to have VRL be performant. Grok parsing is inherently slow since it is regex under the hood, but is usually sufficient for many use cases.

4. It is recommended that VRL provide more log examples of programming languages or services when parsing logs, such as Java, golang, nginx, redis, /var/log, and so on

VRL is the defacto way to parse logs in Vector. It offers many parsing functions including grok parsing and regex parsing.

Hopefully that helps! Let me know if you have any other questions.

[ktpktr0]

Thank you for your reply. I have tried VRL recently, but it never works properly, even if I directly copy the sample log and VRL configuration in the document. Therefore, it is recommended that VRL provide more log examples of programming languages or services when parsing logs, such as Java, golang, nginx, redis, /var/log, and so on.

I also tried parse_ Grok, unfortunately, its value needs to be filled in instead of directly parsing the log like logstash. So I still have to choose VRL as the tool to parse logs.

Whether to consider developing a tool similar to grokdebug for VRL log parsing test tool in the later stage.

[jszwedko]

Thanks @ktpktr0 ! The documentation should be accurate, so if you come across any examples that don't work when copy/pasting them, definitely let us know.

[ktpktr0]

I tried the simplest log transform, but it didn't work. It does not parse or store logs

Log content:

Jul 24 08:53:38 localhost rancher-system-agent: time="2022-07-24T08:53:38+08:00" level=info msg="Rancher System Agent version v0.2.7 (65aac83) is starting"
Jul 24 08:53:38 localhost rancher-system-agent: time="2022-07-24T08:53:38+08:00" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Jul 24 08:53:38 localhost rancher-system-agent: time="2022-07-24T08:53:38+08:00" level=info msg="Starting remote watch of plans"
Jul 24 08:53:39 localhost rancher-system-agent: panic: error while connecting to Kubernetes cluster: Get "https://192.168.10.8/version": dial tcp 192.168.10.8:443: connect: connection refused
Jul 24 08:53:39 localhost rancher-system-agent: goroutine 19 [running]:
Jul 24 08:53:39 localhost rancher-system-agent: github.com/rancher/system-agent/pkg/k8splan.(*watcher).start(0xc000194180, 0x17dfd80, 0xc0004130c0)
Jul 24 08:53:39 localhost rancher-system-agent: /go/src/github.com/rancher/system-agent/pkg/k8splan/watcher.go:97 +0xcb4
Jul 24 08:53:39 localhost rancher-system-agent: created by github.com/rancher/system-agent/pkg/k8splan.Watch
Jul 24 08:53:39 localhost rancher-system-agent: /go/src/github.com/rancher/system-agent/pkg/k8splan/watcher.go:58 +0xbe
Jul 24 08:53:39 localhost systemd: rancher-system-agent.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 24 08:53:39 localhost systemd: Unit rancher-system-agent.service entered failed state.
Jul 24 08:53:39 localhost systemd: rancher-system-agent.service failed.

vector config:

[sources.in]
type     = "file"
include  = ["/tmp/test.log"]
ignore_older = 86400


[transforms.transform]
type = "remap"
inputs = [ "in" ]
source = '''
  . |= parse_syslog!(.message)
'''

[sinks.out]
type = "loki"
inputs = [ "transform" ]
endpoint = "http://loki:3100"
compression = "none"

  [sinks.out.labels]
  agent = "vector"
  file = "{{ path }}"

  [sinks.out.encoding]
  codec = "json"

2022-07-24T00:57:55.524379Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,kube=info"
2022-07-24T00:57:55.524453Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2022-07-24T00:57:55.544181Z  INFO vector::topology::running: Running healthchecks.
2022-07-24T00:57:55.544728Z  INFO vector: Vector has started. debug="false" version="0.23.0" arch="x86_64" build_id="38c2435 2022-07-11"
2022-07-24T00:57:55.544743Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2022-07-24T00:57:55.544817Z  INFO source{component_kind="source" component_id=in component_type=file component_name=in}: vector::sources::file: Starting file server. include=["/tmp/test.log"] exclude=[]
2022-07-24T00:57:55.545305Z  INFO source{component_kind="source" component_id=in component_type=file component_name=in}:file_server: file_source::checkpointer: Attempting to read legacy checkpoint files.
2022-07-24T00:57:58.070191Z  INFO vector::topology::builder: Healthcheck: Passed.

[jszwedko]

Hi @ktpktr0 !

Is that all of the log output from Vector? It seems to be indicating that test.log doesn't exist.

Also, the parse_syslog expects the logs to be encoded as syslog messages according to RFC5424 or RFC3164. The log content you have appears to RFC3164 messages, but is missing the priority part of the message.