vector 日志分词器

[ZbZn905888]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

现在有如下日志数据：
{
_timestamp:1709261275287183,
file:/data1/kubernetes/logs/im/zero-user-wx/zero-user-wx-base-7d6cb69f56-nsrrb/app.log,
host:vector-wt6vv,
message:{"@timestamp":"2024-03-01T10:47:54.689+08:00","caller":"load/sheddingstat.go:61","content":"(api) shedding_stat [1m], cpu: 0, total: 40, pass: 40, drop: 0","level":"stat"},
offset:518386817,
source_type:file,
timestamp:2024-03-01T02:47:54.818238220Z
}
现在有如下需求：
请问在vector配置中如何采用分词功能，提取file字段中：im为kubernetes_pod_namespace字段的值，zero-user-wx为kubernetes_pod_labels_deploy的值，zero-user-wx-base-7d6cb69f56-nsrrb为kubernetes_pod_name的值。即实现如下效果：
{
_timestamp:1709261275287183,
file:/data1/kubernetes/logs/im/zero-user-wx/zero-user-wx-base-7d6cb69f56-nsrrb/app.log,
kubernetes_pod_name: im,
kubernetes_pod_labels_deploy: zero-user-wx,
kubernetes_pod_name: zero-user-wx-base-7d6cb69f56-nsrrb,
host:vector-wt6vv,
message:{"@timestamp":"2024-03-01T10:47:54.689+08:00","caller":"load/sheddingstat.go:61","content":"(api) shedding_stat [1m], cpu: 0, total: 40, pass: 40, drop: 0","level":"stat"},
offset:518386817,
source_type:file,
timestamp:2024-03-01T02:47:54.818238220Z
}
其中,vector 0.35版本不支持regex_parser插件

Attempted Solutions

No response

Proposal

No response

References

No response

Version

No response

[jszwedko]

If the input is JSON, which it kind of looks like except the keys aren't quoted, you can use https://vector.dev/docs/reference/vrl/functions/#parse_json in a remap transform. If the input is not JSON then https://vector.dev/docs/reference/vrl/functions/#parse_key_value is probably your best-bet. The regex_parser transform was replaced by https://vector.dev/docs/reference/vrl/functions/#parse_regex