How to fuzzy matching？

[marksugar]

Let's say my data is like this

{
"body":"2023-08-02T00:18:34.866228161+08:00 stdout F 2023-08-02 00:18:34.865 INFO 7 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer WebApplicationContext : Tomcat started on port(s): 8080 (http) with context path '..."
"fields":{
"containername":"java-demo"
"environment":"dev"
"logconfig":"java-demo"
"namespace":"linuxea-dev"
"nodename":"172.16.100.83"
"podname":"production-java-demo-5cf5b97645-tsmxx"
"topic":"java-demo"
}
}

file

[api]
enabled = true
address = "0.0.0.0:8686"

[sources.my_source_id]
type = "kafka"
bootstrap_servers = ":9092"
group_id = "consumer-group-name"
topics = [ "pod-dev-java-demo" ]
[sources.my_source_id.decoding]
codec = "json"


[transforms.ftest]
type = "filter"
inputs = ["my_source_id"]
condition = '.body == "*WebApplicationContext*"'


[transforms.remap_alert_udev]
type = "remap"
inputs = ["ftest"]
source = '''
. = {
    "startsAt": .happen_time,
    "labels": {
        "hostname": .hostname,
        "device": .data.pass_through_name,
        "created_time": .timestamp,
        "event": downcase!(.name),
        "service": "udev"
    }
}
if (.labels.event == "add_device") {
    .labels.alertname = "event_add_device"
    .labels.severity = "info"
} else if (.labels.event == "remove_device") {
    .labels.alertname = "event_remove_device"
    .labels.severity = "critical"
}
'''

[sinks.sink0]
inputs = ["remap_alert_*"]
target = "stdout"
type = "console"
[sinks.sink0.encoding]
codec = "json"

I'm not going to configure logs as json
only want to match everything that contains "WebApplicationContext" in the body
how should I fuzzy matches to get everything that contains the "WebApplicationContext" keyword
Can you correct my problem

thanks

[jszwedko]

Hi @marksugar ,

I'm not sure I completely understand what you are looking for, but it seems like regular expression matching would be applicable. You can use the https://vector.dev/docs/reference/vrl/functions/#match for that in VRL.

[marksugar]

hello @jszwedko

It took me a day and I didn't get through the whole process

What I've always wanted to accomplish is sending matching log passages to dingtalk's webhooks, #4568 when I searched for past issues about webhooks

Then it is intended to be sent to the alertmanager first
But I didn't find a direct example, I'm going to describe my situation again, I hope your help

[api]
enabled = true
address = "0.0.0.0:8686"

[sources.filetest]
type = "file"
include = ["/var/log/test.log"]

[transforms.ftest]
type = "filter"
inputs = ["filetest"]
condition = '.body == "*WebApplicationContext*"'


[transforms.remap_alert_udev]
type = "remap"
inputs = ["ftest"]
source = '''
. = {
    "labels": {
        "alertname": .fields.podname,      
        "namespace": .fields.namespace,
        "environment": .fields.environment,
        "podname": .fields.podname,
        "nodename": .fields.nodename,
        "topic": ".fields.topic",
        "body": .body,
        "severity": "critical"
    }
}
'''

[sinks.alertmanager]
type = "http"
inputs = ["remap_alert_*"]
uri = "http://alertmanager:9093/api/v2/alerts"
compression = "none"
acknowledgements.enabled = true

[sinks.alertmanager.encoding]
codec = "json"

Add data manually:

echo '{"body":"2023-08-02T00:18:34.866228161+08:00 stdouts.b.w.embedded.tomcat.TomcatWebServer WebApplicationContext","fields":{"containername":"java-demo","environment":"dev","logconfig":"java-demo","namespace":"linuxea-dev","nodename":"172.16.100.83","podname":"production-java-demo-5cf5b97645-tsmxx","topic":"java-demo"}}' >> /var/log/test.log

VECTOR_LOG=debug vector -c verctor.toml

2023-08-02T04:28:14.757985Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2023-08-02T04:28:15.759249Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2023-08-02T04:28:15.853518Z DEBUG source{component_kind="source" component_id=filetest component_type=file component_name=filetest}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 2.1285994e-5, "other": 4.3755623e-5, "reading": 9.693164e-7, "sending": 2.619346e-5, "sleeping": 0.9999079}
2023-08-02T04:28:16.743668Z DEBUG sink{component_kind="sink" component_id=alertmanager component_type=http component_name=alertmanager}: vector::utilization: utilization=0.0010833808543255549
2023-08-02T04:28:16.759865Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2023-08-02T04:28:17.761182Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2023-08-02T04:28:17.903483Z DEBUG source{component_kind="source" component_id=filetest component_type=file component_name=filetest}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 2.0888716e-5, "other": 4.0019782e-5, "reading": 6.4652846e-7, "sending": 1.9845822e-5, "sleeping": 0.99991864}
2023-08-02T04:28:18.761693Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2023-08-02T04:28:19.763176Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2023-08-02T04:28:19.952609Z DEBUG source{component_kind="source" component_id=filetest component_type=file component_name=filetest}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 2.1275971e-5, "other": 3.6337253e-5, "reading": 4.8506547e-7, "sending": 1.611563e-5, "sleeping": 0.99992585}
2023-08-02T04:28:20.763784Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2023-08-02T04:28:21.744070Z DEBUG sink{component_kind="sink" component_id=alertmanager component_type=http component_name=alertmanager}: vector::utilization: utilization=0.00010837228260045288
2023-08-02T04:28:21.765184Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2023-08-02T04:28:22.001606Z DEBUG source{component_kind="source" component_id=filetest component_type=file component_name=filetest}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 2.2051821e-5, "other": 3.5453322e-5, "reading": 3.8813843e-7, "sending": 1.4162175e-5, "sleeping": 0.999928}
2023-08-02T04:28:22.001912Z DEBUG transform{component_kind="transform" component_id=ftest component_type=filter component_name=ftest}: vector::utilization: utilization=0.0017274864549245983
2023-08-02T04:28:22.001982Z DEBUG transform{component_kind="transform" component_id=ftest component_type=filter component_name=ftest}: vector_common::internal_event::component_events_dropped: Events dropped intentional=true count=1 reason="Events matched filter condition." internal_log_rate_limit=true
2023-08-02T04:28:22.002004Z DEBUG transform{component_kind="transform" component_id=ftest component_type=filter component_name=ftest}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.
2023-08-02T04:28:22.768203Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=2
2023-08-02T04:28:23.033690Z DEBUG source{component_kind="source" component_id=filetest component_type=file component_name=filetest}:file_server: file_source::file_server: event_throughput=5.000/sec bytes_throughput=1.590k/sec ratios={"discovery": 4.931193e-5, "other": 6.713035e-5, "reading": 5.2721738e-5, "sending": 7.048298e-5, "sleeping": 0.99976087}
2023-08-02T04:28:23.768870Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2023-08-02T04:28:24.059437Z DEBUG source{component_kind="source" component_id=filetest component_type=file component_name=filetest}:file_server: file_source::file_server: event_throughput=2.000/sec bytes_throughput=795.000/sec ratios={"discovery": 4.9981783e-5, "other": 8.547604e-5, "reading": 2.8059503e-5, "sending": 4.0806313e-5, "sleeping": 0.999796}
2023-08-02T04:28:24.770558Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2023-08-02T04:28:25.771895Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2023-08-02T04:28:26.109425Z DEBUG source{component_kind="source" component_id=filetest component_type=file component_name=filetest}:file_server: file_source::file_server: event_throughput=1.000/sec bytes_throughput=397.000/sec ratios={"discovery": 3.6414247e-5, "other": 5.8929923e-5, "reading": 1.4793949e-5, "sending": 2.3433695e-5, "sleeping": 0.9998665}

Thank you so much

[jszwedko]

Hi @marksugar !

What you have:

condition = '.body == "*WebApplicationContext*"'

Will match body against the literal string "*WebApplicationContext*". I think what you want is:

condition = "match!(.body, r`.*WebApplicationContext.*')"

To match body against a regular expression.

[marksugar]

Hi @marksugar !

What you have:

condition = '.body == "*WebApplicationContext*"'

Will match body against the literal string "*WebApplicationContext*". I think what you want is:

condition = "match!(.body, r`.*WebApplicationContext.*')"

To match body against a regular expression.