-
Help me pleaseArchitecture
Version
Original logs collected by kafka
My all config is as follows: [sources.dummy_logs] [transforms.parse_logs] [sinks.elasticsearch] |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 12 replies
-
|
Can you please share the full error logs, rather than the shortened lines included in your image? |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
I want to realize the same function as logstash, cut off a log according to the correct rule, and then save it in es, so that it can be filtered according to the data segment |
Beta Was this translation helpful? Give feedback.
-
|
It looks like you're getting unexpected content in your [transforms.parse_logs]
type = "remap"
inputs = ["dummy_logs"]
drop_on_error = true
source = '''
parsed, err = parse_regex(.message, r'^(?P\d+-\d+-\d+) (?P.*)$')
if err != null {
log(.message, level: "error")
abort
}
. = parsed
'''I'd also note your existing |
Beta Was this translation helpful? Give feedback.
-
I try to run it,error is follows: May 01 22:55:10 VM-20-8-centos vector[25197]: 2023-05-01T14:55:10.676019Z ERROR transform{component_kind="transform" component_id=parse_logs component_type=remap component_name=parse_logs}: vrl_stdlib::log: function call error for "parse_regex" at (17:79): could not find any pattern matches internal_log_rate_secs=1 vrl_position=104 |
Beta Was this translation helpful? Give feedback.
-
|
Apologies, copy paste mistake: [transforms.parse_logs]
type = "remap"
inputs = ["dummy_logs"]
drop_on_error = true
source = '''
parsed, err = parse_regex(.message, r'^(?P\d+-\d+-\d+) (?P.*)$')
if err != null {
log(.message, level: "error")
abort
}
. = parsed
''' |
Beta Was this translation helpful? Give feedback.
-
Log is as follows: |
Beta Was this translation helpful? Give feedback.
-
|
Well, looking at your output and testing locally - you haven't provided the configuration you're actually using. The configuration you provided in the original question is invalid: ❯ vector validate test.toml
√ Loaded ["test.toml"]
2023-05-01T15:17:48.995753Z WARN vector::internal_events::http_client: HTTP error. error=error trying to connect: tcp connect error: Connection refused (os error 61) error_type="request_failed" stage="processing" internal_log_rate_limit=true
2023-05-01T15:17:48.997449Z WARN vector::sinks::elasticsearch::common: Failed to determine Elasticsearch version from `/_cluster/state/version`. Please fix the reported error or set an API version explicitly via `api_version`. assumed_version=8 error=Failed to get Elasticsearch API version: Failed to make HTTP(S) request: error trying to connect: tcp connect error: Connection refused (os error 61)
2023-05-01T15:17:49.325795Z ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 1ms in state CONNECT)
2023-05-01T15:17:49.325862Z ERROR rdkafka::client: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
Component errors
----------------
x Transform "parse_logs":
error[E101]: invalid regular expression
┌─ :1:28
│
1 │ . = parse_regex!(.message, r'^(?P\d+-\d+-\d+) (?P.*)$')
│ ^^^^^^^^^^^^^^^^^^^^^^^^^^^ regex parse error: unrecognized flag: ^(?P\d+-\d+-\d+) (?P.*)$
│
= see documentation about regular expressions at https://vrl.dev/expressions/#regular-expression
= learn more about error code 101 at https://errors.vrl.dev/101
= see language documentation at https://vrl.dev
= try your code in the VRL REPL, learn more at https://vrl.dev/examplesPlease confirm what you're actually running and share the correct configuration for more assistance, thanks! |
Beta Was this translation helpful? Give feedback.
-
It is a problem with the github page, that character is not displayed，I provide my configuration
|
Beta Was this translation helpful? Give feedback.
-
|
You can provide your configurations in code blocks, by wrapping the contents with triple `s. parsed, err = parse_regex(.message, r'^(?P<time>\d+-\d+-\d+) (?P<haha>.*)$')
if err != null {
log(., level: "error")
abort
}
. = parsedIt seems likely your event is not structured as you expect. |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
[transforms.parse_logs]
|
Beta Was this translation helpful? Give feedback.
-
|
Because you used |
Beta Was this translation helpful? Give feedback.
-
Excuse me, how can I achieve the effect I want? |
Beta Was this translation helpful? Give feedback.
-
I solved it with parse_grok，my config is as follows There is one last problem that has not been solved, how to convert 2023-01-04 14:03:30.745 to 2023.01.04 |
Beta Was this translation helpful? Give feedback.
-
|
You can leverage format_timestamp(). |
Beta Was this translation helpful? Give feedback.
-
thank u very much, it is solved ,my all config is as follows
|
Beta Was this translation helpful? Give feedback.
thank u very much, it is solved ,my all config is as follows