Generate Unique ID (uuid) and pass across all remap transforms (parse_regex & parse_json)

[kbitra]

Hi
I want to generate an an uid in the initial transform and have that available across all transforms. I was able to accomplish this if all transforms uses "parse_regex" but when I combine parse_regex and parse_json the uuid is coming as null..

**When I print the transform to console.. this transform "read_log_uid_s" shows
id=78209ab9-c5ec-4a1c-8289-d3992e6ad633

but when I print the transform "read_log_process_log_s" it show
id=null**

Here is my toml
`
#####read logs from file ########
[sources.read_log]
type = "file"
include = ["/home/logs/*.log"]
read_from = "end"

####ADD uuid<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
[transforms.read_log_uid]
type = "remap"
inputs = ["read_log"]
source ='''
. |= parse_regex!(.message, r'(.+)(?P{.+)')
.sourcedata = .file
.id=uuid_v4()
'''
####<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

####<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
[transforms.read_log_process_log]
type = "remap"
drop_on_error = true
inputs = ["read_log_uid"]
source = '''. = parse_json!(.rawlogmessage)
.sourcedata = .sourcedata
.sourcetype = "file"
.id=.id
'''
####<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

####<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
[sinks.read_log_uid_s]
type = "console"
inputs = ["read_log_uid"]
encoding.codec = "json"

[sinks.read_log_process_log_s]
type = "console"
inputs = ["read_log_process_log"]
encoding.codec = "json"
`

[jszwedko]

Hi @kbitra !

It looks like the read_log_process_log transform is overwriting the event with . = parse_json!(...). This will replace any fields on the event. Instead, you could consider merging the fields like: . = merge(., object!(parse_json!(. rawlogmessage))).

[kbitra]

Awesome.. super thanks.. it worked