Why Vector VRL remap can't del field of JSON file?

[epmpub]

test json data file - input.json:

{
    "file": "/var/log/logify/app666788eedd77.log",
    "host": "vector-test",
    "message": "{\"status\": 50e, \"ip\": \"127.0.0.1\", \"level\": 30, \"emailAddress\": \"[email protected]\", \"msg\": \"Task completed successfully\", \"pid\": 12655, \"ssn\": \"407-01-2433\", \"time\": 1694551048}",
    "source_type": "file",
    "timestamp": "2023-09-12T20:40:21.582883690Z"
}

my Vector toml configure file:

[sources.json_file]
type = "file"
include = ["c:\\config\\input.json"]
data_dir = "c:\\config"

[transforms.tf01]
type = "remap"
# drop_on_abort = true
timezone = "local"
inputs = ["json_file"]
source = """. , err = parse_json(.message)
    del(.ip)
    del(.status)
    .newID = "new IO"

"""


[sinks.out]
inputs = ["tf01"]
type = "console"
encoding.codec = "text"

Test Case command:

OS : windows 10

vector -c C:\config\test.toml -q

Screen output:

PS C:\config> vector -c C:\config\test.toml
2023-11-20T01:06:25.724598Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=info,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-11-20T01:06:25.726452Z  INFO vector::app: Loading configs. paths=["C:\\config\\test.toml"]
2023-11-20T01:06:25.730280Z  INFO vector::topology::running: Running healthchecks.2023-11-20T01:06:25.730627Z  INFO vector::topology::builder: Healthcheck passed.  
2023-11-20T01:06:25.730730Z  INFO source{component_kind="source" component_id=json_file component_type=file}: vector::sources::file: Starting file server. include=["c:\\config\\input.json"] exclude=[]
2023-11-20T01:06:25.730965Z  INFO vector: Vector has started. debug="false" version="0.34.0" arch="x86_64" revision="c909b66 2023-11-07 15:07:26.748571656"
2023-11-20T01:06:25.731485Z  INFO vector::app: API is disabled, enable by setting 
`api.enabled` to `true` and use commands like `vector top`.
2023-11-20T01:06:25.731782Z  INFO source{component_kind="source" component_id=json_file component_type=file}:file_server: file_source::checkpointer: Loaded checkpoint data.
2023-11-20T01:06:25.733116Z  INFO source{component_kind="source" component_id=json_file component_type=file}:file_server: vector::internal_events::file::source: Resuming to watch file. file=c:\config\input.json file_position=346
2023-11-20T01:06:38.226094Z  INFO source{component_kind="source" component_id=json_file component_type=file}:file_server: vector::internal_events::file::source: Found new file to watch. file=c:\config\input.json
{"status": 50e, "ip": "127.0.0.1", "level": 30, "emailAddress": "[email protected]", "msg": "Task completed successfully", "pid": 12655, "ssn": "407-01-2433", "time": 1694551048}
2023-11-20T01:06:46.559949Z  INFO source{component_kind="source" component_id=json_file component_type=file}:file_server: vector::internal_events::file::source: Stopped watching file. file=c:\config\input.json

why .ip and .status is not deleted.?

thanks.

Andy

[epmpub]

The vector version:

PS C:\Users\sheng> vector --version
vector 0.34.0 (x86_64-pc-windows-msvc c909b66 2023-11-07 15:07:26.748571656)

[bruceg]

Assuming your input JSON data file has the provided record as a single line, here is what is going on. When the file source reads the data file, it reads each line as a string. So, when the remap program invokes parse_json(.message), it is turning that string into a JSON object. However, that JSON object has a message field that itself contains an embedded JSON object as a string, so you need to add another ., err = parse_json(.message) to decode that. However, that will fail, because the embedded JSON is invalid due to the "status": 50e which contains an invalid number. Correcting that number allows the VRL to produce the results you are looking for.

[csh0101]

as I mentioned in this #19223 ... Because of the message is always appear in log .. But also the message is the whole output of source in vector. so I think .. there are maybe some mess on the Semantics @bruceg

can you consider my advice for this? maybe this is redundant..

[csh0101]

I think this is redundant . with deep learning about vrl , i aware of the office post is enough to distinguish the message on .. one are fields of source , maybe be is file .. maybe others ,and message in content should be deal by developer .. I will close #19223