csv parsing help please

[Manniac07]

What is the best way to parse a multiline csv into metrics? The first row is header row, 1st column is to be ignored, metrics are nested 2nd column then 3rd column. 4th-115th columns are the actual metrics values.
(Not sure if it makes a difference but each csv will have a different name based on timestamp)

New csv will come in every minute.
here is a sample of the the csv:

<style> </style>

TimeStamp	Realm	Message/Event	Request Server Total	Request Client Total	Retransmissions Server Total	Retransmissions Client Total	100 Trying Server Total	100 Trying Client Total	180 Ringing Server Total	180 Ringing Client Total	181 Forward Server Total	181 Forward Client Total	182 Queued Server Total	182 Queued Client Total	183 Progress Server Total	183 Progress Client Total	1xx Server Total	1xx Client Total	200OK Server Total	200OK Client Total	202 Accepted Server Total	202 Accepted Client Total	2xx Success Server Total	2xx Success Client Total	30x Moved Server Total	30x Moved Client Total	305 Use ProxyServer Total	305 Use ProxyClient Total	380 Alternative Server Total	380 Alternative Client Total	3xx Redirect Server Total	3xx Redirect Client Total	400 Bad Request Server Total	400 Bad Request Client Total	401 Unauthorized Server Total	401 Unauthorized Client Total	403 Forbidden Server Total	403 Forbidden Client Total	404 Not Found Server Total	404 Not Found Client Total	405 Not Allowed Server Total	405 Not Allowed Client Total	406 Not Acceptable Server Total	406 Not Acceptable Client Total	407 Proxy Auth Req Server Total	407 Proxy Auth Req Client Total	408 Request Timeout Server Total	408 Request Timeout Client Total	415 Bad Media Type Server Total	415 Bad Media Type Client Total
1.66E+09	AttFlexInt	INVITE	6394898	820302	10254	129	3542599	818618	3065044	15856	0	0	11	0	2145226	0	0	0	6164120	807437	0	0	0	0	0	0	0	0	0	0	0	0	0	13	0	0	0	198	0	8936	0	0	0	0	0	0	11281	0	0	2
1.66E+09	AttFlexInt	ACK	6381923	819511	790	261	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
1.66E+09	AttFlexInt	BYE	2352087	1289229	9	0	0	0	0	0	0	0	0	0	0	0	0	0	2351273	1288542	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
1.66E+09	AttPriFlexExt	INVITE	427360	3235672	3663	64	157319	3235703	7917	1621092	0	0	0	4	0	1123035	0	0	419232	3116994	0	0	0	0	0	0	0	0	0	0	0	0	1	38	0	0	16	0	1122	0	0	0	0	0	0	0	326	22751	1	0
1.66E+09	AttPriFlexExt	ACK	426927	3229761	119	1419	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
1.66E+09	AttPriFlexExt	BYE	615533	1182702	1518	137	0	0	0	0	0	0	0	0	0	0	0	0	615270	1182681	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0

[Manniac07]

The reason for my question above is because I created a toml using csv.lua, and sending to Victoria Metrics via Prometheus Remote Write. the metrics are written fine for a random amount of time but then Victoria Metrics stops receiving metrics. The logs for Vector have no errors or warnings it just keeps watching new files. I've tried using buffer memory, and then disc. I've increased disc size, I've tried to increase batch.timeout_secs just incase that was where it was backing up, and I've tried to increase fingerprint just incase it was an issue with same values across the top. nothing seems to work. Below is my source and sink toml configs my transform is using csv.lua to put all values into a 2d array and then spinning through every Message/Event for each realm and sending the value for each metric header.
`Source:

Change this to use a non-default directory for Vector data storage:

data_dir = "/.../.../vector"

Random Syslog-formatted logs

[sources.vectorsiprealmmethod]
type = "file"
fingerprint.lines = 50
fingerprint.strategy = "checksum"
fingerprint.ignored_header_bytes = 0
glob_minimum_cooldown_ms = 60000

include = ["/.../.../.../sip-realm-method/*.csv"]
oldest_first = true
read_from = "beginning"

Sink:
[sinks.vectorsiprealmmethodConsole]
type = "prometheus_remote_write"
inputs = ["vectorsiprealmmethodmetric"]
endpoint = "https://.../insert/"
healthcheck.enabled = false
buffer.type = "disk"
buffer.max_size = 1073741824
buffer.when_full = "block"

[sinks.vectorsiprealmmethodConsole.request]
   concurrency = "adaptive"`

[jszwedko]

The easiest way to parse CSV data in Vector is to use the https://vector.dev/docs/reference/vrl/functions/#parse_csv function in a remap transform.

[Manniac07]

Yes I saw that but it is very vague, and I couldn't find any good examples of how to use it in my case. @jszwedko do you know of any good examples to use parse_csv in a transform?

[Manniac07]

oh and how does that work if everything needs to be dynamic? so the 1st row of headers have to be read in and used dynamically, not hard coded. I guess my other question (please excuse me if it's a stupid one) is what does it mean by parsing a single line csv? mine is multiple rows of data.

[jszwedko]

It depends on the source, but most sources read line-by-line so you won't be able to reuse the headers from the file. Instead you'll end up with an array of fields.

[Manniac07]

@jszwedko, so will parse_csv work being that I will need to rotate thru a multi dimensional array? for example I will need the following output for the csv above:
AttFlexInt, INVITE - Request Server Total 6394898
AttFlexInt, INVITE - Request Client Total 820302
AttFlexInt, INVITE - Retransmissions Server Total 10254
(to end of Columns)
...
AttFlexIn, ACK - Request Server Total 6381923
...
AttFlexIn, BYE - Request Server Total 2352087
...
AttPriFlexExt, INVITE - Request Server Total 427360
...
to end of rows

[Manniac07]

@jszwedko also do you have any idea looking at my configs above why what I have is continuously working for anywhere from 1 hr to 2days and then will stop sending data to victoria metrics (using Prometheus remote write)without any error logs, all logs say it is still processing. if I restart it starts sending again and will work for another period of time then stops sending again.