to_timestamp does not honor timezone global setting?

[arve0]

I cannot seem to get to_timestamp to honor global option timezone. Am I holding it wrong, or is this a bug?

Documentation says:

The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone.

Simple reproduction:

timezone = "Europe/Oslo" # +01:00 at this time

[sources.my_source_id]
type = "stdin"

[transforms.my_transform_id]
type = "remap"
inputs = [ "my_source_id" ]
source = ".timestamp, err = to_timestamp(.message)\nlog(err)"

[sinks.my_sink_id]
type = "console"
inputs = [ "my_transform_id" ]
encoding.codec = "json"

Run: docker run -ti --rm -v $(pwd):/etc/vector timberio/vector:0.28.1-debian

Output zulu timestamp is OK:

2023-03-21T13:52:21.187226Z
2023-03-21T13:52:27.282141Z  INFO transform{component_kind="transform" component_id=my_transform_id component_type=remap component_name=my_transform_id}: vrl_stdlib::log: <null> internal_log_rate_secs=1 vrl_position=41
{"host":"ec7487bf20ef","message":"2023-03-21T13:52:21.187226Z","source_type":"stdin","timestamp":"2023-03-21T13:52:21.187226Z"}

Same timestamp with local time and milliseconds fails, reported as feature request 10. january:

2023-03-21T14:52:21.18722
2023-03-21T13:52:55.158379Z  INFO transform{component_kind="transform" component_id=my_transform_id component_type=remap component_name=my_transform_id}: vrl_stdlib::log: function call error for "to_timestamp" at (18:40): No matching timestamp format found for "2023-03-21T14:52:21.18722" internal_log_rate_secs=1 vrl_position=41
{"host":"ec7487bf20ef","message":"2023-03-21T14:52:21.18722","source_type":"stdin","timestamp":"1970-01-01T00:00:00Z"}

Same timestamp works with +01:00:

2023-03-21T14:52:21.18722+01:00
2023-03-21T13:54:42.329474Z  INFO transform{component_kind="transform" component_id=my_transform_id component_type=remap component_name=my_transform_id}: vrl_stdlib::log: <null> internal_log_rate_secs=1 vrl_position=41
{"host":"ec7487bf20ef","message":"2023-03-21T14:52:21.18722+01:00","source_type":"stdin","timestamp":"2023-03-21T13:52:21.187220Z"}

Removing split seconds makes it parse, but time is wrong. Here I expected 2023-03-21T13:52:21Z, not 2023-03-21T14:52:21Z:

2023-03-21T14:52:21
2023-03-21T13:53:13.552961Z  INFO transform{component_kind="transform" component_id=my_transform_id component_type=remap component_name=my_transform_id}: vrl_stdlib::log: <null> internal_log_rate_secs=1 vrl_position=41
{"host":"ec7487bf20ef","message":"2023-03-21T14:52:21","source_type":"stdin","timestamp":"2023-03-21T14:52:21Z"}

[jszwedko]

Hi @arve0 !

You can see the list of "automatic" timestamp formats here:

vector/lib/vrl/core/src/conversion.rs

Lines 230 to 254 in ad3b913

	const TIMESTAMP_LOCAL_FORMATS: &[&str] = &[
	"%F %T", // YYYY-MM-DD HH:MM:SS
	"%v %T", // DD-Mmm-YYYY HH:MM:SS
	"%FT%T", // ISO 8601 / RFC 3339 without TZ
	"%m/%d/%Y:%T", // ???
	"%a, %d %b %Y %T", // RFC 822/2822 without TZ
	"%a %d %b %T %Y", // `date` command output without TZ
	"%A %d %B %T %Y", // `date` command output without TZ, long names
	"%a %b %e %T %Y", // ctime format
	];
	/// The list of allowed "automatic" timestamp formats for UTC
	const TIMESTAMP_UTC_FORMATS: &[&str] = &[
	"%s", // UNIX timestamp
	"%FT%TZ", // ISO 8601 / RFC 3339 UTC
	];
	/// The list of allowed "automatic" timestamp formats with time zones
	const TIMESTAMP_TZ_FORMATS: &[&str] = &[
	"%+", // ISO 8601 / RFC 3339
	"%a %d %b %T %Z %Y", // `date` command output
	"%a %d %b %T %z %Y", // `date` command output, numeric TZ
	"%a %d %b %T %#z %Y", // `date` command output, numeric TZ
	"%d/%b/%Y:%T %z", // Common Log
	];

It looks like timestamps with fractional seconds but missing a timezone is not one of them. I think it would be reasonable to add though, if you wanted to open a PR, given we do support it for timestamps without fractional seconds and without a timezone.

Alternatively you can use parse_timestamp with a specific timestamp format. For example:

$ parse_timestamp!("2023-03-21T14:52:21.187221", "%FT%T.%6f")
t'2023-03-21T17:52:21.187221Z'

Hope this helps!

[arve0]

Thanks ☺️

What about the missing time zone adjustment +01:00? Am I using the config parameter wrong?

That is the format "%F %T", // YYYY-MM-DD HH:MM:SS I believe, which parses, but does not adjust to local time zone:

Here I expected 2023-03-21T13:52:21Z, not 2023-03-21T14:52:21Z

[jszwedko]

With the +01:00 it does seem to be working. The input you have is:

2023-03-21T14:52:21.18722+01:00

And the timestamp in the output is:

2023-03-21T13:52:21.187220Z

Which we can see is an hour different. Internally all timestamps in Vector are UTC; it's just the parsing that adjusts for local time. #3333 is tracking an enhancement to encode the parsed timezone.

[arve0]

I expected it to default to global time zone set, when no time zone is given in the date time string.

2023-03-21T14:52:21.18722

should parse to

2023-03-21T13:52:21.187220Z

when timezone: Europe/Oslo is given as global setting.

I see that you first example did in fact parse as some timezone (2023-03-21T17:52:21.187221Z).