S3 sort in reverse chronological order

[alanwguo]

Hey all,

using the S3 sink, is there a way to format the file names so they are naturally sorted in reverse chronological order? The main usecase is I'm creating many small batches of events and want to make it faster to iterate through the recent ones.

One work-around I am doing is calculating the timestamp of the events and subtracting it from a large number (similar to 9s complement), but the issue with that approach is that I cannot pad the result with leading zeroes.

Alternatively, if I can format integers in VRL to pad with leading zeroes that would be helpful as well.

[StephenWakely]

The way you are doing it is the best way I can think of.

I'm not quite sure what the issue is with padding the number? Given that it is over 250 years until the string length of the unix timestamp will take another digit, you should just be able to prepend the required number of zeros without worrying about increasing the length of the resulting string.

$ "000" + to_string(9_999_999_999 - to_unix_timestamp(now()))
"0008313431890"

Presumably you will want to round the timestamp to avoid creating a new object every second.

[alanwguo]

Thanks for confirming the approach.

You're right about the padding. It will be a long while before a missing leading 0 will be a problem.

[srstrickland]

stumbled upon this thread while looking for something else, and thought I'd share a trick I used the other day to pad a base64 string with = to make it decodable. the issue for me was that there wasn't an easy way to dynamically generate the pad length, as VRL doesn't have anything like "=" * pad_length or repeat("=", pad_length). but you can define your "padding mask" and then slice it. here's an example below with unixtime (you can change now to any shorter string and the padding still works):

now = to_string(to_unix_timestamp(now()))
padding = "000000000000"
pad_len = length(padding) - length(now)
padded = slice!(padding, 0, pad_len) + now

here's the code above in action.