vrl format_timestamp timezone always +0

[coredump17]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

parse_timestamp should use local timezone, global timezone variable or a passed in timezone as format_timestamp does. Instead, incorrectly defaults to UTC.

Configuration

$ .timestamp = "06/03/2024 13:23:00"
"06/03/2024 13:23:00"

$ .timestamp2 = parse_timestamp!(.timestamp, format: "%d/%m/%Y %T")
t'2024-03-06T13:23:00Z'


$ .
{ "timestamp": "06/03/2024 13:23:00", 
  "timestamp2": t'2024-03-06T13:23:00Z'
}

$ .t = get_timezone_name!()
"Europe/London"


Europe/London is +1 due to BST at the moment. But appears to to default to +0.

A workaround can be to get the timezone from now() and append to the original timestamp but it would be much more cleaner to have parse_timestamp handle timezones (default to system if timezone is not passed).

$ .timestamp = "06/03/2024 20:27:00"
"06/03/2024 20:27:00"

$ .tz = format_timestamp!(value: now() , timezone: "Europe/London", format: "%z")
"+0100"

$ .t_and_tz = to_string(.timestamp) + ":" + .tz
"06/03/2024 20:27:00:+0100"

$ .timestamp2 = parse_timestamp!(.t_and_tz, format: "%d/%m/%Y %T:%z")
t'2024-03-06T19:27:00Z'

Version

vector 0.38.0

Debug Output

see vrl above

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Hi @coredump17 !

Thanks for filing this. I'm having a hard time reproducing though. My local timezone is PDT:

❯ date +'%z'
-0700

And, in the VRL console, it correctly parses with the offset:

$ .timestamp = "06/03/2024 13:23:00"
"06/03/2024 13:23:00"

$ .timestamp2 = parse_timestamp!(.timestamp, format: "%d/%m/%Y %T")
t'2024-03-06T21:23:00Z'

Here you can see it added 8 hours (rather than 7) because it was "standard time" (-0800) at the date encoded in that timestamp.

global timezone variable

This also works per the first example you see in vectordotdev/vrl#853.

Can you provide the output of date wherever you are running vector vrl? I'm wondering what timezone it reports.

[coredump17]

Hi @jszwedko

$ date +'%z'
+0100

$ date
Tue 4 Jun 00:34:57 BST 2024

$ .t = get_timezone_name!()
"Europe/London"

initially i thought it might not be seeing the system timezone but get_timezone_name!() shows the correct timezone name (above). Wondering if the UK's BST (+1) and GMT (+0) time change could be the cause ( https://www.gov.uk/when-do-the-clocks-change )

i have just managed to replicate this using your alpine dockerhub container.

ENV TZ="Europe/London" 
command: vrl

terminal:

/ # env | grep TZ;  date +'%z'
TZ=Europe/London
+0100

vrl:

$ .timestamp = "06/03/2024 13:23:00"
"06/03/2024 13:23:00"

$ .timestamp2 = parse_timestamp!(.timestamp, format: "%d/%m/%Y %T")
t'2024-03-06T13:23:00Z'

$ get_timezone_name!()
"+01:00"

[jszwedko]

@coredump17 I think the issue is that the timezone for the date you have, March 6th 2024, will not be BDT, but instead BWT (British Winter Time) (+0000).

[coredump17]

@jszwedko - legend! You are very correct "%d/%m/%Y %T should be "%m/%d/%Y %T for this application log!

Sorry to waste your time but thanks for helping me with this!

[jszwedko]

No problem! Happy we got it resolved.