Tzdata 2.0 plans

[lau]

Ideas for a new version of Tzdata:

Compile time data with update warnings

• Allow the data to be a separate Hex package, rather than downloaded directly from the IANA servers. Use this data at compile time, similar to Tzdata version 0.1. Macros or persistent_term.
• Do unverified HTTPS calls to check if that current data-Hex package is out of data, and warn about it. It should be on by default so that it is seen. Options for different kinds of alerts including at run time.
• This allows getting rid of Hackney and having 0 required dependencies since the check does not have to be secure. The idea is to have a check for convenience - even though the check is not secure the worst case should be that the check tells you the data is up to date even though it isn't - or vice versa. In case castore or hackney is available upon compile time, one of those it could however be used for doing the check securely.

Automatic updates

With the automatic check, the automatic updates are less needed. There might not be many people that need the automatic updates if they get the warnings and can deploy the new version of the software each time it is available.

It could be kept as a configurable option and have optional dependencies for secure downloads. Although it would be more complexity. So I'm learning towards not continuing with the automatic updates in version 2.0 unless there is a lot of demand for it. Version 1.x would still have it.

Background

A big issue that the automatic updates solves is that in many systems - including ones made by trillion dollar tech companies - are out of date even when a new version of tzdata from IANA is released. This causes issues for users. With the automatic updates the data is kept up to date without having to wait for manual intervention.

Some downsides are that performance with ETS is not as good as with macros. Another is the lack of control of when the updates happen. Currently it can still be done manually, but it the process could be simpler.

To help out developers and sysadmins, a the data-in-a-hex-package solution with warning system is an alternative that still helps to keep the data up to date, even though it requires manual intervention in the form of bumping a hex package version.

Warnings

It is important that these warnings are visible by default so that users won't miss it. Warnings during run time in development and production. Perhaps making it easy to integrate with Continuous Integration systems as well. E.g. CI could be configured to fail if the data is out of date by more than a week.

Issues with users not configuring a store for the ets files

Currently you can store the files with the timezone releases on disk and have the newest ones available at all times even between deploys. However many people don't do this when releasing. In that case every time they release there are a few seconds where the tzdata version shipped with Tzdata is the one that is being used. Until a newer one is downloaded - if available. So a deploy happen with an old version, then after a few seconds the new version is downloaded and used. Then for the next deploy the old version is used again for a few second. And then the new version.

Again this can be avoided by configuring a place on disk between deploys, which is possible even with containerization, but most people don't actually do that. It takes some work to configure your system like this and is not prioritized by many.

Details

The hex package containing the data would simply contain the data found in IANA. Some automation could be done to help publish the hex packages quickly after a new version is available from IANA.

[cdegroot]

It is important that these warnings are visible by default so that users won't miss it

I can honestly state that I've never seen the warning. Logs go to Splunk/Sumo/Cloudwatch and one time that I did notice it was when I specifically dug into an outgoing firewall warning ("why is this production system trying to talk with IP a.b.c.d"?).

Did y'all consider just forcing the user to make a choice? Crash when Application.get_env(:tzdata, :check_for_updates) is nil? Given that this is a discussion where I can defend either default, this might just be a situation where "no default" is the best outcome.

[lau]

I can honestly state that I've never seen the warning

They haven't been created yet. This is a plan for a future version.

Logs go to Splunk/Sumo/Cloudwatch and one time that I did notice it was when I specifically dug into an outgoing firewall warning ("why is this production system trying to talk with IP a.b.c.d"?).

That sounds like an anecdote about the current version. The answer to your question is in one of the first lines of the README. The README also covers instructions for how to configure this.

[cdegroot]

The answer to your question is in one of the first lines of the README

I think we've been here before - the library is an important indirect dependency in the Elixir ecosystem for most users, and only few users will end up stumbling upon the README.

And yes, that's "hey, what's going on here?" is about the current version. I'm happy that attempts are made to address that and just trying to add something to the thinking by suggesting forcing the user to make an explicit decision here - whether a transient dependency reaches out to IANA or hex.pm, it's the call-home-by-default decision that's troublesome (at least it will, based on this proposal, not update a deployed package which is a big step forward, thanks for considering that). But I guess that that station has been long passed.

[epinault]

is this still in the work? Hackney is depending on a very old lib and also seems it would be best to rely on a more default client within erlang