using reduced "timezones-now" dataset

[jannikmi]

There is an alternative dataset variation with reduced set of timezones: timezones-now
Let's consider using this simplified version of the dataset:

In this dataset the original 444 timezones get merged whenever they share the same behavior (as of now).
Reduced timezones: https://github.com/eggert/tz/blob/main/zonenow.tab
Example: Coordinates which currently get "Europe/Berlin" output would instead get "Europe/Paris"

PR for this: #324
Data report

This would have the following significant benefints:

• reduced amount of timezones 92 instead of 444
• less regions with timezone borders, less ambiguity, less need for differentiation (21.8% instead of 25.5% of the shortcuts would not have a unique timezone)
• reduced amount of timezone polygons, 855 instead of >1300, 579 holes instead of 753
• simplified timezone polygons
• reduced amount of coordinates
• smaller storage space, 32MB instead of 61MB binary data (50% reduction), making it very relevant for reduce release memory footprint #317
• faster response time

Drawbacks:

• will provide incorrect data for observed timekeeping methods in the past at certain locations. only suitable for use cases that need to know the current or future time at a location
• the information about the location encoded in the more specific timezones is lost
• would make testing the behavior a bit harder, as the "correct" result now does not correlate so strongly with the location/country any more.

Community input required:

• Are there any objections to making this significant change?
• Does anyone care about determining the timezone (offset) in the past?
• Are there reasons why it is not feasable for the affected users to parse their own data (file_converter.py)?

[jannikmi]

@ringsaturn have you considered this for the tzfpy ecosystem?

[jannikmi]

Berlin city is covered by both "Europe/Paris" and "Europe/Berlin"

When I check the data, there is no "Europe/Berlin" timezone left.
So this bug must have been fixed in the latest version.

[ringsaturn]

Berlin city is covered by both "Europe/Paris" and "Europe/Berlin"

When I check the data, there is no "Europe/Berlin" timezone left. So this bug must have been fixed in the latest version.

You are right. it’s my browser’s cache.

[ringsaturn]

some services also use timezone names for their geo relation rules, for example "Asia/Tokyo" means using Japan's local data first, especially for complex micro-services' gateway or BFF layer services.
It's convenient and performance friendly, without another calling for lnglat-to-country/region.

That's valuable input. Can some of the users confirm this kind of use?

Those services are some companies projects, not open sourced projects. So it’s difficult to provide details.

[jannikmi]

I understand. The details are not required, just some high-level information from some users to estimate the relevance.

[jannikmi]

For everyone requiring the precise dataset usable for localisation beyong time behavior, I would argue there can be a simple custom CI/CD (script call) for parsing their own data. Especially for companies that should be manageable.
I would favour 70% of the users getting a better experience (faster, smaller package, more use cases in edge computing...) with the reduced dataset.
If it turns out that the simplified data is not sufficient for most users, then that is of course a good argument to keep the current behavior (data set).

[jannikmi]

@theirix (maintainer)

Popular open source projects using timezonefinder:
@drbacke https://github.com/Akkudoktor-EOS/EOS/blob/ec8b226ef4e9ab5813506299dd6128a39da81280/src/akkudoktoreos/utils/datetimeutil.py#L58
@brickbots https://github.com/brickbots/PiFinder/blob/472ffa00f5f0010a936511b5829eed2da1a819c8/python/PiFinder/state.py#L314
@joaovitoriasilva https://github.com/joaovitoriasilva/endurain/blob/1a5a26f8b2d998e24a98858b30c3af8edbd1456a/backend/app/activities/activity_sets/utils.py#L20 (using zoneinfo)

The way I see it, all of these examples use this library for actual timezone behavior specification at the current point in time. not more. Am I correct?

As critical users of this project, what is your take on this topic?
Do you think the larger quantity of timezones, allowing more precise localisation than it is required for determining the timezone behavoir, should be kept?
Do you know of critical use case requiring this?

[joaovitoriasilva]

Hi! If I am not mistaken, I do not look to this part of the code for a few months, Endurain uses this for two actions:

• to convert a lat and long pair to a timezone
• convert a naive time to the specific timezone

[theirix]

Hello @jannikmi .

This change is huge, helpful and deserves a major semantic version bump with a clear announcement and usage scenarios.

The behaviour for current, non historical, data looks good to me.

Internally, we use the package to translate an event geo position to an expected timezone.

Sometimes people could use this project to find a country (I guess), and this change could affected them. I don't believe it's a proper usage for timezoneresolver (not country resolver).

[jannikmi]

@theirix, thanks for your input!
that confirms me in my point of view: the main focus of timezonefinder should be intersection of geolocation and timezone behavior.
resolving geolocation to a country is NOT the main concern of this package.

I decided that timezonefinder will use the reduced timezone-now dataset format in the next major release.

Do you have a suggestion for how to include clear announcement and usage scenarios?
I would add a description of the change in the Changelog and add disclaimers for "country resolving" to the documentation.
The announcement for making this change can also be found in this discussion (below).
Do you think that is sufficient?

Anyone is welcome to contribute to #324

[theirix]

Hello @jannikmi . I could be late to the party for one hour, but your announcement is great and fully reflects what I wanted to announce

[jannikmi]

Announcement

timezonefinder will switch to using the reduced timezones-now dataset version starting from version 8.0.0

Rationale

The core task of this package is to convert a given coordinate into timezone for determining timezone behavior at the current point in time.
The current possibility to use the more precise 444 zones for localisation (resolving to a country) is not relevant for this core task.
The majority of target users actually only have drawbacks from this: larger installation footprint, slower responses, etc.

Note: The localisation use case can still be supported by parsing the original dataset version individually.

[jannikmi]

@oPromessa thanks for your input.
If timezonefinder used the reduced timezone-now dataset version, the timezones would only change if the time behavior changed.
In case of your examples:
(lng, lat)
https://timezonefinder.michelfe.it/gui/0_3.96_61.08 -> Etc/GMT (ocean timezones stay untouched)
https://timezonefinder.michelfe.it/gui/0_5.03_60.09 -> Europe/Oslo (would become Europe/Paris due to the same 2h UTC offset)

What do you use the timezone for? Just the time behaviour at the location and time the picture was taken?

[oPromessa]

Thanks for the quick reply.

What do you use the timezone for? Just the time behaviour at the location and time the picture was taken?

To assign the correct time zone to the pic to have all pics correctly sorted by the actual date/time taken based on time zone. Personally I'd prefer to have the pic tagged Oslo and not Paris.

On another note would the reduced set consider DST info correctly (which is a pain in my existence😊) ?

Maybe dumb question: would it make sense to have an option for the user/programmer to run/select on the detailed or reduced data set ? I guess runtime efficiency could be obtained, but installation size would still not be solved.

[jannikmi]

Thanks for clarifying. Does Oslo/Paris make any difference? The way I understand you are only interested in sorting the pictures and not the name of the timezones, right?

The DST behavior is being considered. In your concrete example the behavior of Paris and Oslo is identical and hece the two zones are merged:

Paris and Oslo are in the Central European Time (CET) zone (UTC+1), and they both observe Central European Summer Time (CEST) (UTC+2) during DST.
They switch to DST on the same date and revert on the same date, following European Union DST rules, even though Norway is not in the EU.

your valid question:

would it make sense to have an option for the user/programmer to run/select on the detailed or reduced data set ?

unfortuately not. then both datasets would need to part of the distributions and the size would increase.
the easiest solution is to parse the desired dataset for the use case individually (all scripts for this are included in the timezonefinder repository)

[oPromessa]

Thanks for clarifying. Does Oslo/Paris make any difference? The way I understand you are only interested in sorting the pictures and not the name of the timezones, right?

Yes. That's the end goal. But I need the time zone name as input to the osxphotos tool to change pics' timezone on the Mac Photos app where all my pics are.

Some more detail/background:

• My pics are on Mac Photos.
• Throughout the years and with multiple updates and versions of iPhoto and Photos, resulted in incorrect time zone info on many pics.
• I'm fixing/adding GPS location to all my Photos pics.
• with timezonefinder I determine the appropriate timezone name.
• I then use the osxphotos tool to set the timezone name on Photos with a command like: osxphotos timewarp --uuid 3538147D-4D49-4E0A-98A2-D69D7007DA10 --timezone 'Europe/Madrid' --force --verbose
• Simply for reference this came from this debate: issue1788

Of course my use case may be a particular one and I don't want to influence your decision, should the majority be fine with a reduced data set.

[jannikmi]

Cool. I understand that in your case the more accurate timezone name would be more intuitive for the human user, but concerning the desired time behavoir both versions (e.g. "Europe/Paris" instead of "Europe/Oslo") are equal and hence sufficient.

It is intersting to see that in the discussion you linked the large 50 MB size of the current timezonefinder distribution has been stated as a major drawback. Even though tzfpy will always win in this category with its simplified polygons, a 50% size reduction to me sounds like a significant improvement in that regard.
Perhaps this could enable more users to use the more precise boundary data of timezonefinder.

[jannikmi]

@evansiroky: First of all, thanks for your continuous work on this critical timezone dataset!
Do you know of any other similar client packages which use the timezones-now or any other reduced version (have made the transition from the full original version)?
Do you see any limitations not considered here so far?
Thanks for your input on this.

[jannikmi]

UPDATE:

Major finding: Multiple users are using this library not solely to determine timezone behavior, but for localisation purposes (cf. #363). For this the full dataset is required.

Consequence: I have decided to switch back to the original full dataset with the next release.

Rationale: All solutions to this problem (separate data dependency, data parsing script) require more development effort and hence will delay the resolution of this issue.
I don't want users like @tyler-8 @314eter to experience the drawbacks of the dataset change any longer.

Since the storage limit of the PyPI repository has been increased and @theirix drastically reduced the amount of prebuild wheels in the release, the memory footprint of each release is not as critical any more.