List of mirror URLs for IERS download

[lpsinger]

Transferred from astropy/astropy

Both of the USNO hosts (maia.usno.navy.mil, toshi.nofs.navy.mil) for retrieving the IERS Bulletin A dataset are flaky. Instead of a primary URL specified by the configuration setting utils.iers.iers_auto_url and mirror URL by utils.iers.iers_auto_url_mirror, I suggest a single list of URLs.

Here is what I propose for backward compatibility:

• Understand both iers_auto_url and iers_auto_url_mirror to be either a string or a list of strings.
• Construct the list of URLs to try by concatenating all of the URLs from iers_auto_url and iers_auto_url_mirror and removing duplicates.
• Deprecate the iers_auto_url_mirror configuration setting for removal in a future release.

Here is a possible default URL list:

• http://maia.usno.navy.mil/ser7/finals2000A.all
• http://toshi.nofs.navy.mil/ser7/finals2000A.all
• ftp://cddis.gsfc.nasa.gov/pub/products/iers/finals2000A.all
• https://datacenter.iers.org/data/9/finals2000A.all

[pllim]

Pinging @taldcroft, @mhvk, @eteq, and @adrn since iers is primarily used in table and coordinates.

Opsie... turns out I did not have enough foresight when I implemented the mirror in astropy/astropy#8308 😬

[mhvk]

Seems like a good idea. My only question would be if we are not doing something similar already elsewhere - just to be sure we don't duplicate different mechanisms (but the suggested scheme sounds good).

[rtolesnikov]

At least as of today, https://maia.usno.navy.mil/ser7/finals2000A.all is required for access. http:// (without the 's') no longer responds.

[pllim]

@lpsinger , is it less flaky now with latest astropy dev after astropy/astropy#8993 is merged?

[lpsinger]

@pllim, no, it's still really flaky. Our project (growth-astro/growth-too-marshal) has to download the IERS files in its CI tests, and even with the https URL, it's failing more often than it is succeeding. I think that we really do need a list of fallback URLs. Also, astroplan needs to be able to use the fallbacks.

[wtgee]

Chiming in here as well (in addition to astropy/astroplan#356 (comment)). We have no problem providing our own mirror, mitigating the flakiness issue, but there still seem to be some consistency issues with how it is detected by the library. This might be specific to astroplan itself rather than astropy, but wanted to at least link the issues.

[lpsinger]

I don't think that you should provide your own mirror unless you can guarantee that you can update it whenever a new copy is issued.

[mhvk]

Agreed with @lpsinger. I am also not sure whether it is so necessary to download IERS-A in every CI run - it is a hefty file and why test someone else's server? (Or add unnecessary pressure to it) It seems to me for testing one's own code, it is much more important to check that a given state leads to the right outcome.

Anyway, I'm still in favour of the simple solution proposed on top, of just making a list of URLs, starting with just those sites that already claim to provide up-to-date copies.

[lpsinger]

The issue with Astroplan is that it directly checks for the existence of the file in the cache, and the filename in the cache would depend on which URL is successfully downloaded.

Can Astropy instead manage this?

[wtgee]

I don't think that you should provide your own mirror unless you can guarantee that you can update it whenever a new copy is issued.

The issue for us is described in astropy/astroplan#410, namely that the military domains are banned from certain countries and the ftp download doesn't work with the astropy setup (because it is redirecting to ftps). The https://datacenter.iers.org/data/9/finals2000A.all should work but this limits us to only one server and as far as I can tell all of the URLs listed above are hosted on single-server instances that have proved to be far from reliable.

It is trivial to host a 3MB file in a cloud-based CDN and it is trivial to set up something that keeps that file up to date (indeed, it took me five minutes using a combination of Google Cloud Storage, Cloud Functions, and Cloud Scheduler).

The issue with Astroplan is that it directly checks for the existence of the file in the cache, and the filename in the cache would depend on which URL is successfully downloaded.

Yes, this is our bigger issue and a reliable global override of iers.IERS_A_URL. But perhaps I'm missing something?

[eteq]

I'm also 👍 to the list of URLs solution as in the OP and @mhvk's comment.

@wtgee

It is trivial to host a 3MB file in a cloud-based CDN and it is trivial to set up something that keeps that file up to date

In principle I agree but logistically there's a lot more to it if it's supposed to be permanent. "Easy right now" is not the same thing as "easy to keep running for the next 10 years". After all, that explicitly is the purpose of the IERS itself (the service, not the file)!

That said, combined with the list of URLs this isn't so bad because it makes it one of several options and thus adds something like more fault tolerance.

[wtgee]

@eteq agreed about the long-term viability. The real hope would be to host this as a public google dataset. However, I'm not sure that's in keeping with the nature of google public datasets as the data doesn't seem entirely useful by itself. Tagging @mimming for input in that regard.

[wtgee]

While I think this PR is good, it doesn't help my problem as the last two urls are invalid and the top two urls are still military domains.

How do I actually change the value of astropy.utils.iers.conf.iers_auto_url?

I've tried to follow the guides in the page on the config system but there is nothing about IERS in there and I can't seem to enter a value in any single file that will persist. The documentation is far from clear. Sorry for posting here, but we don't really need more IERS issues/PRs floating around.

[pllim]

@wtgee , before I propose a solution for you, how do you envision you want to customize your URL? Is it per session? Are you okay with modifying ~/.astropy/config.astropy.cfg manually? I can think of at least two different ways.

Addendum: IERS_A can be changed using the config system, but not IERS_B. I don't know why.