NOAA GIBBS

[nickrsan]

New Dataset

Please fill out the New Issue form so we can easily organise the issues and put a priority on certain data. The title should reflect the dataset you want us to save. In the body, please include the following information.

• Agency: NOAA
• Agency Division: NCEI
• Data Type: Raster, Satellite Remote Sensing
• Data Format: Various
• FTP/HTTP URL: https://www.ncdc.noaa.gov/gibbs/

I received an email with this information as a suggestion. Some of the facts are a little off, but it may be worth looking into if an archive exists somewhere of this data.

It has come to my attention that NOAA's NCEI "National Centers for Environmental Information" GIBBS archive is being taken offline on March 31. I have been unable to find mirrors of this data from other sources.

GIBBS is a public browsing system of image data collected by the International Satellite Cloud Climatology Project. The archive contains global geostationary satellite data, from nearly every location on the Earth at three-hour intervals from 1980 to present, roughly 0.6 terabytes of images.

A message on the website urges users to send them an email about how they use GIBBS, so that they can create a suitable substitute, but knowing how these things often go, I have a lot of doubt that we'll ever see it online again. Better safe than sorry. Additionally, updates to GIBBS stopped only two months after the Presidential inauguration.

[rustyguts]

Any recommended wget commands for grabbing this site? @nickrsan

[JeremiahCurtis]

Wonder if httrack would work here

on a side note, I wonder, given this dataset, how many other NCEI data are not incorporated into https://www.ncei.noaa.gov/data/ ?

[rustyguts]

httrack came back with about 100mb.

[kimmerin]

Hi,

I started a WinHTTrack-session and it seems to work. After 3 hours it has saved 13000 files (just the calender overview pages) and about 850 MB of data.

I'll keep it running until it fails or my provider starts complaining. Problem is that I'm sitting at an asymmetric ADSL-line so I won't be able to set up a server here that can be reached. I think that's what you're calling a "private mirror" ;-)

[JeremiahCurtis]

Fascinating answer (?) to the question I posed earlier regarding our coverage of NCEI data:

from https://www.nodc.noaa.gov/oceanacidification/stewardship/data_portal_help.html

"National Centers for Environmental Information (NCEI) Oceans
NOAA's National Centers for Environmental Information (NCEI) hosts and provides public access to one of the most significant archives for environmental data on Earth. Through the Center for Weather and Climate and the Center for Coasts, Oceans, and Geophysics, we provide over 25 petabytes of comprehensive atmospheric, coastal, oceanic, and geophysical data."

Also, and I hope I'm freaking out for nothing, but on this same page, I'm getting the following: "This National Centers for Environmental Information (NCEI) Oceans web site is currently unavailable. We apologize for any inconvenience." I can't seem to get ftp://ftp.nodc.noaa.gov/pub/ either, as I am attempting to grab some of the symlinks on that directory, and finish mirroring the archive directory

[kimmerin]

BTW:

Additionally, updates to GIBBS stopped only two months after the Presidential inauguration.

Make that five days. The inauguration was at Jan 20, 2017, the last image being uploaded was at Jan 25, 2017

[kimmerin]

Short status update. WinHTTrack is downloading HTML pages sequentially and doesn't start with the images after the download of the HTML pages. With a thoughput of a bit over 1 page per second and ca. 1.6 million pages to download that would never finish before the deadline.

Good thing is that the HTML pages are a generic creation from a script so there is no actual need to download them, you just need the information what images are available which is the case when you have the pages containing the availability (files in /gibbs/availability). The rest can be generated dynamically again with a homebrewed script.

So I switched tactics and wrote my own Java programs downloading the availability-files, extract the image-links into a file and downloading these files in parallel. At the moment I have downloaded about 691,000 images with 865,000 still to go.

If you do the math and compare the sum with the reported number of images hosted by the site, you might ask about the difference of nearly 20,000 images. At the end of this download this number will ride to more than 100,000. The organization of these images on the server seems to be less than optimal. I have the theory that an URL like https://www.ncdc.noaa.gov/gibbs/image/GRD-1/WV/2016-12-31-21 is served from a resource in a filesystem with path that looks something like /some/path//image/GRD-1/WV/2016-12-31-21. The example here is the problematic one already. There are about 200,000 images within the GRD-1-path and most filesystems' access times increase exponentially with the number of files in a directory. My personal record was 60 seconds just to check for the existance of a file (no actual access of the content) in a directory containing a million files. Attempts to download an image from the GRD-1 path leads to more and more HTTP-503-errors slowing down the download process, so I excluded them from the main download process. This increased the download throughput from 50,000 images per day to now about 200,000 images per day.

I will start a parallel download from another system only for the GRD-1 files. With one image per second this should be finished within a day but there is plenty of time before the deadline in case the throughput should go down more than that.

[JeremiahCurtis]

@kimmerin As of 3/20/2017, NOAA reports 1,573,397 images. Are the GRD-1 images the only ones not entirely available from the main directory, or are there others? Would be glad to help but this one appears to be a labryinth...

Also, the /availability url appears not to be working...maybe it's just me

[kimmerin]

@JeremiahCurtis The files are available per se. The only problem is that there seems to be some kind of timeout the server is willing to wait for the resource to be available and sends an HTTP-error 503 in case it takes to long. So you have to do multiple attempts to get the actual file. This happens across all satellite images (at the moment MET-7-images also lead to multiple 503-error before there is success) but GRD-1 images often surpassed the maximum number of attempts I programmed (20). With one second wait time between attempts it really slowed down the whole process of downloading images.

The GRD-1 images are "only" the result of a stitching of the different images leading to an image covering the whole world. Even if we're unsuccessful in downloading the GRD-1-images in time it would be possible to stitch them ourselves using the images that are downloaded.

Also, the /availability url appears not to be working...maybe it's just me

It wasn't an URL, a complete URL would be e.g. https://www.ncdc.noaa.gov/gibbs/availability/2017-03-17.html
The URL is quite straight forward and it was easy to cerate a program that is generating the URLs by formatting the corresponding dates.

For what it's worth, here is the list of GRD-1-images that are excluded from my current download.
grdfiles_offset.zip
wc -l grdfiles_offset.txt
103328 grdfiles_offset.txt

[pelerin]

@kimmerin As of tonight looks like GIBBS is offline. Hope this is temporary, or that you got all the images!!

[kimmerin]

I hope it's temporary as well since I haven't been able to finish the download, yet. About 400,000 images are still missing.

[JeremiahCurtis]

Still not working......wonder if we could contact NOAA about this issue

[sa7mon]

Good news!

Information

The GIBBS web service will not end March 31, 2017.

Based on feedback from users, we will continue to support the GIBBS service. However, we apologize 
for any gaps in service that may occur (likely to occur in early April) as we strive to maintain the same level of service. 

[entr0p1]

I've found that all of the sat Images I've looked at so far have the id of "satImage" in their tags in the HTML code. I'm working on a crawler to grab the images at the moment.