Historical data

[romanmartushev]

Im looking for record temperature highs and lows. Is this possible with your api? If not, any chance this could be added?

[tgherzog]

I'm looking for this too. I'm trying to send a query such as https://api.weather.gov/stations/KBWI/observations/2021-01-15T19:54:00+00:00 but it's returning 404

[netbymatt]

I'm looking for this too. I'm trying to send a query such as https://api.weather.gov/stations/KBWI/observations/2021-01-15T19:54:00+00:00 but it's returning 404

There are several ins and outs to querying by date. Please see my post on stack overflow. https://stackoverflow.com/questions/50473592/api-weather-gov-start-and-end-times

[sullynole]

It's true that those endpoints work, but they don't help get historical data, which is the point of this thread. Unless I'm missing something? (I just checked, and it looks like /observations has ~1 week of data. That's not what I'd consider "historical".)

Correct, api.weather.gov does not contain observations beyond the last 7 days. Historical data should be sourced from NCEI.

[NateRL]

Thanks so much! I didn't realize that api.weather.gov was limited to the last 7 days (although I'm not surprised).

Shot in the dark, but do you have any pointers on reconciling historical observational data from NCEI with ongoing observational data from NWS? I'm building a machine learning model. I will train it on historical data (from NCEI, I guess) but then I need to monitor it in near-real time, so for that I'll need ongoing observational data (from NWS, I supposed). It would be really, really swell if I could find NCEI data and NWS data from the same observations so that the historical / observational data are actually the same. Any pointers?

[dwhitemv25]

Can you clarify "from the same observations"? The data is by station, so it sources from the same "data."

Note that the NCEI data goes through a QC process before becoming "official" and may vary from observed max/min.

[NateRL]

I haven't yet reconciled the stations between the NWS (weather.gov API) and NCEI sources, but I've already noticed that stations in the NWS data have 4-character identifiers and stations in the NCEI have 5-digit identifiers, so I wasn't sure if the data sources (the stations themselves) were the same or not. That's the next thing I'm working on.

The QC process is what I am going to look into after that, to see what the deviations look like between the raw NWS data and the QCd NCEI data. Depending on the variance (and how long it takes for the data to go through the QC process and show up in NCEI datasets), it may or may not be a problem. If I just have to wait, say, a week to get the QCd NCEI data that's what I'll do, and ignore the NWS observations (I still need their forecasts).

I'm still new to this whole area, and the data is really remarkably poorly documented, so I appreciate any of the help folks can offer.

Can you point me to a method for reconciling stations across NWS and NCEI data?

[dwhitemv25]

Sounds like you're trying to map between the MADIS identifiers and WMO/SYNOP identifiers. This might help (not sure how up to date it is) -- NCEI is having a bad day, its taking forever or failing to load pages, and several datasets are down, so can't do much more research on that right now:

https://weather.rap.ucar.edu/surface/stations.txt

The WMO/SYNOP ID can also be found in the Station Information on MesoWest.

There's probably somewhere with a better mapping, but Google isn't helping much.

[mattdm]

See the Applied Climate Information System (ACIS) Web Services API, documented at https://www.rcc-acis.org/docs_webservices.html.