Stale/outdated gridpoint and zone forecasts from some Akamai servers #329

ktrue · 2020-06-24T20:58:57Z

ktrue
Jun 24, 2020

On some requests to gridpoint and/or zone forecasts like:

https://api.weather.gov/gridpoints/MTR/97,101/forecast
https://api.weather.gov/zones/forecast/CAZ513/forecast

very old/stale data will be returned from Akamai servers.
Currently, it seems to be happening with responses from 23.200.224.17, and 184.84.228.8 but intermittently.

A response from 23.200.224.17 for https://api.weather.gov/gridpoints/MTR/97,101/forecast returned headers of

HTTP/1.1 200 OK
Server: nginx/1.16.1
Content-Type: application/ld+json
Last-Modified: Wed, 01 Jul 2020 01:00:00 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Feature-Flags
X-Request-ID: 659ede63-d703-4419-b770-262a40c700ce
X-Correlation-ID: 659ede63-d703-4419-b770-262a40c700ce
X-Server-ID: vm-lnx-nids-apiapp6.ncep.noaa.gov
Cache-Control: public, max-age=883, s-maxage=3600
Expires: Wed, 24 Jun 2020 20:59:18 GMT
Date: Wed, 24 Jun 2020 20:44:35 GMT
Content-Length: 9164
Connection: keep-alive
Vary: Accept,Feature-Flags,Accept-Language
Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload

and the data was 23:43:12 (h:m:s) old at the time.

A followon query to https://api.weather.gov/zones/forecast/CAZ513/forecast answered by 23.200.224.17 returned headers of

HTTP/1.1 200 OK
Server: nginx/1.16.1
Content-Type: application/ld+json
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Feature-Flags
X-Request-ID: 486634f2-df51-45e9-a5f6-ab7311ce5e84
X-Correlation-ID: 486634f2-df51-45e9-a5f6-ab7311ce5e84
X-Server-ID: vm-bldr-nids-apiapp1.ncep.noaa.gov
Cache-Control: public, max-age=216, s-maxage=120
Expires: Wed, 24 Jun 2020 20:48:11 GMT
Date: Wed, 24 Jun 2020 20:44:35 GMT
Content-Length: 3328
Connection: keep-alive
Vary: Accept,Feature-Flags,Accept-Language
Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload

and actual data that had updated: at 127:43:35 (h:m:s) old.

It appears that not all the Akamai cache servers are regularly refreshing the gridpoint and/or forecast zone forecasts.

StephenClouse · 2020-06-24T21:26:31Z

StephenClouse
Jun 24, 2020
Maintainer

We are actively working this issue but still trying to narrow it down.

Something that would help us is adding this header to your request:

Pragma: akamai-x-cache-on, akamai-x-get-request-id

This will give some additional details in the response headers that we can pass along.

x-akamai-request-id: 22a92082.59b1977
x-cache: TCP_REFRESH_MISS from a23-43-57-214.deploy.akamaitechnologies.com (AkamaiGHost/10.0.5-29826511) (S)

If you can provide those in any follow up here it would be very helpful to us.

0 replies

ktrue · 2020-06-24T22:21:31Z

ktrue
Jun 24, 2020
Author

I've updated my cURL request in PHP to have

    curl_setopt($ch, CURLOPT_HTTPHEADER, // request LD-JSON format
    array(
      "Accept: application/ld+json",
      "Cache-control: no-cache",
      "Pragma: akamai-x-cache-on, akamai-x-get-request-id"
    ));

Right now, I seem to have api.weather.gov responses from 23.200.224.17 and that Akamai cache is working fine (current data).
The headers from that server are showing

HTTP/1.1 200 OK
Server: nginx/1.16.1
Content-Type: application/ld+json
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Feature-Flags
X-Request-ID: cc6acbf5-069f-4b84-9f95-ab56494f2d1a
X-Correlation-ID: cc6acbf5-069f-4b84-9f95-ab56494f2d1a
X-Server-ID: vm-bldr-nids-apiapp5.ncep.noaa.gov
Cache-Control: public, max-age=79982, s-maxage=120
Expires: Thu, 25 Jun 2020 20:32:25 GMT
Date: Wed, 24 Jun 2020 22:19:23 GMT
Content-Length: 2270
X-Cache: TCP_MEM_HIT from a23-198-5-54.deploy.akamaitechnologies.com (AkamaiGHost/10.0.4-29780773) (-)
Connection: keep-alive
Vary: Accept,Feature-Flags,Accept-Language
Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
X-Akamai-Request-ID: 5e888c80

I'll keep watching for more failure.. seems to happen in the mornings.
Thanks for the assistance.

0 replies

ktrue · 2020-06-24T22:45:13Z

ktrue
Jun 24, 2020
Author

Got a stale one:

<!-- curl fetching 'https://api.weather.gov/gridpoints/MTR/97,101/forecast' -->
<!-- HTTP stats:  RC=200 dest=104.123.204.17 port=443 (from sce=192.168.1.104)
      Times: dns=0.060 conn=0.087 pxfer=0.116 get=0.331 total=0.448 secs -->
<!-- headers returned:
HTTP/2 200 
server: nginx/1.16.1
content-type: application/ld+json
last-modified: Wed, 01 Jul 2020 01:00:00 GMT
access-control-allow-origin: *
access-control-allow-headers: Feature-Flags
x-request-id: 659ede63-d703-4419-b770-262a40c700ce
x-correlation-id: 659ede63-d703-4419-b770-262a40c700ce
x-server-id: vm-lnx-nids-apiapp6.ncep.noaa.gov
x-akamai-request-id: 3ecc0321.34133166.2c51c365.3a81d32
cache-control: public, max-age=860, s-maxage=3600
expires: Wed, 24 Jun 2020 22:56:48 GMT
date: Wed, 24 Jun 2020 22:42:28 GMT
content-length: 9164
x-cache: TCP_REFRESH_MISS from a104-124-1-30.deploy.akamaitechnologies.com (AkamaiGHost/10.0.5-29826511) (S)
vary: Accept,Feature-Flags,Accept-Language
strict-transport-security: max-age=31536000 ; includeSubDomains ; preload
 -->
<!-- loaded point-forecast https://api.weather.gov/gridpoints/MTR/97,101/forecast - 9951 bytes -->
<!-- wrote cache file ./cache/forecast-CAZ513-0-json.txt -->
<!-- point forecast more than 18hrs old (age h:m:s is 25:41:05) updated:'2020-06-23T21:01:23+00:00' .. use Zone forecast instead -->

0 replies

ktrue · 2020-06-24T22:48:22Z

ktrue
Jun 24, 2020
Author

And another sample using a different gridpoint (RC=500 this time)

<!-- curl fetching 'https://api.weather.gov/gridpoints/OAX/82,59/forecast' -->
<!-- HTTP stats:  RC=500 dest=184.27.36.18 port=443 (from sce=192.168.1.104)
      Times: dns=0.001 conn=0.027 pxfer=0.074 get=0.623 total=0.697 secs -->
<!-- headers returned:
HTTP/2 500 
server: nginx/1.16.1
content-type: application/problem+json
access-control-allow-origin: *
access-control-allow-headers: Feature-Flags
x-request-id: 23ed1dff-0268-42ce-b1f9-dc2a82d0c35f
x-correlation-id: 23ed1dff-0268-42ce-b1f9-dc2a82d0c35f
x-server-id: vm-bldr-nids-apiapp2.ncep.noaa.gov
pragma: no-cache
content-length: 325
x-akamai-request-id: 7c212e9.1b7f81b.3aaf9c90
cache-control: private, must-revalidate, max-age=865
expires: Wed, 24 Jun 2020 23:00:36 GMT
date: Wed, 24 Jun 2020 22:46:11 GMT
x-cache: TCP_MISS from a184-26-53-199.deploy.akamaitechnologies.com (AkamaiGHost/10.0.4-29780773) (-)
vary: Accept,Feature-Flags,Accept-Language
strict-transport-security: max-age=31536000 ; includeSubDomains ; preload
 -->

0 replies

StephenClouse · 2020-06-25T00:19:33Z

StephenClouse
Jun 25, 2020
Maintainer

Awesome, thank you so much.

This is a weird one but I think I have an idea what's going on now. The good news is we might have just fixed it accidentally while addressing another issue. I'm trying to get these bad objects purged out of Akamai and hopefully I'm right and they don't return.

0 replies

ktrue · 2020-06-25T01:46:28Z

ktrue
Jun 25, 2020
Author

Here's another one: -- this time from 104.123.204.17

<!-- curl fetching 'https://api.weather.gov/gridpoints/MTR/97,101/forecast' -->
<!-- HTTP stats:  RC=200 dest=104.123.204.17 port=443 (from sce=192.168.1.104)
      Times: dns=0.060 conn=0.087 pxfer=0.116 get=0.331 total=0.448 secs -->
<!-- headers returned:
HTTP/2 200 
server: nginx/1.16.1
content-type: application/ld+json
last-modified: Wed, 01 Jul 2020 01:00:00 GMT
access-control-allow-origin: *
access-control-allow-headers: Feature-Flags
x-request-id: 659ede63-d703-4419-b770-262a40c700ce
x-correlation-id: 659ede63-d703-4419-b770-262a40c700ce
x-server-id: vm-lnx-nids-apiapp6.ncep.noaa.gov
x-akamai-request-id: 3ecc0321.34133166.2c51c365.3a81d32
cache-control: public, max-age=860, s-maxage=3600
expires: Wed, 24 Jun 2020 22:56:48 GMT
date: Wed, 24 Jun 2020 22:42:28 GMT
content-length: 9164
x-cache: TCP_REFRESH_MISS from a104-124-1-30.deploy.akamaitechnologies.com (AkamaiGHost/10.0.5-29826511) (S)
vary: Accept,Feature-Flags,Accept-Language
strict-transport-security: max-age=31536000 ; includeSubDomains ; preload
 -->
<!-- loaded point-forecast https://api.weather.gov/gridpoints/MTR/97,101/forecast - 9951 bytes -->
<!-- wrote cache file ./cache/forecast-CAZ513-0-json.txt -->
<!-- point forecast more than 18hrs old (age h:m:s is 25:41:05) updated:'2020-06-23T21:01:23+00:00' .. use Zone forecast instead -->

0 replies

StephenClouse · 2020-06-25T01:56:08Z

StephenClouse
Jun 25, 2020
Maintainer

Was that from earlier today? The date suggests it was.

We flushed Akamai at 6/25 0100 UTC and I'm hoping that resolves this.

Please let me know if you are still seeing this with requests after that point.

0 replies

ktrue · 2020-06-25T02:06:06Z

ktrue
Jun 25, 2020
Author

Ahh.,. you're correct. Thought I'd done a shift-reload to refresh the page, but apparently got the old page from earlier in the day.

A fresh fetch shows correctly:

<!-- curl fetching 'https://api.weather.gov/gridpoints/MTR/97,101/forecast' -->
<!-- HTTP stats:  RC=200 dest=23.200.224.17 port=443 (from sce=74.208.252.86)
      Times: dns=0.061 conn=0.075 pxfer=0.110 get=0.038 total=0.148 secs -->
<!-- headers returned:
HTTP/1.1 200 OK
Server: nginx/1.16.1
Content-Type: application/ld+json
Last-Modified: Wed, 24 Jun 2020 22:01:38 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Feature-Flags
X-Request-ID: 96c4c23f-90c2-4fe4-b103-e87e3329be78
X-Correlation-ID: 96c4c23f-90c2-4fe4-b103-e87e3329be78
X-Server-ID: vm-bldr-nids-apiapp11.ncep.noaa.gov
Cache-Control: public, max-age=225, s-maxage=3600
Expires: Thu, 25 Jun 2020 02:03:57 GMT
Date: Thu, 25 Jun 2020 02:00:12 GMT
Content-Length: 9084
X-Cache: TCP_REFRESH_HIT from a23-198-5-54.deploy.akamaitechnologies.com (AkamaiGHost/10.0.4-29780773) (S)
Connection: keep-alive
Vary: Accept,Feature-Flags,Accept-Language
Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
X-Akamai-Request-ID: 5fff3cd7
 -->
<!-- loaded point-forecast https://api.weather.gov/gridpoints/MTR/97,101/forecast - 9873 bytes -->
<!-- wrote cache file ./cache/forecast-CAZ513-0-json.txt -->
<!-- point forecast processing -->

<!-- forecast updated '2020-06-24T22:01:38+00:00' (Wed, 24-Jun-2020 3:01pm PDT) for lat,long or zone 'POLYGON((-122.0385572 37.275548,-122.0330058 37.2536879,-122.005479 37.2581132,-122.011025 37.2799738,-122.0385572 37.275548))' -->

I'm hoping your cache flush will do the trick.. I'll keep checking on it.
Thank you very much for your responsive handling of the issue -- a large number of folks use my script for NWS forecast displays on their personal weather websites, and I've been fielding a bunch of questions about the stale forecast issues for a couple of weeks now.
Best regards,
Ken

0 replies

StephenClouse · 2020-06-25T02:21:19Z

StephenClouse
Jun 25, 2020
Maintainer

Yeah, thanks for all that, and keep us posted if you see more.

The root issue is with an internal service that decodes the forecast grids from the field offices. It seems to be randomly spitting out bad data, which is how you saw a forecast that was last modified in the future. Unfortunately a full fix for this is going to take some time.

The last hotfix added some sanity checking to what we pull from this internal service, so you may see 500 errors occasionally as this code does its job, but it will hopefully stop totally bogus data from making it to the edge and getting stuck in Akamai like this.

0 replies

duncanwerner · 2020-06-25T19:34:45Z

duncanwerner
Jun 25, 2020

I just got a stale page. I've been getting them intermittently for the last week or so. I only just found this issue, so I don't have cache headers, but I will add them and check.

0 replies

duncanwerner · 2020-06-25T19:45:52Z

duncanwerner
Jun 25, 2020

See below. This was retrieved about 5 minutes before the timestamp on this comment (it's many hours out of date). Once I start getting cached data I can get it pretty reliably, I assume akamai is pinning me to a particular endpoint.

Request

> GET /gridpoints/MTR/88,126/forecast HTTP/2
> Host: api.weather.gov
> User-Agent: [omitted, but I am sending a UA and an email address]
> Accept: application/ld+json
> Pragma: akamai-x-cache-on, akamai-x-get-request-id
>

Response headers

< HTTP/2 200
< server: nginx/1.16.1
< content-type: application/ld+json
< last-modified: Thu, 25 Jun 2020 11:02:42 GMT
< access-control-allow-origin: *
< access-control-allow-headers: Feature-Flags
< x-request-id: b2365b48-f631-4441-a7f9-a4a89048e3e8
< x-correlation-id: b2365b48-f631-4441-a7f9-a4a89048e3e8
< x-server-id: vm-bldr-nids-apiapp6.ncep.noaa.gov
< cache-control: public, max-age=460, s-maxage=3600
< expires: Thu, 25 Jun 2020 19:45:03 GMT
< date: Thu, 25 Jun 2020 19:37:23 GMT
< content-length: 9392
< x-cache: TCP_MEM_HIT from a23-1-237-21.deploy.akamaitechnologies.com (AkamaiGHost/10.0.5-29826511) (-)
< vary: Accept,Feature-Flags,Accept-Language
< strict-transport-security: max-age=31536000 ; includeSubDomains ; preload
< x-akamai-request-id: 82b5211
<

data

{
    "@context": {
        "@version": "1.1",
        "wx": "https://api.weather.gov/ontology#",
        "geo": "http://www.opengis.net/ont/geosparql#",
        "unit": "http://codes.wmo.int/common/unit/",
        "@vocab": "https://api.weather.gov/ontology#"
    },
    "geometry": "POLYGON((-122.4263966 37.7812349,-122.4207393 37.7594296,-122.3931757 37.7638958,-122.3988275 37.7857015,-122.4263966 37.7812349))",
    "updated": "2020-06-25T11:02:42+00:00",
    "units": "us",
    "forecastGenerator": "BaselineForecastGenerator",
    "generatedAt": "2020-06-25T11:30:05+00:00",
    "updateTime": "2020-06-25T11:02:42+00:00",
    "validTimes": "2020-06-25T05:00:00+00:00/P7DT20H",
    "elevation": {
        "value": 39.928800000000003,
        "unitCode": "unit:m"
    },
    "periods": [
        {
            "number": 1,
            "name": "Overnight",
            "startTime": "2020-06-25T04:00:00-07:00",
            "endTime": "2020-06-25T06:00:00-07:00",
            "isDaytime": false,
            "temperature": 56,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "7 mph",
            "windDirection": "SW",
            "icon": "https://api.weather.gov/icons/land/night/bkn?size=medium",
            "shortForecast": "Mostly Cloudy",
            "detailedForecast": "Mostly cloudy, with a low around 56. Southwest wind around 7 mph."
        },
        {
            "number": 2,
            "name": "Thursday",
            "startTime": "2020-06-25T06:00:00-07:00",
            "endTime": "2020-06-25T18:00:00-07:00",
            "isDaytime": true,
            "temperature": 69,
            "temperatureUnit": "F",
            "temperatureTrend": "falling",
            "windSpeed": "6 to 14 mph",
            "windDirection": "WSW",
            "icon": "https://api.weather.gov/icons/land/day/sct?size=medium",
            "shortForecast": "Mostly Sunny",
            "detailedForecast": "Mostly sunny. High near 69, with temperatures falling to around 66 in the afternoon. West southwest wind 6 to 14 mph, with gusts as high as 18 mph."
        },
        {
            "number": 3,
            "name": "Thursday Night",
            "startTime": "2020-06-25T18:00:00-07:00",
            "endTime": "2020-06-26T06:00:00-07:00",
            "isDaytime": false,
            "temperature": 58,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "9 to 14 mph",
            "windDirection": "WSW",
            "icon": "https://api.weather.gov/icons/land/night/bkn?size=medium",
            "shortForecast": "Mostly Cloudy",
            "detailedForecast": "Mostly cloudy, with a low around 58. West southwest wind 9 to 14 mph, with gusts as high as 18 mph."
        },
        {
            "number": 4,
            "name": "Friday",
            "startTime": "2020-06-26T06:00:00-07:00",
            "endTime": "2020-06-26T18:00:00-07:00",
            "isDaytime": true,
            "temperature": 69,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "9 to 15 mph",
            "windDirection": "WSW",
            "icon": "https://api.weather.gov/icons/land/day/sct?size=medium",
            "shortForecast": "Mostly Sunny",
            "detailedForecast": "Mostly sunny, with a high near 69. West southwest wind 9 to 15 mph, with gusts as high as 20 mph."
        },
        {
            "number": 5,
            "name": "Friday Night",
            "startTime": "2020-06-26T18:00:00-07:00",
            "endTime": "2020-06-27T06:00:00-07:00",
            "isDaytime": false,
            "temperature": 56,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "10 to 14 mph",
            "windDirection": "WSW",
            "icon": "https://api.weather.gov/icons/land/night/sct?size=medium",
            "shortForecast": "Partly Cloudy",
            "detailedForecast": "Partly cloudy, with a low around 56. West southwest wind 10 to 14 mph, with gusts as high as 18 mph."
        },
        {
            "number": 6,
            "name": "Saturday",
            "startTime": "2020-06-27T06:00:00-07:00",
            "endTime": "2020-06-27T18:00:00-07:00",
            "isDaytime": true,
            "temperature": 67,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "12 to 17 mph",
            "windDirection": "WSW",
            "icon": "https://api.weather.gov/icons/land/day/sct?size=medium",
            "shortForecast": "Mostly Sunny",
            "detailedForecast": "Mostly sunny, with a high near 67."
        },
        {
            "number": 7,
            "name": "Saturday Night",
            "startTime": "2020-06-27T18:00:00-07:00",
            "endTime": "2020-06-28T06:00:00-07:00",
            "isDaytime": false,
            "temperature": 54,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "12 to 16 mph",
            "windDirection": "WSW",
            "icon": "https://api.weather.gov/icons/land/night/sct?size=medium",
            "shortForecast": "Partly Cloudy",
            "detailedForecast": "Partly cloudy, with a low around 54."
        },
        {
            "number": 8,
            "name": "Sunday",
            "startTime": "2020-06-28T06:00:00-07:00",
            "endTime": "2020-06-28T18:00:00-07:00",
            "isDaytime": true,
            "temperature": 68,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "9 to 21 mph",
            "windDirection": "WSW",
            "icon": "https://api.weather.gov/icons/land/day/wind_sct?size=medium",
            "shortForecast": "Mostly Sunny",
            "detailedForecast": "Mostly sunny, with a high near 68."
        },
        {
            "number": 9,
            "name": "Sunday Night",
            "startTime": "2020-06-28T18:00:00-07:00",
            "endTime": "2020-06-29T06:00:00-07:00",
            "isDaytime": false,
            "temperature": 56,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "6 to 21 mph",
            "windDirection": "W",
            "icon": "https://api.weather.gov/icons/land/night/wind_few?size=medium",
            "shortForecast": "Mostly Clear",
            "detailedForecast": "Mostly clear, with a low around 56."
        },
        {
            "number": 10,
            "name": "Monday",
            "startTime": "2020-06-29T06:00:00-07:00",
            "endTime": "2020-06-29T18:00:00-07:00",
            "isDaytime": true,
            "temperature": 71,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "5 to 15 mph",
            "windDirection": "W",
            "icon": "https://api.weather.gov/icons/land/day/few?size=medium",
            "shortForecast": "Sunny",
            "detailedForecast": "Sunny, with a high near 71."
        },
        {
            "number": 11,
            "name": "Monday Night",
            "startTime": "2020-06-29T18:00:00-07:00",
            "endTime": "2020-06-30T06:00:00-07:00",
            "isDaytime": false,
            "temperature": 57,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "3 to 15 mph",
            "windDirection": "WSW",
            "icon": "https://api.weather.gov/icons/land/night/few?size=medium",
            "shortForecast": "Mostly Clear",
            "detailedForecast": "Mostly clear, with a low around 57."
        },
        {
            "number": 12,
            "name": "Tuesday",
            "startTime": "2020-06-30T06:00:00-07:00",
            "endTime": "2020-06-30T18:00:00-07:00",
            "isDaytime": true,
            "temperature": 72,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "3 to 14 mph",
            "windDirection": "SW",
            "icon": "https://api.weather.gov/icons/land/day/few?size=medium",
            "shortForecast": "Sunny",
            "detailedForecast": "Sunny, with a high near 72."
        },
        {
            "number": 13,
            "name": "Tuesday Night",
            "startTime": "2020-06-30T18:00:00-07:00",
            "endTime": "2020-07-01T06:00:00-07:00",
            "isDaytime": false,
            "temperature": 57,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "3 to 14 mph",
            "windDirection": "WSW",
            "icon": "https://api.weather.gov/icons/land/night/few?size=medium",
            "shortForecast": "Mostly Clear",
            "detailedForecast": "Mostly clear, with a low around 57."
        },
        {
            "number": 14,
            "name": "Wednesday",
            "startTime": "2020-07-01T06:00:00-07:00",
            "endTime": "2020-07-01T18:00:00-07:00",
            "isDaytime": true,
            "temperature": 72,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "3 to 14 mph",
            "windDirection": "SW",
            "icon": "https://api.weather.gov/icons/land/day/few?size=medium",
            "shortForecast": "Sunny",
            "detailedForecast": "Sunny, with a high near 72."
        }
    ]
}

0 replies

StephenClouse · 2020-06-25T19:56:33Z

StephenClouse
Jun 25, 2020
Maintainer

@duncanwerner This one does not appear to be a bug. MTR has not actually issued a new forecast grid since 1107Z.

0 replies

duncanwerner · 2020-06-25T20:00:05Z

duncanwerner
Jun 25, 2020

This is the correct current forecast (which I got by changing headers):

{
    "@context": {
        "@version": "1.1",
        "wx": "https://api.weather.gov/ontology#",
        "geo": "http://www.opengis.net/ont/geosparql#",
        "unit": "http://codes.wmo.int/common/unit/",
        "@vocab": "https://api.weather.gov/ontology#"
    },
    "geometry": "POLYGON((-122.4263966 37.7812349,-122.4207393 37.7594296,-122.3931757 37.7638958,-122.3988275 37.7857015,-122.4263966 37.7812349))",
    "updated": "2020-06-25T11:02:42+00:00",
    "units": "us",
    "forecastGenerator": "BaselineForecastGenerator",
    "generatedAt": "2020-06-25T16:19:02+00:00",
    "updateTime": "2020-06-25T11:02:42+00:00",
    "validTimes": "2020-06-25T05:00:00+00:00/P7DT20H",
    "elevation": {
        "value": 39.928800000000003,
        "unitCode": "unit:m"
    },
    "periods": [
        {
            "number": 1,
            "name": "Today",
            "startTime": "2020-06-25T09:00:00-07:00",
            "endTime": "2020-06-25T18:00:00-07:00",
            "isDaytime": true,
            "temperature": 69,
            "temperatureUnit": "F",
            "temperatureTrend": "falling",
            "windSpeed": "6 to 14 mph",
            "windDirection": "WSW",
            "icon": "https://api.weather.gov/icons/land/day/sct?size=medium",
            "shortForecast": "Mostly Sunny",
            "detailedForecast": "Mostly sunny. High near 69, with temperatures falling to around 66 in the afternoon. West southwest wind 6 to 14 mph, with gusts as high as 18 mph."
        },
        {
            "number": 2,
            "name": "Tonight",
            "startTime": "2020-06-25T18:00:00-07:00",
            "endTime": "2020-06-26T06:00:00-07:00",
            "isDaytime": false,
            "temperature": 58,
            "temperatureUnit": "F",
            "temperatureTrend": null,
            "windSpeed": "9 to 14 mph",
            "windDirection": "WSW",
            "icon": "https://api.weather.gov/icons/land/night/bkn?size=medium",
            "shortForecast": "Mostly Cloudy",
            "detailedForecast": "Mostly cloudy, with a low around 58. West southwest wind 9 to 14 mph, with gusts as high as 18 mph."
        },
        {
            "number": 3,
            "name": "Friday",
            "startTime": "2020-06-26T06:00:00-07:00",
            "endTime": "2020-06-26T18:00:00-07:00",
            "isDaytime": true,
            "temperature": 69,
            "temperatureUnit": "F",
            "temperatureTrend": "falling",
            "windSpeed": "9 to 15 mph",
            "windDirection": "WSW",
            "icon": "https://api.weather.gov/icons/land/day/sct?size=medium",
            "shortForecast": "Mostly Sunny",
            "detailedForecast": "Mostly sunny. High near 69, with temperatures falling to around 67 in the afternoon. West southwest wind 9 to 15 mph, with gusts as high as 20 mph."
        },

(truncated this time)

0 replies

StephenClouse · 2020-06-25T20:07:21Z

StephenClouse
Jun 25, 2020
Maintainer

@duncanwerner Oh, sorry, now I understand what you're pointing at.

Pretty sure that's a bug. It's correctly recognizing that the forecast grid hasn't changed but glossing over the fact that we've rolled over into the morning period. Something I failed to account for because they usually re-issue grids well before this would be noticed.

0 replies

duncanwerner · 2020-06-25T20:30:57Z

duncanwerner
Jun 25, 2020

The first time I had this problem I was getting data that was more than 24 hours old. It's possible there was no new forecast in that period, but it seems unlikely. It might be that that was the issue you were discussing at the top of this thread, and that you fixed it.

Unfortunately my first attempt to debug it cleared it up, so I don't have any useful records. More recently I have gotten a couple of forecasts like the one I sent, which show an unexpected period 1 in the past. It's not a big deal to work around it, although it is confusing. (I will admit I had been ignoring timestamps and just showing periods in order).

API is great, btw, thanks for maintaining it!

0 replies

onedotover · 2020-12-30T19:18:18Z

onedotover
Dec 30, 2020

I'm getting either stale results on this endpoint: https://api.weather.gov/gridpoints/PDT/23,41/forecast or the following response:
{
"correlationId": "1557f17",
"title": "Forecast Grid Expired",
"type": "https://api.weather.gov/problems/ForecastGridExpired",
"status": 503,
"detail": "The requested forecast grid was issued 2020-10-22T10:31:21+00:00 and has expired.",
"instance": "https://api.weather.gov/requests/1557f17"
}

I've noticed that I get updated information locally, but then stale information from my server. Additionally, this link does not have stale results:
https://mobile.weather.gov/index.php?lat=44.0274&lon=-121.6494

Detailed info for this gridpoint was current up till yesterday and now it's pulling in data from October 22nd.
https://api.weather.gov/gridpoints/PDT/23,41

Can I provide any detailed information to help with this? Is there any workaround?

0 replies

KCRotary · 2021-01-07T14:57:30Z

KCRotary
Jan 7, 2021

I'm having the same issue with location Kearney, MO. The data returned is from Christmas 2020.

0 replies

colonieweather · 2021-01-07T16:05:15Z

colonieweather
Jan 7, 2021

Ditto. I've given up on this API. NWS doesn't seem to be rushing to fix, I'll pay a little extra for an API that supported.

0 replies

scadergit · 2021-01-07T17:23:58Z

scadergit
Jan 7, 2021
Maintainer

Your frustration is valid. The small team has been rushing non-stop since we inherited the project, and are about to release a major upgrade that will stabilize the entire platform. After that, we should be able to address these critical issues more quickly.

0 replies

StephenClouse · 2021-01-07T17:27:34Z

StephenClouse
Jan 7, 2021
Maintainer

These specific issues should be reported as operational outages as described here.

0 replies

onedotover · 2021-01-07T17:31:04Z

onedotover
Jan 7, 2021

Thanks @scadergit and @StephenClouse. I appreciate your efforts on this and the fact that this API is available. Let me know if there's anything I can do or test with the upgrade.

0 replies

JJKraw · 2021-01-07T18:47:02Z

JJKraw
Jan 7, 2021

I also appreciate the effort and am glad to help in any testing.

0 replies

colonieweather · 2021-01-07T19:00:11Z

colonieweather
Jan 7, 2021

I sincerely apologize for my earlier post. It was uncalled for, especially being a programmer myself. I do understand issues are being worked on and, I too, appreciate the effort. Wish you guys had more help over there.

0 replies

StephenClouse · 2021-01-07T19:11:23Z

StephenClouse
Jan 7, 2021
Maintainer

I understand all the frustration. We're frustrated too. We know the root cause of this but it requires developing a complete replacement for a particular backend service we inherited. Right now we are working to meet commitments the United States has made to the World Meteorological Organization and its member states. So we are just having to limp along with that broken service for now and deal with it as needed. We are very close to the end of that WMO effort and then fixing this for good is a high priority in our 2021 development cycle.

As noted previously, please report these issues to the NCO ops team. They do have ways to fix things temporarily.

1 reply

sdnwser Apr 21, 2021

Stephen: that's a pretty funny and honest response! The weather API is funded by the US tax payer, as anything *.gov is. However, it appears that non tax payers are prioritized ahead of the tax payers. I got an idea, the stale data issue was all observed by WMO users. Can we get bumped up to the top of the priority list too? ;-)

ghankerson · 2021-06-30T14:31:29Z

ghankerson
Jun 30, 2021

Certainly understand the issues preventing a solution here. 2021 is 50% gone could we get an update even if it's just things are in the same place?

Thank you

5 replies

ktrue Jun 30, 2021
Author

The issue still exists, and (for me) appears to be the same Akamai server at 184.87.164.14 that is returning stale (>18hrs old) gridpoint forecasts. I've implemented a browser for the error logs at https://saratoga-weather.org/NWS-analyze/ which shows all the failures (by date) with all the additional headers in the requests needed to diagnose. Since this 'stale' issue has gone on for a year now, I'd hoped to see some progress at fixing the NWS->Akamai cache issues, but so far, no joy.

scadergit Jun 30, 2021
Maintainer

The good news is we kicked off the sprint to refactor the service that @StephenClouse mentioned which directly impacts anything related to the netcdf grids. The bad news is that it immediately got derailed supporting the team completing the QA process for the other project he mentioned. On the plus side, the other effort puts the entire program on better footing. We're not responsible for deploy, but I estimate the other project should be released in a few months, and then the refactor for the grid service a few months after that.

ghankerson Jun 30, 2021

This is good to know. Thanks so much

jah1007 Jul 1, 2021

The QA effort backing things up should be wrapped up by mid-September.

ryanfox Oct 4, 2021

Was this issue resolved? I haven’t been checking extensively, but haven’t encountered out-of-date data recently.

sdnwser · 2021-06-30T16:55:55Z

sdnwser
Jun 30, 2021

Thank you for the update!

…

On Wed, Jun 30, 2021 at 9:20 AM Jared Holmberg ***@***.***> wrote: The good news is we kicked off the sprint to refactor the service that @StephenClouse <https://github.com/StephenClouse> mentioned which directly impacts anything related to the netcdf grids. The bad news is that it immediately got derailed supporting the team completing the QA process for the other project he mentioned. On the plus side, the other effort puts the entire program on better footing. We're not responsible for deploy, but I estimate the other project should be released in a few months, and then the refactor for the grid service a few months after that. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#329 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATYUQZBE55UXZ2J6PBNN2LTTVM73RANCNFSM43JQCMEA> .

0 replies

sdnwser · 2021-10-10T01:38:39Z

sdnwser
Oct 10, 2021

Is there an update on fixing this bug? As of 10/9/2021, I am still getting stale data.

1 reply

StephenClouse Oct 10, 2021
Maintainer

Unfortunately we do not have an update. Our development schedule got hit with a nuke.

We are looking at some stopgap measures but no ETA.

sdnwser · 2021-10-10T04:48:03Z

sdnwser
Oct 10, 2021

Thanks for the update!

…

On Oct 9, 2021, at 9:39 PM, Stephen Clouse ***@***.***> wrote: Unfortunately we do not have an update. Our development schedule got hit with a nuke. We are looking at some stopgap measures but no ETA. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

0 replies

ryanfox · 2022-01-13T16:42:05Z

ryanfox
Jan 13, 2022

Is there a timeline for a fix on this yet? It would be great to have functioning again.

0 replies

ktrue · 2022-01-25T21:20:51Z

ktrue
Jan 25, 2022
Author

I'd reported it to nco.ops last year as INC0272004 . They've now merged in another (later one) from INC0322796 onto this Master ticket.

2022-01-25 09:31:25 EST - Guest Additional commentsreply from: [email protected] this issue has been open since last July and there is no progress on the solution?Can we get an update on the issue and maybe an ETA on the fix?
--
2022-01-25 07:06:12 EST - John PercyAdditional commentsAt 1200z, merging POC from ticket INC0322796 onto this Master ticket. [email protected] added. //JOHN//
--
2022-01-25 06:07:57 EST - John PercyAdditional comments1-24-22 @ 1435z, IDP Eng, John posted: Thank you for the additional feedback.  --  The issue is with intermittent future 'Last-Modified' headers  coming out of the api.weather.gov/grid points content, which is causing  caching issues.  --  We are testing a workaround and plan to implement it in  operations next week (2/2), which will adjust the cache scheme until we  can implement the permanent solution later        this year. ---    Caching workaround now planned for 2/2.     This is the same issue reported under INC0272004, therefore we will  close this ticket and add the customers on distro list to ticket:  INC0321457 because it was opened in July, so has more history and people  already on the distro list. //JOHN//
--
2022-01-25 05:59:06 EST - John PercyAdditional commentsAt  1055z, merging ticket INC0321457 into this ticket.  Updating distro  list to add customers: [email protected],  [email protected]. //JOHN//
--
2022-01-18 06:16:10 EST - John PercyAdditional commentsOn  1-18-22, this issue is still pending development of a software patch to  fully resolve stale data issue.  There have been inroads made, but work  continues. //JOHN//
--
2021-12-01 13:01:29 EST - Leonardo DaSilvaAdditional commentsAt 17:15z received updates from tier 2:  We are aware of the stale data issue and our development team is in  the process of fixing this issue. Unfortunately, it is a large effort  which we do not yet have a date for completion.  (LRS)

Maybe, the end is in sight? I hope so.. been a long slog.

0 replies

Stale/outdated gridpoint and zone forecasts from some Akamai servers #329

Uh oh!

Uh oh!

Replies: 73 comments · 7 replies

Uh oh!

StephenClouse Jun 24, 2020 Maintainer

Uh oh!

ktrue Jun 24, 2020 Author

Uh oh!

ktrue Jun 24, 2020 Author

Uh oh!

ktrue Jun 24, 2020 Author

Uh oh!

StephenClouse Jun 25, 2020 Maintainer

Uh oh!

ktrue Jun 25, 2020 Author

Uh oh!

StephenClouse Jun 25, 2020 Maintainer

Uh oh!

ktrue Jun 25, 2020 Author

Uh oh!

StephenClouse Jun 25, 2020 Maintainer

Uh oh!

Uh oh!

Uh oh!

StephenClouse Jun 25, 2020 Maintainer

Uh oh!

Uh oh!

StephenClouse Jun 25, 2020 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

scadergit Jan 7, 2021 Maintainer

Uh oh!

StephenClouse Jan 7, 2021 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

StephenClouse Jan 7, 2021 Maintainer

Uh oh!

Uh oh!

Uh oh!

ktrue Jun 30, 2021 Author

Uh oh!

scadergit Jun 30, 2021 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 73 comments 7 replies

StephenClouse
Jun 24, 2020
Maintainer

ktrue
Jun 24, 2020
Author

ktrue
Jun 24, 2020
Author

ktrue
Jun 24, 2020
Author

StephenClouse
Jun 25, 2020
Maintainer

ktrue
Jun 25, 2020
Author

StephenClouse
Jun 25, 2020
Maintainer

ktrue
Jun 25, 2020
Author

StephenClouse
Jun 25, 2020
Maintainer

StephenClouse
Jun 25, 2020
Maintainer

StephenClouse
Jun 25, 2020
Maintainer

scadergit
Jan 7, 2021
Maintainer

StephenClouse
Jan 7, 2021
Maintainer

StephenClouse
Jan 7, 2021
Maintainer

ktrue Jun 30, 2021
Author

scadergit Jun 30, 2021
Maintainer