Starting 3-4 months ago, getting old forecast (cache issue?)

[engpok]

I’ve been accessing weather.gov api for more than 5 years. I have been using my own Labview app which uses a “curl” command to request a weather forecast. I request the forecast once an hour. In the past 3-4 months I’ve discovered that many times I get a day old or sometimes two day old forecast. I’m using the following command (from Labview 2011, on MacOS 10.14.6):

curl https://api.weather.gov/gridpoints/LWX/79,75
User-Agent: personal/Labview 2011 (Internal Application, [email protected])

I’m suspecting that cacheing is the issue, but every time I use the same address on Chrome’s browser, I always get the current forecast.

Any idea what I might be doing wrong?

Thanks, in advance

[scadergit]

Will you provide the correlation id for a request that returns a forecast that is old? That will help us determine if the cached reply came from the CDN or origin.

[engpok]

Thanks Jared, OK, I had to do some research into these new terms of “correlation ID” and “CDN”. I added the “verbose” option to my curl command, and I have the following example: - curl command sent on 17 Mar, 0730 eastern time - forecast data shows 15 Mar: "properties": { ***@***.***": "https://api.weather.gov/gridpoints/LWX/79,75", ***@***.***": "wx:Gridpoint", "updateTime": "2021-03-15T11:35:36+00:00", "validTimes": "2021-03-15T05:00:00+00:00/P7DT20H", - correlation ID: . . . Administration; CN=weather.gov * start date: Sep 18 00:00:00 2020 GMT * expire date: Oct 18 12:00:00 2021 GMT . . . < x-request-id: c49de7b8-c895-4d42-be3e-1539858ad0e1 < x-correlation-id: 1085dc59 < x-server-id: vm-bldr-nids-apiapp5.ncep.noaa.gov < cache-control: public, max-age=900, s-maxage=3600 < expires: Wed, 17 Mar 2021 12:01:27 GMT < date: Wed, 17 Mar 2021 11:46:27 GMT Hmmmmm… is this maybe considered a valid forecast since the “validTimes” covers 17 Mar, 0730?

…

On Mar 17, 2021, at 12:22 AM, Jared Holmberg ***@***.***> wrote: Will you provide the correlation id for a request that returns a forecast that is old? That will help us determine if the cached reply came from the CDN or origin. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/weather-gov/api/issues/177#issuecomment-800782476>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIOBA3NQAAHKIABH4VF6E3TEAU7FANCNFSM4ZJTEZYQ>.

[engpok]

Hi Jared, have you found any clues?

[scadergit]

Sorry, I wasn't able to act on this quick enough. Can you provide another, more recent correlation id? I'll look into it soon as you send it over.

[ryanfox]

I believe you’re seeing the same issue that used to live at this URL:

https://github.com/weather-gov/api/issues/71

It seems issues are turned off for this repo now, so it’s 404’d. Here is the wayback machine’s copy:

https://web.archive.org/web/20201029024056/https://github.com/weather-gov/api/issues/71

I encounter this on nearly a daily basis. Hope they can fix it soon.

[engpok]

Thanks Ryan, I was afraid that I wasn’t requesting the weather properly...

…

On Apr 16, 2021, at 9:31 AM, Ryan Fox ***@***.***> wrote: I believe you’re seeing the same issue that used to live at this URL: https://github.com/weather-gov/api/issues/71 <https://github.com/weather-gov/api/issues/71> It seems issues are turned off for this repo now, so it’s 404’d. Here is the wayback machine’s copy: https://web.archive.org/web/20201029024056/https://github.com/weather-gov/api/issues/71 <https://web.archive.org/web/20201029024056/https://github.com/weather-gov/api/issues/71> I encounter this on nearly a daily basis. Hope they can fix it soon. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#224 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIOBA3Y4KB6GU5JO2QXSYLTJA3ZTANCNFSM43BRLREA>.

[engpok]

Thanks Jared, I got a reply from another user who stated that this problem is known… and was identified back in June 2020, so I know folks are looking into this: https://web.archive.org/web/20201029024056/https://github.com/weather-gov/api/issues/71 <https://web.archive.org/web/20201029024056/https://github.com/weather-gov/api/issues/71> I’ll send you a fresh errant data set when I get one. And, if I can help you in any other way to troubleshoot this, let me know. Richard

…

On Apr 16, 2021, at 9:58 AM, Jared Holmberg ***@***.***> wrote: Sorry, I wasn't able to act on this quick enough. Can you provide another, more recent correlation id? I'll look into it soon as you send it over. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#224 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIOBAZWG4BZAZIDA7G37ODTJA7CDANCNFSM43BRLREA>.

[scadergit]

There have been several iterations of this issue, we've addressed a few. A new correlation id would be helpful so I can identify if this was related or another issue.

[ryanfox]

I didn't realize there were multiple issues. If it helps, I am seeing 48+ hour old forecast information right now, X-Correlation-ID is 2fb976e.

[scadergit]

Mutiple layers to the issue...the first was the CDN side, next is the service that feeds the API grid data, which is why finding a local cache MISS on @engpok's request was good as that would be addressed with the current effort.

[engpok]

Here is another stale update that just happened: Actual date: 17 April curl standard output: "properties": { ***@***.***": "https://api.weather.gov/gridpoints/LWX/79,75", ***@***.***": "wx:Gridpoint", "updateTime": "2021-04-16T20:36:20+00:00", "validTimes": "2021-04-16T14:00:00+00:00/P7DT11H", curl Standard Error: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 23.196.33.67... * TCP_NODELAY set * Connected to api.weather.gov (23.196.33.67) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@strength * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): } [221 bytes data] * TLSv1.2 (IN), TLS handshake, Server hello (2): { [102 bytes data] * TLSv1.2 (IN), TLS handshake, Certificate (11): { [4030 bytes data] * TLSv1.2 (IN), TLS handshake, Server key exchange (12): { [333 bytes data] * TLSv1.2 (IN), TLS handshake, Server finished (14): { [4 bytes data] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): } [70 bytes data] * TLSv1.2 (OUT), TLS change cipher, Client hello (1): } [1 bytes data] * TLSv1.2 (OUT), TLS handshake, Finished (20): } [16 bytes data] * TLSv1.2 (IN), TLS change cipher, Client hello (1): { [1 bytes data] * TLSv1.2 (IN), TLS handshake, Finished (20): { [16 bytes data] * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=Maryland; L=College Park; O=National Oceanic and Atmospheric Administration; CN=weather.gov * start date: Mar 22 00:00:00 2021 GMT * expire date: Oct 16 23:59:59 2021 GMT * subjectAltName: host "api.weather.gov" matched cert's "api.weather.gov" * issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7fe53c009400)

GET /gridpoints/LWX/79,75 HTTP/2 Host: api.weather.gov User-Agent: curl/7.54.0 Accept: */*

* Connection state changed (MAX_CONCURRENT_STREAMS updated)! < HTTP/2 200 < server: nginx/1.16.1 < content-type: application/geo+json < last-modified: Tue, 20 Apr 2021 04:00:00 GMT < access-control-allow-origin: * < access-control-allow-headers: Feature-Flags < x-request-id: 24fb82c7-b051-4350-b3ce-ecb801cf8bb9 < x-correlation-id: 55e00a3 < x-server-id: vm-lnx-nids-apiapp9.ncep.noaa.gov < cache-control: public, max-age=825, s-maxage=3600 < expires: Sat, 17 Apr 2021 15:20:23 GMT < date: Sat, 17 Apr 2021 15:06:38 GMT < x-edge-request-id: 25692d6e < vary: Accept,Feature-Flags,Accept-Language < strict-transport-security: max-age=31536000 ; includeSubDomains ; preload < { [2486 bytes data] 100 192k 0 192k 0 0 449k 0 --:--:-- --:--:-- --:--:-- 450k * Connection #0 to host api.weather.gov left intact

…

On Apr 16, 2021, at 5:45 PM, Jared Holmberg ***@***.***> wrote: There have been several iterations of this issue, we've addressed a few. I new correlation id would be helpful so I can identify if this was related or another issue. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#224 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIOBAYE45RRWI2N3PVJU5LTJCVV3ANCNFSM43BRLREA>.

[scadergit]

Am I crazy or does that also show last modified three days from now?

This is a good correlation to have, as a quick review shows the API's local cache was a miss, so it generated a new response. That helps us know where to look.

[engpok]

Yup, in the other thread that Ryan referenced, the same issue was noted… a future “last modified” date. I can modify my code to trap these stale updates and store (and time tag) them if that would be helpful.

…

On Apr 17, 2021, at 11:52 AM, Jared Holmberg ***@***.***> wrote: Am I crazy or does that also show last modified three days from now? This is a good correlation to have, as a quick review shows the API's local cache was a miss, so it generated a new response. That helps us know where to look. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#224 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIOBA3DE4SMS3QRM2FTN5TTJGVFVANCNFSM43BRLREA>.

[engpok]

And here is another example that just recently happened (1700 eastern, 20 April). Note that I’m only getting forecasts once per hour, and only during waking hours. "properties": { ***@***.***": "https://api.weather.gov/gridpoints/LWX/79,75", ***@***.***": "wx:Gridpoint", "updateTime": "2021-04-18T14:33:28+00:00", "validTimes": "2021-04-18T08:00:00+00:00/P7DT17H", "elevation": { ========================================================= % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 23.220.120.15... * TCP_NODELAY set * Connected to api.weather.gov (23.220.120.15) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@strength * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): } [221 bytes data] * TLSv1.2 (IN), TLS handshake, Server hello (2): { [102 bytes data] * TLSv1.2 (IN), TLS handshake, Certificate (11): { [4030 bytes data] * TLSv1.2 (IN), TLS handshake, Server key exchange (12): { [333 bytes data] * TLSv1.2 (IN), TLS handshake, Server finished (14): { [4 bytes data] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): } [70 bytes data] * TLSv1.2 (OUT), TLS change cipher, Client hello (1): } [1 bytes data] * TLSv1.2 (OUT), TLS handshake, Finished (20): } [16 bytes data] * TLSv1.2 (IN), TLS change cipher, Client hello (1): { [1 bytes data] * TLSv1.2 (IN), TLS handshake, Finished (20): { [16 bytes data] * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=Maryland; L=College Park; O=National Oceanic and Atmospheric Administration; CN=weather.gov * start date: Mar 22 00:00:00 2021 GMT * expire date: Oct 16 23:59:59 2021 GMT * subjectAltName: host "api.weather.gov" matched cert's "api.weather.gov" * issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7f9dc0805e00)

GET /gridpoints/LWX/79,75 HTTP/2 Host: api.weather.gov User-Agent: curl/7.54.0 Accept: */*

* Connection state changed (MAX_CONCURRENT_STREAMS updated)! < HTTP/2 200 < server: nginx/1.16.1 < content-type: application/geo+json < last-modified: Sat, 24 Apr 2021 13:00:00 GMT < access-control-allow-origin: * < access-control-allow-headers: Feature-Flags < x-request-id: 3be8554e-17ca-4048-981a-90b500e3fcf6 < x-correlation-id: 1c6c4d50 < x-server-id: vm-lnx-nids-apiapp7.ncep.noaa.gov < cache-control: public, max-age=844, s-maxage=3600 < expires: Tue, 20 Apr 2021 21:14:10 GMT < date: Tue, 20 Apr 2021 21:00:06 GMT < x-edge-request-id: 29509461 < vary: Accept,Feature-Flags,Accept-Language < strict-transport-security: max-age=31536000 ; includeSubDomains ; preload < { [2485 bytes data] 100 200k 0 200k 0 0 462k 0 --:--:-- --:--:-- --:--:-- 462k * Connection #0 to host api.weather.gov left intact

…

On Apr 17, 2021, at 12:00 PM, richard pokrass ***@***.***> wrote: Yup, in the other thread that Ryan referenced, the same issue was noted… a future “last modified” date. I can modify my code to trap these stale updates and store (and time tag) them if that would be helpful. > On Apr 17, 2021, at 11:52 AM, Jared Holmberg ***@***.*** ***@***.***>> wrote: > > > Am I crazy or does that also show last modified three days from now? > > This is a good correlation to have, as a quick review shows the API's local cache was a miss, so it generated a new response. That helps us know where to look. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub <#224 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIOBA3DE4SMS3QRM2FTN5TTJGVFVANCNFSM43BRLREA>. >

[sdnwser]

I have been battling the stale data issue for several months as well. My attempts to contact the ops team resulted in no response. I do however have an ugly solution to the issue. Since this appears to be a problem on their back end caching and providing stale data, I started dithering the forecast grid location requests. So if my proper grid location is (10,10), I make my requests to range from 9 to 11 on both the X and Y grid points. The actual call picks a grid point randomly. This in effect spreads my location requests to a 3 X 3 box. Since each grid location is 2.5 km, I have extended the forecast box to 7.5 X 7.5 km which is fine by me.

And just for reference, I was seeing typical staleness of 12 hours and in a few cases even 48 hours.

[engpok]

That’s clever. I thought about doing the same after I discovered that I got current data from a nearby city right after I had stale data from my preferred location. But, once (out of 20) I got stale data on the alternate city, so I thought that approach would still get stale data sometimes. But, I don’t fully understand this staleness/cache issue: - is it stale because the same user is repeating data requests within a limited time period… regardless of location? - or is stale because the same location gets repeated data requests within a limited time period… regardless of user?

…

On Apr 20, 2021, at 10:48 PM, sdnwser ***@***.***> wrote: I have been battling the stale data issue for several months as well. My attempts to contact the ops team resulted in no response. I do however have an ugly solution to the issue. Since this appears to be a problem on their back end caching and providing stale data, I started dithering the forecast grid location requests. So if my proper grid location is (10,10), I make my requests to range from 9 to 11 on both the X and Y grid points. The actual call picks a grid point randomly. This in effects spreads my location requests to a 3X3 box. Since each grid location is 2.5 km, I have extended the forecast box to 7.5 X 7.5 km which is fine by me. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#224 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIOBA72ZZRHQPBAQI2KUVDTJY4GNANCNFSM43BRLREA>.

[sdnwser]

engpok:
I can't answer it with absolute certainty but it seems to be a combination of the requesting user and the actual request made. I will say that I tried several approaches to force it to retrieve current data:

1. reduced the number of requests to once every 2-3 hours.
2. salted the context with random numbers on top of the basic information they requested.
   Those approaches didn't yield effective results. Dithering the location request seems to be effective but isn't every elegant. Would be nice if this is addressed on the backend. The NWS API is a very nice service!

[engpok]

Thanks, I agree - it’s a great service. So, I’m trying to think of ways to help them troubleshoot this...

…

On Apr 21, 2021, at 9:55 AM, sdnwser ***@***.***> wrote: engpok: I can't answer it with absolute certainty but it seems to be a combination of the requesting user and the actual request made. I will say that I tried several approaches to force it to retrieve current data: reduced the number of requests to once every 2-3 hours. salted the context with random numbers on top of the basic information they requested. Those approaches didn't yield effective results. Dithering the location request seems to be effective but isn't every elegant. Would be nice if this is addressed on the backend. The NWS API is a very nice service! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#224 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIOBA2J3ITKPK6YPOEMVDDTJ3KNXANCNFSM43BRLREA>.

[sdnwser]

I can provide NWS with more than a years worth of time tagged queries in SQL if it would help debug. Heck, I can even start a dedicate SQL capture if that helps them :)

[jfbauer432]

Likely related, On 2021-04-22 14:43:43 (US/Eastern) I fetched https://api.weather.gov/gridpoints/LWX/89,86/forecast and got a properties.updateTime of 2021-04-30T00:00:00+00:00 which was (and still is) well into the future. That lasted until 2021-04-22 15:29:27, then the properties.updateTime jumped way back to 2021-04-20T10:26:16+00:00.

Here are some details from my logs (timestamps at beginning of lines are US/Eastern time). Hope it helps.

2021-04-22 14:43:43,392 DEBUG datafetch[976] urlcache/get() 66 : request parameters: None
2021-04-22 14:43:43,392 DEBUG datafetch[976] urlcache/get() 67 : request headers: 
  User-Agent: https://gitlab.com/tempest-weather/tempest
  Pragma: akamai-x-cache-on, akamai-x-get-request-id
2021-04-22 14:43:43,393 DEBUG datafetch[976] urlcache/get() 68 : response headers: 
  Server: nginx/1.16.1
  Content-Type: application/geo+json
  Last-Modified: Thu, 22 Apr 2021 13:27:42 GMT
  Access-Control-Allow-Origin: *
  Access-Control-Allow-Headers: Feature-Flags
  X-Request-ID: 44d17da0-e274-475a-8906-da640fc0ff8f
  X-Correlation-ID: 4e17b9c3
  X-Server-ID: vm-bldr-nids-apiapp11.ncep.noaa.gov
  Content-Encoding: gzip
  Content-Length: 1497
  X-Akamai-Request-ID: 4f09f297.15029f7a.ab99cbb
  Cache-Control: public, max-age=886, s-maxage=3600
  Expires: Thu, 22 Apr 2021 18:58:29 GMT
  Date: Thu, 22 Apr 2021 18:43:43 GMT
  X-Cache: TCP_REFRESH_MISS from a23-32-17-132.deploy.akamaitechnologies.com (AkamaiGHost/10.3.4.1-33174363) (S)
  Connection: keep-alive
  Vary: Accept-Encoding, Accept,Feature-Flags,Accept-Language
  X-Edge-Request-ID: ab99cbb
  Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
2021-04-22 14:43:43,394 DEBUG datafetch[976] nws/is_json_newer() 391 : src.name=forecast, time_field=properties.updateTime
2021-04-22 14:43:43,397 DEBUG datafetch[976] nws/is_json_newer() 404 : got old_time=2021-04-22T13:27:42+00:00
2021-04-22 14:43:43,397 DEBUG datafetch[976] nws/is_json_newer() 406 : got new_time=2021-04-30T00:00:00+00:00

2021-04-22 14:58:51,495 DEBUG datafetch[976] urlcache/get() 66 : request parameters: None
2021-04-22 14:58:51,496 DEBUG datafetch[976] urlcache/get() 67 : request headers: 
  User-Agent: https://gitlab.com/tempest-weather/tempest
  Pragma: akamai-x-cache-on, akamai-x-get-request-id
2021-04-22 14:58:51,497 DEBUG datafetch[976] urlcache/get() 68 : response headers: 
  Server: nginx/1.16.1
  Content-Type: application/geo+json
  Last-Modified: Thu, 22 Apr 2021 13:27:42 GMT
  Access-Control-Allow-Origin: *
  Access-Control-Allow-Headers: Feature-Flags
  X-Request-ID: 44d17da0-e274-475a-8906-da640fc0ff8f
  X-Correlation-ID: 4e17b9c3
  X-Server-ID: vm-bldr-nids-apiapp11.ncep.noaa.gov
  Content-Encoding: gzip
  Content-Length: 1497
  Cache-Control: public, max-age=900, s-maxage=3600
  Expires: Thu, 22 Apr 2021 19:13:51 GMT
  Date: Thu, 22 Apr 2021 18:58:51 GMT
  X-Cache: TCP_REFRESH_HIT from a23-32-17-132.deploy.akamaitechnologies.com (AkamaiGHost/10.3.4.1-33174363) (S)
  Connection: keep-alive
  Vary: Accept-Encoding, Accept,Feature-Flags,Accept-Language
  X-Edge-Request-ID: adcbfbe
  Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
  X-Akamai-Request-ID: adcbfbe
2021-04-22 14:58:51,500 DEBUG datafetch[976] nws/is_json_newer() 391 : src.name=forecast, time_field=properties.updateTime
2021-04-22 14:58:51,505 DEBUG datafetch[976] nws/is_json_newer() 404 : got old_time=2021-04-30T00:00:00+00:00
2021-04-22 14:58:51,506 DEBUG datafetch[976] nws/is_json_newer() 406 : got new_time=2021-04-30T00:00:00+00:00

2021-04-22 15:14:00,609 DEBUG datafetch[976] urlcache/get() 66 : request parameters: None
2021-04-22 15:14:00,610 DEBUG datafetch[976] urlcache/get() 67 : request headers: 
  User-Agent: https://gitlab.com/tempest-weather/tempest
  Pragma: akamai-x-cache-on, akamai-x-get-request-id
2021-04-22 15:14:00,611 DEBUG datafetch[976] urlcache/get() 68 : response headers: 
  Server: nginx/1.16.1
  Content-Type: application/geo+json
  Last-Modified: Thu, 22 Apr 2021 13:27:42 GMT
  Access-Control-Allow-Origin: *
  Access-Control-Allow-Headers: Feature-Flags
  X-Request-ID: 44d17da0-e274-475a-8906-da640fc0ff8f
  X-Correlation-ID: 4e17b9c3
  X-Server-ID: vm-bldr-nids-apiapp11.ncep.noaa.gov
  Content-Encoding: gzip
  Content-Length: 1497
  Cache-Control: public, max-age=868, s-maxage=3600
  Expires: Thu, 22 Apr 2021 19:28:28 GMT
  Date: Thu, 22 Apr 2021 19:14:00 GMT
  X-Cache: TCP_REFRESH_HIT from a23-32-17-132.deploy.akamaitechnologies.com (AkamaiGHost/10.3.4.1-33174363) (S)
  Connection: keep-alive
  Vary: Accept-Encoding, Accept,Feature-Flags,Accept-Language
  X-Edge-Request-ID: afecfae
  Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
  X-Akamai-Request-ID: afecfae
2021-04-22 15:14:00,613 DEBUG datafetch[976] nws/is_json_newer() 391 : src.name=forecast, time_field=properties.updateTime
2021-04-22 15:14:00,618 DEBUG datafetch[976] nws/is_json_newer() 404 : got old_time=2021-04-30T00:00:00+00:00
2021-04-22 15:14:00,619 DEBUG datafetch[976] nws/is_json_newer() 406 : got new_time=2021-04-30T00:00:00+00:00

2021-04-22 15:29:07,913 DEBUG datafetch[976] urlcache/get() 66 : request parameters: None
2021-04-22 15:29:07,914 DEBUG datafetch[976] urlcache/get() 67 : request headers: 
  User-Agent: https://gitlab.com/tempest-weather/tempest
  Pragma: akamai-x-cache-on, akamai-x-get-request-id
2021-04-22 15:29:07,914 DEBUG datafetch[976] urlcache/get() 68 : response headers: 
  Server: nginx/1.16.1
  Content-Type: application/geo+json
  Last-Modified: Thu, 22 Apr 2021 13:27:42 GMT
  Access-Control-Allow-Origin: *
  Access-Control-Allow-Headers: Feature-Flags
  X-Request-ID: 44d17da0-e274-475a-8906-da640fc0ff8f
  X-Correlation-ID: 4e17b9c3
  X-Server-ID: vm-bldr-nids-apiapp11.ncep.noaa.gov
  Content-Encoding: gzip
  Content-Length: 1497
  Cache-Control: public, max-age=900, s-maxage=3600
  Expires: Thu, 22 Apr 2021 19:44:07 GMT
  Date: Thu, 22 Apr 2021 19:29:07 GMT
  X-Cache: TCP_REFRESH_HIT from a23-32-17-132.deploy.akamaitechnologies.com (AkamaiGHost/10.3.4.1-33174363) (S)
  Connection: keep-alive
  Vary: Accept-Encoding, Accept,Feature-Flags,Accept-Language
  X-Edge-Request-ID: b0f712b
  Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
  X-Akamai-Request-ID: b0f712b
2021-04-22 15:29:07,915 DEBUG datafetch[976] nws/is_json_newer() 391 : src.name=forecast, time_field=properties.updateTime
2021-04-22 15:29:07,918 DEBUG datafetch[976] nws/is_json_newer() 404 : got old_time=2021-04-30T00:00:00+00:00
2021-04-22 15:29:07,918 DEBUG datafetch[976] nws/is_json_newer() 406 : got new_time=2021-04-30T00:00:00+00:00

2021-04-22 15:44:15,639 DEBUG datafetch[976] urlcache/get() 66 : request parameters: None
2021-04-22 15:44:15,639 DEBUG datafetch[976] urlcache/get() 67 : request headers: 
  User-Agent: https://gitlab.com/tempest-weather/tempest
  Pragma: akamai-x-cache-on, akamai-x-get-request-id
2021-04-22 15:44:15,641 DEBUG datafetch[976] urlcache/get() 68 : response headers: 
  Server: nginx/1.16.1
  Content-Type: application/geo+json
  Last-Modified: Thu, 22 Apr 2021 13:27:42 GMT
  Access-Control-Allow-Origin: *
  Access-Control-Allow-Headers: Feature-Flags
  X-Request-ID: 44d17da0-e274-475a-8906-da640fc0ff8f
  X-Correlation-ID: 4e17b9c3
  X-Server-ID: vm-bldr-nids-apiapp11.ncep.noaa.gov
  Content-Encoding: gzip
  Content-Length: 1497
  Cache-Control: public, max-age=794, s-maxage=3600
  Expires: Thu, 22 Apr 2021 19:57:29 GMT
  Date: Thu, 22 Apr 2021 19:44:15 GMT
  X-Cache: TCP_REFRESH_HIT from a23-32-17-132.deploy.akamaitechnologies.com (AkamaiGHost/10.3.4.1-33174363) (S)
  Connection: keep-alive
  Vary: Accept-Encoding, Accept,Feature-Flags,Accept-Language
  X-Edge-Request-ID: b2cdb83
  Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
  X-Akamai-Request-ID: b2cdb83
2021-04-22 15:44:15,643 DEBUG datafetch[976] nws/is_json_newer() 391 : src.name=forecast, time_field=properties.updateTime
2021-04-22 15:44:15,649 DEBUG datafetch[976] nws/is_json_newer() 404 : got old_time=2021-04-30T00:00:00+00:00
2021-04-22 15:44:15,650 DEBUG datafetch[976] nws/is_json_newer() 406 : got new_time=2021-04-30T00:00:00+00:00

2021-04-22 15:59:27,405 DEBUG datafetch[976] urlcache/get() 66 : request parameters: None
2021-04-22 15:59:27,406 DEBUG datafetch[976] urlcache/get() 67 : request headers: 
  User-Agent: https://gitlab.com/tempest-weather/tempest
  Pragma: akamai-x-cache-on, akamai-x-get-request-id
2021-04-22 15:59:27,407 DEBUG datafetch[976] urlcache/get() 68 : response headers: 
  Server: nginx/1.16.1
  Content-Type: application/geo+json
  Last-Modified: Tue, 27 Apr 2021 09:00:00 GMT
  Access-Control-Allow-Origin: *
  Access-Control-Allow-Headers: Feature-Flags
  X-Request-ID: 45e2b892-5ffd-4493-9142-d83d7beda5de
  X-Correlation-ID: 31b34885
  X-Server-ID: vm-lnx-nids-apiapp4.ncep.noaa.gov
  Content-Encoding: gzip
  Content-Length: 1591
  X-Akamai-Request-ID: de2e37d.15999843.b493889
  Cache-Control: public, max-age=900, s-maxage=3600
  Expires: Thu, 22 Apr 2021 20:14:27 GMT
  Date: Thu, 22 Apr 2021 19:59:27 GMT
  X-Cache: TCP_REFRESH_MISS from a23-32-17-132.deploy.akamaitechnologies.com (AkamaiGHost/10.3.4.1-33174363) (S)
  Connection: keep-alive
  Vary: Accept-Encoding, Accept,Feature-Flags,Accept-Language
  X-Edge-Request-ID: b493889
  Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
2021-04-22 15:59:27,410 DEBUG datafetch[976] nws/is_json_newer() 391 : src.name=forecast, time_field=properties.updateTime
2021-04-22 15:59:27,415 DEBUG datafetch[976] nws/is_json_newer() 404 : got old_time=2021-04-30T00:00:00+00:00
2021-04-22 15:59:27,416 DEBUG datafetch[976] nws/is_json_newer() 406 : got new_time=2021-04-20T10:26:16+00:00

[sdnwser]

Based on the number of posts, it's clear that this is a backend issue. Let's hope they get some time to work it out.

[scadergit]

This is a backend issue. The service that feeds the API forecast grids is returning invalid times. The service is not in a healthy place, and is the current focus to replace. It will likely be a few months before that is complete and gets through the NCO process to deploy.

[sdnwser]

Thank you! Looking forward to an upgraded back end. Is there anything the community can do to assist?

[JJKraw]

Here's another I just got on a fetch of https://api.weather.gov/gridpoints/BOX/64,79

HTTP/2 200 
server: nginx/1.16.1
content-type: application/geo+json
last-modified: Mon, 26 Apr 2021 23:00:00 GMT
access-control-allow-origin: *
access-control-allow-headers: Feature-Flags
x-request-id: 462aee81-6121-462a-ad42-20eae0fc3db7
x-correlation-id: 66f1e436
x-server-id: vm-lnx-nids-apiapp7.ncep.noaa.gov
x-akamai-request-id: 72599e78.249f5973.13514fa2
cache-control: public, max-age=880, s-maxage=3600
expires: Mon, 26 Apr 2021 13:52:40 GMT
date: Mon, 26 Apr 2021 13:38:00 GMT
x-cache: TCP_REFRESH_MISS from a23-48-12-100.deploy.akamaitechnologies.com (AkamaiGHost/10.3.4.1-33174363) (S)
x-edge-request-id: 13514fa2
vary: Accept,Feature-Flags,Accept-Language
strict-transport-security: max-age=31536000 ; includeSubDomains ; preload

[JJKraw]

Forgot to mention the data was over 26 hours old.

[pauliniguez]

I've been getting the same behavior with a python3 script. When using the requests library, at 00Z it would return two-day old data until about 07Z. I tried adding various headers but still had the same behavior. Yesterday I switched to using urllib.request and for whatever reason, that seems to behave differently and not return old data.

[sdmcclain]

This suggestion worked for me, too! Thanks @pauliniguez 🙏 🙏 🙏

[pauliniguez]

And now urllib is returning incomplete data. Is every NWS tool destined to fail?

[scadergit]

The library wasn't the cause, likely just returned a different response that was not impacted by the actual issue. There are challenges the API has to overcome that are not ordinary. I can't speak for NWS overall, but the API (and the solution that supports it) will see significant improvements this summer, as we finally have the opportunity to address core services.

[engpok]

That’s great news… thanks for the update… looking forward to this fantastic service being more reliable

…

On Apr 28, 2021, at 2:01 PM, Jared Holmberg ***@***.***> wrote: This is a backend issue. The service that feeds the API forecast grids is returning invalid times. The service is not in a healthy place, and is the current focus to replace. It will likely be a few months before that is complete and gets through the NCO process to deploy. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#224 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIOBAYTKJ65C5OLW2B7POLTLBEP3ANCNFSM43BRLREA>.

[sdmcclain]

Hi all, another big fan of this weather API, hoping to have it back in action very soon.

If another example helps, I have a Google Cloud Function that makes a GET request to https://api.weather.gov/gridpoints/PHI/34,71/forecast/hourly every hour, and an example response header returned by the cloud function is:

{
 "Server": "nginx/1.16.1",
 "Content-Type": "application/geo+json",
 "Last-Modified": "Thu, 30 Dec 2021 21:17:00 GMT",
 "Access-Control-Allow-Origin": "*",
 "Access-Control-Allow-Headers": "Feature-Flags",
 "X-Request-ID": "2d9c272a-c805-4ed1-ba48-51a5b25f3f33",
 "X-Correlation-ID": "2c5eb7",
 "X-Server-ID": "vm-lnx-nids-apiapp3.ncep.noaa.gov",
 "Content-Encoding": "gzip",
 "Content-Length": "3518",
 "Cache-Control": "public, max-age=783, s-maxage=3600",
 "Expires": "Sat, 01 May 2021 23:03:14 GMT",
 "Date": "Sat, 01 May 2021 22:50:11 GMT",
 "Connection": "keep-alive",
 "Vary": "Accept-Encoding, Accept,Feature-Flags,Accept-Language",
 "X-Edge-Request-ID": "70b8057",
 "Strict-Transport-Security": "max-age=31536000 ; includeSubDomains ; preload"
 }

Which is a last-modified date at the end of this year 😢 and the response contains hourly forecasts from 2020-12-29 to 2021-01-05, so it is no longer usable.

Notably, requests to this same endpoint using curl or Chrome on my local machine do not have this issue (only GCP). Any advice on a short-term workaround?

[sdmcclain]

After posting this I saw @pauliniguez's suggestion to use python's urllib instead of requests, which seems to have resolved the issue for me.

[engpok]

Hmmmm… you reminded me of something… I never saw the problem when I used Chrome to get the forecast. I can’t say that I tried more than 10 times, but when I saw the problem with my app, I immediately went to Chrome to see if I saw the same problem… never did.

…

On May 1, 2021, at 7:02 PM, Steve McClain ***@***.***> wrote: Hi all, another big fan of this weather API, hoping to have it back in action very soon. If another example helps, I have a Google Cloud Function that makes a GET request to https://api.weather.gov/gridpoints/PHI/34,71/forecast/hourly <https://api.weather.gov/gridpoints/PHI/34,71/forecast/hourly> every hour, and an example response header returned by the cloud function is: { "Server": "nginx/1.16.1", "Content-Type": "application/geo+json", "Last-Modified": "Thu, 30 Dec 2021 21:17:00 GMT", "Access-Control-Allow-Origin": "*", "Access-Control-Allow-Headers": "Feature-Flags", "X-Request-ID": "2d9c272a-c805-4ed1-ba48-51a5b25f3f33", "X-Correlation-ID": "2c5eb7", "X-Server-ID": "vm-lnx-nids-apiapp3.ncep.noaa.gov", "Content-Encoding": "gzip", "Content-Length": "3518", "Cache-Control": "public, max-age=783, s-maxage=3600", "Expires": "Sat, 01 May 2021 23:03:14 GMT", "Date": "Sat, 01 May 2021 22:50:11 GMT", "Connection": "keep-alive", "Vary": "Accept-Encoding, Accept,Feature-Flags,Accept-Language", "X-Edge-Request-ID": "70b8057", "Strict-Transport-Security": "max-age=31536000 ; includeSubDomains ; preload" } Which is a last-modified date at the end of this year 😢 and the response contains hourly forecasts from 2020-12-29 to 2021-01-05, so it is no longer usable. Notably, requests to this same endpoint using curl or Chrome on my local machine do not have this issue (only GCP). Any advice on a short-term workaround? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#224 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIOBA5PRNEZQ633QUXT23DTLSB75ANCNFSM43BRLREA>.

[sdmcclain]

Yes, there definitely does seem to be some aspect of the client that's relevant. Changing the python library I was using to make the request seems to have made a difference for me 🤷‍♂️

[sdnwser]

I have seen similar behavior but I believe it's related to the different http context that a browser sends and a script sends. I also updated the context to make it appear like a browser (not compliant with NWS's API identification request - sorry!) and it only seemed to fix it for a short time. I believe that this is related to the interaction between the CDN and the NWS backend. Take this with a grain of salt since my comments are purely speculative.

[scadergit]

If you're getting a different correlation id then it's a different cache response, which can provide unexpected results until we fix the service that feeds the API. If you get different results with the same correlation id, then that's a whole other ball game.

[sdnwser]

What is "correlation id"?

[scadergit]

If you look in the response headers (examples above from others), you'll see "x-correlation-id" which is a number that is unique to every request to the API. If you see two requests with the same correlation id, that's because you got a cached response from the second request.

[ktrue]

I, too, have been seeing this 'stale cache' issue for a gridpoint forecast consistently for multiple months. I'm the author of a widely-used PHP script (advforecast2.php) and had incorporated logging of failures in the script. I now have a display script set to show the logs and allow sort/search of the daily logs (by day).

https://saratoga-weather.org/NWS-analyze/ for my current displays. The script set is downloadable from a link at the bottom of the page.

Looking at multiple days, it appears that API access to https://api.weather.gov/gridpoints/MTR/93,79/forecast results in a stale forecast from vm-bldr-nids-apiapp7.ncep.noaa.gov and a104-71-218-95.deploy.akamaitechnologies.com (AkamaiGHost/10.4.0-33449709) starting after midnight (local time) and persisting until late morning. The associated HTTP response data is shown for each entry. Also, the Last-modified: date seems far in the future for something that is to be refreshed every 6 hours with new GFS data. Maybe the Expires: and Last-modified: headers from the Akamai server are swapped??
Example:

2021-05-25T17:01:08+00:00 (local: Tue, 25 May 2021 10:01:08 -0700) - advforecast2.php (JSON) - V5.17 - 15-Jul-2020
Stale
Forecast Stale Age=27:47:04 from gridpoint - use Zone forecast instead



HTTP/1.1 200 OK
Server: nginx/1.16.1
Content-Type: application/geo+json
Last-Modified: Thu, 27 May 2021 08:00:00 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Feature-Flags
X-Request-ID: 341a3d16-905b-471b-be5b-759100be71b9
X-Correlation-ID: 40c9f792
X-Server-ID: vm-bldr-nids-apiapp7.ncep.noaa.gov
Cache-Control: public, max-age=295, s-maxage=3600
Expires: Tue, 25 May 2021 17:06:03 GMT
Date: Tue, 25 May 2021 17:01:08 GMT
Content-Length: 10934
X-Cache: TCP_MEM_HIT from a104-71-218-95.deploy.akamaitechnologies.com (AkamaiGHost/10.4.0-33449709) (-)
Connection: keep-alive
X-Edge-Request-ID: 57e14568
Vary: Accept,Feature-Flags,Accept-Language
Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
X-Akamai-Request-ID: 57e14568

I request that someone on the API team use the information to resolve the long running stale cache issue.

[JJKraw]

I have Ken's code up on my site now. Data can be accessed at:

https://lexmaweather.info/NWS-analyze/