NWS returning stale data

[fancygaphtrn]

I use the NWS component in Home Assistant and it depends on the lib.

This week I am getting stale data returned from the NWS. All week it returns the forecast starting on Monday.

If I would open the url via my browser it would get current data.

I did some experimenting via curl and is seems to be an issues with the Accept header.

This returns stale data.
curl -i -H "Accept: application/geo+json" -H "User-Agent: pynws {[email protected]}" https://api.weather.gov/gridpoints/RNK/49,40/forecast

"periods": [ { "number": 1, "name": "This Afternoon", "startTime": "2020-07-27T13:00:00-04:00", "endTime": "2020-07-27T18:00:00-04:00", "isDaytime": true,

This return current data
curl -i -H "Accept: /" -H "User-Agent: pynws {[email protected]}" https://api.weather.gov/gridpoints/RNK/49,40/forecast

"periods": [ { "number": 1, "name": "Today", "startTime": "2020-07-30T11:00:00-04:00", "endTime": "2020-07-30T18:00:00-04:00", "isDaytime": true,
As a work around for me I changed the Accept in const.py to /

[MatthewFlamm]

Thanks for the report.

This behavior has been noted in https://github.com/weather-gov/api/issues/71 and https://github.com/weather-gov/api/issues/86. Although, it is interesting that you can change the accept header to / with good results.

I'm planning on filtering out the older forecasts, and this is already implemented for the next version. I'm still thinking through whether we should also check for stale data that is past the expiration time in the header. If so, then the data should be treated as unavailable. Having written this out, it sounds like we should. There is no handling of the expiration time in the library currently, so I need to think through how best to incorporate it.

[fancygaphtrn]

After reading the report I noticed that many of the responses were coming back as "application/ld+json". Using curl I noticed it works when you include the accept header like the following.

curl -i -H "Accept: application/geo+json,application/ld+json" -H "User-Agent: pynws {xxx@xxx}" https://api.weather.gov/gridpoints/RNK/49,40/forecast

I then changed the const.py to API_ACCEPT = "application/geo+json,application/ld+json". I did this because some the responses are text/html and it seemed your lib wanted only json.

[MatthewFlamm]

It seems changing anything in the header allows you to circumvent the stale data issue. But, this should not be used in the library IMO. It should be fixed upstream. We can make the library more robust to this problem however.

[MatthewFlamm]

v1.3.0 implements the first half of the logic needed to make pynws more robust to this.

PR submitted in home assistant home-assistant/core#40386

I will keep this open until the forecast expiration time is also handled.