how to filter out duplicate alerts?

[ckolovson]

I would have thought that alerts with an expiration time < now would not be considered active. What defines an active alert? I see this question was last discussed in February (#573), but it doesn't seem to be resolved or answered. I thought I had a workaround of iterating through the results and checking for expiredTime > now. I have a Python script that attempted to do this (#677), but that doesn't appear to be correct.

[scadergit]

There is a difference between product expiration, which is the announcement, and event expiration, which defines active.

…

On Wed, Nov 29, 2023, 6:44 PM Curt Kolovson ***@***.***> wrote: I would have thought that alerts with an expiration time < now would not be considered active. What defines an active alert? — Reply to this email directly, view it on GitHub <#678>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEFQSTQCGPFJ32CBMUHH47LYG7JF5AVCNFSM6AAAAABAALGWFCVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZVHEYDONRWGQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[sullynole]

Do you have an example of an alert that is active that shouldn't be? There is a known bug with active alerts that should be fixed in the next month.

[ckolovson]

https://api.weather.gov/alerts/active?area=CA
Here's what it returns (results summary):

Air Stagnation Advisory for Western Siskiyou County; Central Siskiyou County; Northeast Siskiyou and Northwest Modoc Counties; Modoc County; Central Douglas County; Eastern Curry County and Josephine County; Eastern Douglas County Foothills; Jackson County; Siskiyou Mountains and Southern Oregon Cascades; Klamath Basin; Northern and Eastern Klamath County and Western Lake County; Central and Eastern Lake County till Thursday, November 30, 2023 at 10:00 AM UTC-08:00

Wind Advisory for Interstate 5 Corridor; Western Antelope Valley Foothills till Saturday, December 02, 2023 at 03:00 AM UTC-08:00

Wind Advisory for Santa Barbara County Southwestern Coast; Santa Ynez Mountains Western Range till Friday, December 01, 2023 at 03:00 PM UTC-08:00

Wind Advisory for Western San Gabriel Mountains and Highway 14 Corridor till Friday, December 01, 2023 at 03:00 PM UTC-08:00

Wind Advisory for Mojave Desert Slopes till Friday, December 01, 2023 at 10:00 AM UTC-08:00

Air Stagnation Advisory for Western Siskiyou County; Central Siskiyou County; Northeast Siskiyou and Northwest Modoc Counties; Modoc County; Eastern Curry County and Josephine County; Eastern Douglas County Foothills; Jackson County; Siskiyou Mountains and Southern Oregon Cascades; Klamath Basin; Northern and Eastern Klamath County and Western Lake County; Central and Eastern Lake County till Thursday, November 30, 2023 at 10:00 AM UTC-08:00

Freeze Warning for Los Banos - Dos Palos; Merced - Madera - Mendota; Planada - Le Grand - Snelling; West Side of Fresno and Kings Counties; Caruthers - San Joaquin - Selma; Buttonwillow - Lost Hills - I5; Delano-Wasco-Shafter; Hanford - Corcoran - Lemoore; Visalia - Porterville - Reedley; Southeast San Joaquin Valley; Indian Wells Valley; Mojave Desert till Thursday, November 30, 2023 at 08:00 AM UTC-08:00

Air Stagnation Advisory for Western Siskiyou County; Central Siskiyou County; Northeast Siskiyou and Northwest Modoc Counties; Modoc County; Central Douglas County; Eastern Curry County and Josephine County; Eastern Douglas County Foothills; Jackson County; Siskiyou Mountains and Southern Oregon Cascades; Klamath Basin; Northern and Eastern Klamath County and Western Lake County; Central and Eastern Lake County till Thursday, November 30, 2023 at 10:00 AM UTC-08:00

Freeze Warning for Los Banos - Dos Palos; Merced - Madera - Mendota; Planada - Le Grand - Snelling; West Side of Fresno and Kings Counties; Caruthers - San Joaquin - Selma; Buttonwillow - Lost Hills - I5; Delano-Wasco-Shafter; Hanford - Corcoran - Lemoore; Visalia - Porterville - Reedley; Southeast San Joaquin Valley; Indian Wells Valley; Mojave Desert till Thursday, November 30, 2023 at 08:00 AM UTC-08:00

Freeze Warning for Los Banos - Dos Palos; Merced - Madera - Mendota; Planada - Le Grand - Snelling; West Side of Fresno and Kings Counties; Caruthers - San Joaquin - Selma; Buttonwillow - Lost Hills - I5; Delano-Wasco-Shafter; Hanford - Corcoran - Lemoore; Visalia - Porterville - Reedley; Southeast San Joaquin Valley; Indian Wells Valley; Mojave Desert till Thursday, November 30, 2023 at 08:00 AM UTC-08:00

Freeze Watch for Los Banos - Dos Palos; Merced - Madera - Mendota; Planada - Le Grand - Snelling; West Side of Fresno and Kings Counties; Caruthers - San Joaquin - Selma; Buttonwillow - Lost Hills - I5; Delano-Wasco-Shafter; Hanford - Corcoran - Lemoore; Visalia - Porterville - Reedley; Southeast San Joaquin Valley; Indian Wells Valley; Mojave Desert till Thursday, November 30, 2023 at 08:00 AM UTC-08:00

Freeze Watch for Los Banos - Dos Palos; Merced - Madera - Mendota; Planada - Le Grand - Snelling; West Side of Fresno and Kings Counties; Caruthers - San Joaquin - Selma; Buttonwillow - Lost Hills - I5; Delano-Wasco-Shafter; Hanford - Corcoran - Lemoore; Visalia - Porterville - Reedley; Southeast San Joaquin Valley; Indian Wells Valley; Mojave Desert till Thursday, November 30, 2023 at 08:00 AM UTC-08:00

This returns some duplicate alerts. It appears that expiration is not the issue. My question is, how do I filter out the duplicates? Ideally, there would be a way to do this through the API that doesn't require iterating through all the results to check for duplicates.

[dwhitemv25]

The ones that are "duplicates", do they have the same id property value?

I have seen situations where the WFO incorrectly codes the VTEC in the product and it creates a new lineage of the alert in the API and thus a "duplicate" alert returned in the /alerts/active query, at least until the last product from the original lineage expires. Not a lot can be done about this beyond advanced heuristics.

[ckolovson]

It appears that the "duplicates" are a result of updates to existing alerts. The old versions are still being returned by /alerts/active, presumably because the old versions have not expired. I can see which is the most up-to-date by looking at the "issued" time in the "headline". It would be good if there was a simple way to filter out the old (superseded) alerts, or ideally if they were filtered out by the API. I will check the alert IDs in such cases and report back later.

[sullynole]

There is a standing bug with alert tracking when an alert is extended or canceled early where earlier versions will reappear in the active endpoint. A release is scheduled next week to fix that bug. Basically the expiredReferences needs to be tracked to avoid seeing those until the bug fix.

[ckolovson]

Thanks! I seem to recall a previous discussion about this issue. FYI, here are some examples (including ids). Will the bug fix you mentioned filter out the older versions of alerts show below?

HEADLINE: High Surf Advisory issued December 6 at 2:54AM PST until December 7 at 4:00AM PST by NWS San Francisco CA
id: urn:oid:2.49.0.1.840.0.2afbfa60a954ccd3e12890b58a83134b5cb49490.001.1

DESCRIPTION: * WHAT...Large breaking waves of 18 to 22 feet.

* WHERE...San Francisco, Coastal North Bay Including Point Reyes
National Seashore, San Francisco Peninsula Coast and Southern
Monterey Bay and Big Sur Coast Counties.

* WHEN...Until 4 AM PST Thursday.

* IMPACTS...Dangerous swimming and surfing conditions and
localized beach erosion. Large waves can sweep across the
beach without warning, pulling people into the sea from rocks,
jetties, and beaches. These waves can also move large objects
such as logs, crushing anyone caught underneath.

* ADDITIONAL DETAILS...Large northwest waves will peak this
morning. Northwest facing beaches are most at risk for large
turbulent shore break and strong currents.

INSTRUCTIONS: Remain out of the water to avoid hazardous swimming conditions.
Large breaking waves along the coast will lead to increased run-
up on beaches with waves topping and washing over large rocks and
jetties. These large waves can be erratic and unpredictable. Use
extra caution near the surf zone as these large waves will be
capable of sweeping people into the water. Avoid rocks and
jetties. Avoid steep beaches. Stay much farther back from the
water and never turn your back on the ocean.
--------------------
HEADLINE: High Surf Advisory issued December 5 at 3:22AM PST until December 7 at 4:00AM PST by NWS San Francisco CA
id: urn:oid:2.49.0.1.840.0.7b32fcbe423f1ecb468c4d093c325ff68008a3a3.001.1

DESCRIPTION: * WHAT...For the Beach Hazards Statement, increased risk of
sneaker waves expected. For the High Surf Advisory, large
breaking waves of 18 to 22 feet.

* WHERE...San Francisco, Coastal North Bay Including Point Reyes
National Seashore, San Francisco Peninsula Coast and Southern
Monterey Bay and Big Sur Coast Counties.

* WHEN...For the Beach Hazards Statement, through this evening.
For the High Surf Advisory, from 7 PM this evening to 4 AM PST
Thursday.

* IMPACTS...Dangerous swimming and surfing conditions and
localized beach erosion. Large waves can sweep across the beach
without warning, pulling people into the sea from rocks,
jetties, and beaches. These waves can also move large objects
such as logs, crushing anyone caught underneath.

* ADDITIONAL DETAILS...Large northwest waves will peak Wednesday
morning. Northwest facing beaches are most at risk for large
turbulent shore break and strong currents.

INSTRUCTIONS: Large breaking waves along the coast will lead to increased run-
up on beaches with waves topping and washing over large rocks and
jetties. These large waves can be erratic and unpredictable. Use
extra caution near the surf zone as these large waves will be
capable of sweeping people into the water. Avoid rocks and
jetties. Avoid steep beaches. Stay much farther back from the
water and never turn your back on the ocean.
-----------------------
HEADLINE: Beach Hazards Statement issued December 6 at 2:54AM PST until December 7 at 4:00AM PST by NWS San Francisco CA
id: urn:oid:2.49.0.1.840.0.2afbfa60a954ccd3e12890b58a83134b5cb49490.002.1

DESCRIPTION: * WHAT...Breaking waves of 9 to 13 feet expected.

* WHERE...Northern Monterey Bay County.

* WHEN...Until 4 AM PST Thursday.

* IMPACTS...Dangerous swimming and surfing conditions and
localized beach erosion. Large waves can sweep across the
beach without warning, pulling people into the sea from rocks,
jetties, and beaches.

* ADDITIONAL DETAILS...Large northwest waves will peak this
morning. Northwest facing beaches are most at risk for large,
turbulent shore break and strong currents.

INSTRUCTIONS: Remain out of the water to avoid hazardous swimming conditions.
Larger breaking waves along the coast will lead to increased run-
up on beaches with waves topping and washing over large rocks and
jetties. Use caution near the surf zone as these larger waves
will be capable of sweeping people into the water. Avoid rocks
and jetties. Avoid steep beaches. Stay farther back from the
water and never turn your back on the ocean.
--------------------
HEADLINE: Beach Hazards Statement issued December 5 at 3:22AM PST until December 7 at 4:00AM PST by NWS San Francisco CA
id: urn:oid:2.49.0.1.840.0.7b32fcbe423f1ecb468c4d093c325ff68008a3a3.002.1

DESCRIPTION: * WHAT...Breaking waves of 9 to 13 feet expected.

* WHERE...Northern Monterey Bay County.

* WHEN...From this evening through late Wednesday night.

* IMPACTS...Dangerous swimming and surfing conditions and
localized beach erosion. Large waves can sweep across the beach
without warning, pulling people into the sea from rocks,
jetties, and beaches.

* ADDITIONAL DETAILS...Large northwest waves will peak Wednesday
morning. Northwest facing beaches are most at risk for large,
turbulent shore break and strong currents.

INSTRUCTIONS: Remain out of the water to avoid hazardous swimming conditions.
Larger breaking waves along the coast will lead to increased
run-up on beaches with waves topping and washing over large rocks
and jetties. Use caution near the surf zone as these larger waves
will be capable of sweeping people into the water. Avoid rocks and
jetties. Avoid steep beaches. Stay farther back from the water
and never turn your back on the ocean.
--------------------

[sullynole]

Yes, I only see urn:oid:2.49.0.1.840.0.2afbfa60a954ccd3e12890b58a83134b5cb49490.001.1 and urn:oid:2.49.0.1.840.0.2afbfa60a954ccd3e12890b58a83134b5cb49490.002.1 active.

[ckolovson]

GREAT!! Thanks!! How and when will we know when the fixed API goes live? I assume it will be mentioned in the Updates section of https://www.weather.gov/documentation/services-web-api. Until then, I will use the "preview-api".

[sullynole]

Yes, I'll update you here. It's technically not an update to api code, but in the backend database that ingests the alerts. If all goes well I'd expect it on Wednesday 12/13.

[ckolovson]

Bug is fixed in next version of the API.

[sullynole]

The change went live yesterday 12/13/2023 ~1730z. Duplicates should be gone now (other than expected behavior with Alaska marine alerts).