[BUG] Events are visible in the app even though their ids are in the "originating calendar id" blacklist

[dobromirdobrev]

Describe the bug
An event is visible in the Illinois app even though the id is in the "originating calendar id" black list.

Origin:

Subtracting Webtools Calendars
@dobromirdobrev I moved four of the calendar ids from the "calendar ids" list to the "originating calendar ids" list on Friday, August 22. The events on those calendars are still displaying. I would expect none of the events from these four calendars to display in the app.

Here are the calendars:
https://calendars.illinois.edu/list/5801
https://calendars.illinois.edu/list/937
https://calendars.illinois.edu/list/8300
https://calendars.illinois.edu/list/7127

As one example, you will see the event "The NetMath Office will close at 2 pm on Friday, August 29th and remain closed until 9am Tuesday September 2nd, 2025 in observance of Labor Day" on calendar 5801. In the app, that event still does display. I would expect the event to be removed from displaying in the app after the nightly sync.

Expected behavior
Once the id is in the "originating calendar id" blacklist then the event should not be visible in the Illinois app.
So, neither of these events should be visible in the Illinois app

Additional context
Source: rokwire/illinois-app#4787 (comment)

Observations
From the IDs shown in the original ticket, I noticed that only two events are visible in the Illinois app:

• Id: 5801

  Visible (Incorrect):
  * "The NetMath Office will close at 2 pm on Friday, August 29th and remain closed until 9am Tuesday September 2nd, 2025 in observance of Labor Day"
  * "The NetMath Office will close at 2 pm on Wednesday, November 26th and remain closed until 9am Monday December 1st, 2024 in observance of the Thanksgiving Holidays"

  Not visible (Correct):
  * "NetMath Office Closed for Labor Day: September 1st, 2025"
  * "NetMath Office Closed for Thanksgiving Day: November 27th, 2025"
  * "NetMath Office Closed for Day after Thanksgiving Holiday: November 28th, 2025"

• Id: 937 – no events are visible (correct).

• Id: 8300 – there is only 1 event, and it is not visible in the app (correct).

• Id: 7127 – no events are visible in the app (correct).

[petyos]

@vburgett @dobromirdobrev

We have checked the system and the events for the following calendar IDs: 937, 8300, and 7127 are not shown in the application because they are in the blacklist, and they are working correctly.

We notice a problem only with 5801, and the reason is that this is not the correct value for originatingCalendarID. From WebTools we do not receive 5801, but instead we receive 7037.

Therefore, if you add 7037 to the blacklist and remove 5801 (since it is incorrect), the events will not be displayed after the next synchronization.

As for why WebTools does not send 5801 but sends 7037, I am not sure. Most likely there is some internal logic in WebTools regarding these IDs.

[vburgett]

@petyos Calendar 7037 was listed under "calendar ids" on the admin website.
Yesterday I moved 7037 to the "originating calendar ids."
This morning the event from 7037 is still in the app. The event is "The NetMath Office will close at 2 pm on Wednesday, November 26th and remain closed until 9am Monday December 1st, 2024 in observance of the Thanksgiving Holidays"

Does the update to blacklist the calendar require more than one night's sync?
How do we know when to blacklist a calendar by using the "calendar id" section or the "originating calendar ids" section?

[petyos]

@vburgett , yes, the correct field is originating calendar id. You moved it correctly.

The system extracts events from Webtools using only one calendar id, and that is 6991. In practice, we fetch all events from Webtools with this calendar id:
https://xml.calendars.illinois.edu/eventXML16/6991.xml?pageNumber=%d.

If we blacklist calendar id 6991, then all events will disappear, which makes no sense. That is why we use the originating calendar id, where they differ.

By the way, this is also related to the other issue rokwire/illinois-app#4787 (comment), but I will write there separately.

Currently, the events with originating calendar id 7037 are ignored in the Illinois app.

How it works: every morning at 5 a.m. Illinois time, the gateway BB takes and synchronizes from Webtools. Once a day the calendar BB takes from the gateway, but this is not fixed and it can happen before or after 5 a.m.

So, in general, one day is enough for synchronization, but sometimes two are needed – for example, if the calendar BB syncs at 4 and the gateway at 5.

Right now I see that the calendar synced at 5:22, which means that after 24 hours it will sync again at that time.

[vburgett]

@petyos So a blacklisted Webtools event may need up to 48 hours (or two nights of synchronizing) to no longer display in the Illinois app?

How do I know whether to blacklist a calendar using the "calendar id" section OR the "originating calendar ids" section? Right now I would be guessing.

[petyos]

Yes, that’s right – usually one night is enough, but depending on when the two syncs happen it can take up to 48 hours for a blacklisted event to disappear.

Also, just to connect this with the earlier discussion(rokwire/illinois-app#4787 (comment)): in Webtools we actually see two IDs – calendarId and originatingCalendarId. The gateway always pulls from calendarId = 6991, and inside that feed the events come with different originatingCalendarId values.

That’s why blacklisting should always be done using the originatingCalendarId(at least for now as currently we have only 6991 calendar id). If we blacklist calendarId = 6991, everything would be gone. After your testing on the calendar-level flag, we may have more clarity on how exactly this is determined in Webtools.