Investigating Fire Event Explorer performance issues post-initial load

[dzole0311]

We've been getting a few reports from the EIS Fire team (Tempest, Eli, but also Anthony and I have observed this) about slow performance when interacting with the Fire Event Explorer after initial load. After the initial load (that serves cached fire data), subsequent zooming or interacting with the map results in noticeable delays (sometimes up to 30s) when fetching new fire data for the three 'lf_nrt' collections.

I'm opening this issue to track performance related reports, questions, investigations and potential mitigation strategies we can implement.

[anayeaye]

Subsequent delays don't make sense as a backend performance problem (especially with caching enabled) but we can dig in further.

As an aside, @botanical and I discovered that the production ingest pipeline is not using the RDS proxy connection pool which could cause slower behavior the two times a day an ingest happens--if these user experiences were perfectly timed to be exactly when the ingest is running they performance would be degraded (we have an issue to move production SM2A vector ingest to the connection pool).

[anayeaye]

Quick notes here:

1. Improvements during database write events @botanical and I are looking further into connection pooling on ingest airflow worker

2. Tuning vector requests @sandrahoang686 and I discovered that the vector requests with a high limit=500 are timing out for some /collections / items filtered fire ids but are successful if we constrain that a bit

Here's an example from staging
https://staging.openveda.cloud/api/features/collections/public.eis_fire_lf_perimeter_nrt/items?limit=100&filter=fireid=3750 is OK (maybe even going down to 50 might be better?)

https://staging.openveda.cloud/api/features/collections/public.eis_fire_lf_perimeter_nrt/items?limit=500&filter=fireid=3750 will fail with an internal server error. This is something that could potentially be improved in the API with pagination development but limit=100 seems like a reasonable guardrail

3. Optimizing MVT requests @sandrahoang686 and @dzole0311 are working on a PR that also restricts the datetime range for MVT tile requests

[dzole0311]

Thanks @anayeaye! Re: point 2, I'm already on it via NASA-IMPACT/us-fire-events-tool#40. We recently implemented pagination for the downloads feature in the tool, so I'll be porting the same logic to the fire selection.

Re: point 3, I have a PR ready from a couple of days, but it's not as simple as only limiting the time range and requires deprecating the histogram bars from the chart: NASA-IMPACT/veda-ui#1778, something that we trying to simplify together with @faustoperez

[anayeaye]

@dzole0311 awesome about the pagination--will you also be reducing the limit on the request? I do think there is something server side causing the larger limit=500 to fail in some cases (I should open an issue for us to look at that, too,)

[dzole0311]

@anayeaye do you have a recommendation for the page size? I'm leaning towards 30, but happy to adjust if you think we could increase the value.

[sandrahoang686]

@anayeaye @dzole0311 Similar to what was summarized above. Pushed up the performance test branch script here that led us to playing around with the limit. Script only testing fetchFirePerimeters fn currently. Or the /collections/{collectionId}/items/{itemId} endpoint from /api/features/docs#/. Please feel free to pull branch down to play with it.

Can test fetchLayerPaginated https://github.com/NASA-IMPACT/us-fire-events-tool/blob/main/src/utils/fireUtils.ts#L145-L151 too to see what is the most optimal pageSize. I can get to that later today and add findings. 👍🏼

for fetchFirePerimeters - At a limit=500 (which is what it is at currently here), we can see Internal Server Errors happening consistently. When bumping it down to limit=50, I'm seeing successes across the all the environments. Looks like bumping to limit=100 still also produces all successes but once bumping to limit=150, success rate goes down and past 200, Internal Server Errors are happening consistently. limit=100 looks like a sweet spot.

Findings for 3 different fire Ids:
at limit=500

------------------PRODUCTION----------------------
1 Ids successful out of 3 Ids tested
Success rate: 33.33%
Average success duration: 232.62ms
------------------STAGING----------------------
1 Ids successful out of 3 Ids tested
Success rate: 33.33%
Average success duration: 175.44ms
------------------DEV----------------------
1 Ids successful out of 3 Ids tested
Success rate: 33.33%
Average success duration: 170.47ms
----------------------------------------

at limit=150

2 Ids successful out of 3 Ids tested
Success rate: 66.67%
Average success duration: 388.18ms
------------------STAGING----------------------
2 Ids successful out of 3 Ids tested
Success rate: 66.67%
Average success duration: 491.70ms
------------------DEV----------------------
2 Ids successful out of 3 Ids tested
Success rate: 66.67%
Average success duration: 331.13ms
----------------------------------------

at limit=100

3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 503.82ms
------------------STAGING----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 400.15ms
------------------DEV----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 464.80ms
----------------------------------------

What error looks like:

FireEventsError: Error: Failed to fetch fire perimeters: Internal Server Error
        at handleError (/Users/devseed/Development/VEDA/us-fire-events-tool/src/utils/errorHandler.ts:19:60)
        at fetchFirePerimeters (/Users/devseed/Development/VEDA/us-fire-events-tool/src/utils/fireUtils.ts:37:5)
        at processTicksAndRejections (node:internal/process/task_queues:95:5)
        at testEndpointPerformance (/Users/devseed/Development/VEDA/us-fire-events-tool/src/test-performance.ts:42:22)
        at async Promise.all (index 2)
        at runPerformanceTest (/Users/devseed/Development/VEDA/us-fire-events-tool/src/test-performance.ts:81:29) {
      description: 'Error fetching fire perimeters',
      code: undefined

UPDATE:
Actually looks like we need to test fetchLayerPaginated instead of fetchFirePerimters as it is not the one being referenced any longer

[sandrahoang686]

Alright follow up.. now testing fetchLayerPaginated and changing pageSize in the same test script branch linked above in previous comment...

Here are the results...
when pageSize=30

------------------PRODUCTION----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 982.14ms
------------------STAGING----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 941.20ms
------------------DEV----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 994.94ms
----------------------------------------

when pageSize=100

TEST SUMMARY
------------------PRODUCTION----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 970.01ms
------------------STAGING----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 911.30ms
------------------DEV----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 943.98ms
----------------------------------------

when pageSize=200

------------------PRODUCTION----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 2957.51ms
------------------STAGING----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 3770.14ms
------------------DEV----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 3234.30ms
----------------------------------------

when pageSize=500

------------------PRODUCTION----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 4878.92ms
------------------STAGING----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 4758.20ms
------------------DEV----------------------
3 Ids successful out of 3 Ids tested
Success rate: 100.00%
Average success duration: 4109.14ms
----------------------------------------

Looks like pageSize=30 and pageSize=100 are pretty similar 🤔 . Feel free to pull down and test with different limits.

cc @anayeaye @dzole0311