Sentry bugs/issues

[Ishankoradia]

I see some new type of issues being triggered in Sentry

Endpoint regression (when the duration of a transaction or endpoint containing this transaction increases over time and degrades application performance). These are the apis that sentry notified to have been degraded.

1. /api/prefect/v1/flows/{deployment_id}/flow_run/ | change in duration 2s to 10s (+330.31%) | from Aug 20, 2024 04:00:00 AM UTC
2. /api/prefect/flow_runs/{flow_run_id}/logs | change in duration 1s to 7s (+477.23%) | from Aug 19, 2024 04:00:00 PM UTC
3. /api/prefect/flow_runs/{flow_run_id} | change in duration 2s to 9s (+413.2%) | from Aug 19, 2024 04:00:00 PM UTC
4. /proxy/flow_runs/logs/{flow_run_id} | change in duration 1s to 7s (+558.28%) | from Aug 19, 2024 04:00:00 PM UTC
5. /proxy/flow_runs/{flow_run_id} | change in duration 1s to 9s (+551.03%) | from Aug 19, 2024 04:00:00 PM UTC

Function regression (has a similar meaning ). This could be the reason of the above

1. ddpui.ddpprefect.prefect_service.prefect_post | change in duration 2s to 7s (+298.27%) | from Aug 19, 2024 03:00:00 PM UTC
2. ddpui.utils.custom_logger.CustomLogger.debug | change in duration 30ms to 201ms (+575.74%) | from Aug 19, 2024 06:00:00 PM UTC
3. ddpui.ddpprefect.prefect_service.prefect_get | change in duration 2s to 11s (+617.07%) | from Aug 19, 2024 06:00:00 PM UTC
4. ddpui.utils.custom_logger.CustomLogger.info | change in duration 37ms to 219ms (+490.98%) | from Aug 19, 2024 03:00:00 PM UTC
5. Gunicorn __main__.<module> | change in duration from 41ms to 262ms (+534.56%) | from Aug 19, 2024 03:00:00 PM UTC
6. ddpui.utils.custom_logger.CustomLogger.get_slug | change in duration from 59ms to 201ms (+243.2%) | from Aug 19, 2024 05:00:00 PM UTC

Some N+1 queries sentry picked up

1. /api/prefect/v1/flows/{deployment_id}/flow_run/ | at Aug 20, 6:13 PM
2. ddpui.celeryworkers.tasks.sync_flow_runs_of_deployments | at Aug 21, 12:17 AM
3. /api/transform/dbt_project/graph/ | at Aug 20, 6:50 PM

[Ishankoradia]

Under Endpoint regression it looks like

• (Backend) 2 is due to (proxy) 4
• (Backend) 3 is due to (proxy) 5
• 1 look like it could due Function regresion.1 . It has a prefect_post to create a flow_run

Except for the ddpui.utils.CustomLogger - it feels like. The regression is due to prefect's subpar performance maybe. Django seems to be working fine.

Another thing to notice is all regressions were picked in last 3-4 days. Could it be beause we had some downtime ?

N+1 queries for those apis/functions haven't been optimized in the second round. So that is doable.

[fatchat]

when we moved airbyte to its own machine we downgraded the prod machine from xlarge to large, assuming that with airbyte gone the previous capacity would be excessive

i think these issues are telling us that this decision was wrong

[Ishankoradia]

Could be, but then why prefect is so slow and not django ? Are we saying prefect needs more than 8GB RAM. Too much i feel. Or maybe its the workers.