Change update_smurf_caldbs obsid with detset query #1491
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Still investigating...
update_smurf_caldbsappears to have a runtime that increases slowly over time when running in Prefect. Example flow:https://prefect.simonsobs.org/flow-runs/flow-run/6a855656-459b-4ddd-ac71-de01911155c5?tab=Logs
Even though there were no detsets to add, it still took more than one hour. This slowdown must be occurring either in smurf_detset_info or in
update_det_caldbbut before line 470 meaning that it would be nearly entirely due to loading db retrieval time only since almost all of the Prefect run time is taken up before that line.Running these functions manually on
compute11, I don't immediately reproduce such a long duration. I did see thatget_obs_with_detsetsis slow (on the order of ~5 minutes), so I have updated the query to be much faster.One thing I did find was that when I tried querying the db on compute21 and doing the same on compute11 at the same time, the one on compute21 ran normally, but when it finished (with a locked db error because it tried to execute another query) the one on compute11 dramatically slowed down, putting it exactly in line with how long the
update_smurf_caldbsloading are taking.Looking at this flow (https://prefect.simonsobs.org/flow-runs/flow-run/b0abc8f2-51ca-4d26-a2c5-eb4f8af9537e), there is basically no time spent before line 470, so it is not a consistent duration which I think points to it being the above data loading issue.