Skip to content

Conversation

@mmccrackan
Copy link
Contributor

@mmccrackan mmccrackan commented Dec 9, 2025

Still investigating...

update_smurf_caldbs appears to have a runtime that increases slowly over time when running in Prefect. Example flow:

https://prefect.simonsobs.org/flow-runs/flow-run/6a855656-459b-4ddd-ac71-de01911155c5?tab=Logs

Even though there were no detsets to add, it still took more than one hour. This slowdown must be occurring either in smurf_detset_info or in update_det_caldb but before line 470 meaning that it would be nearly entirely due to loading db retrieval time only since almost all of the Prefect run time is taken up before that line.

Running these functions manually on compute11, I don't immediately reproduce such a long duration. I did see that get_obs_with_detsets is slow (on the order of ~5 minutes), so I have updated the query to be much faster.

One thing I did find was that when I tried querying the db on compute21 and doing the same on compute11 at the same time, the one on compute21 ran normally, but when it finished (with a locked db error because it tried to execute another query) the one on compute11 dramatically slowed down, putting it exactly in line with how long the update_smurf_caldbs loading are taking.

Looking at this flow (https://prefect.simonsobs.org/flow-runs/flow-run/b0abc8f2-51ca-4d26-a2c5-eb4f8af9537e), there is basically no time spent before line 470, so it is not a consistent duration which I think points to it being the above data loading issue.

@mmccrackan mmccrackan changed the title Change obsid with detset query Change update_smurf_caldbs obsid with detset query Dec 9, 2025
@mmccrackan mmccrackan requested a review from tdalford December 10, 2025 15:13
@mmccrackan mmccrackan marked this pull request as ready for review December 11, 2025 20:07
@mmccrackan
Copy link
Contributor Author

Just to follow up on this, but I added an index with

# Create the index if it doesn't exist
    cursor.execute("""
        CREATE INDEX IF NOT EXISTS idx_files_detset
        ON files(detset)
    """)

which speeds up the sqlite query by a lot (a few seconds). The code in this PR is still faster than that but isn't strictly necessary after the change I think.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants