Skip to content

Conversation

@dolfinus
Copy link
Member

@dolfinus dolfinus commented Sep 19, 2025

Change Summary

Change consumer DB rows fetching pattern from:

for item in items:
  existing = await db.get(item)
  if not existing:
    existing = await.db.create(item)

To:

existing_list = await db.get_bulk(items)
for item in items:
  if item not in existing_list:
    await db.create(item)

This is done in new DatabaseSaver class which replaces save_to_db function.

Before - 760RPS, 10.04% on save_to_db() + 5.28% on SQLAlchemy query cache = 15.32% of al traces.
profile_consumer_old

After - 860RPS, 3.51% on DatabaseSaver.save() + 1.02% on SQLAlchemy query cache = 3.53% of all traces.
profile_consumer_bulk

Related issue number

Checklist

  • Commit message and PR title is comprehensive
  • Keep the change as small as possible
  • Unit and integration tests for the changes exist
  • Tests pass on CI and coverage does not decrease
  • Documentation reflects the changes where applicable
  • docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change
    (see CONTRIBUTING.rst for details.)
  • My PR is ready to review.

@dolfinus dolfinus self-assigned this Sep 19, 2025
@github-actions
Copy link

github-actions bot commented Sep 19, 2025

Coverage

Coverage Report •
FileStmtsMissCoverMissing
data_rentgen/consumer
   saver.py1281389%37–38, 140–146, 148–149, 152–153
data_rentgen/db/repositories
   column_lineage.py58787%65, 74, 87, 96, 106, 124, 142
   job.py67395%121, 143–144
   location.py69198%141
   run.py923166%123–125, 133–136, 139–143, 147–149, 152–155, 158–159, 162, 177–179, 187, 204–206, 208–209
   user.py29293%42–43
data_rentgen/db/scripts/seed
   __main__.py60493%78–79, 84–85
TOTAL696439894% 

@dolfinus dolfinus marked this pull request as ready for review September 22, 2025 06:59
@dolfinus dolfinus merged commit 9c9d646 into develop Sep 22, 2025
11 checks passed
@dolfinus dolfinus deleted the feature/DOP-28871 branch September 22, 2025 12:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants