Skip to content

Comments

Add /v1/project/batchDelete API method that deletes with SQL#4383

Merged
nscuro merged 9 commits intoDependencyTrack:masterfrom
mikael-carneholm-2-wcar:master
Dec 19, 2024
Merged

Add /v1/project/batchDelete API method that deletes with SQL#4383
nscuro merged 9 commits intoDependencyTrack:masterfrom
mikael-carneholm-2-wcar:master

Conversation

@mikael-carneholm-2-wcar
Copy link
Contributor

Description

This patch adds a /v1/project/batchDelete method that makes project life-cycle management more effective since multiple projects can be deleted per request.

Addressed Issue

This fixes #3361

Additional Details

This is a modified version of #3407 that instead uses pure SQL to do the deletion. And by doing so, it's much faster:

  • The previous implementation was able to delete 61 projects/second (best case, with a batch size of 1000)
  • This implementation has been verified to delete 430 projects/second (with a batch size of 1000)

A caveat is that it uses SQL Common Table Expressions (CTEs) which are part of the ANSI standard since SQL:1999, but still not supported by all DB vendors. The H2 database used for the unit tests has experimental support for CTEs, but does not support CTE DELETE statements. For this reason the code checks the name of the JDBC driver to conditionally run certain statements if the driver is org.postgresql.Driver.

Benchmarks using a local sample DB populated with ~11000 projects imported from a production system:

batch_size      time
100             0m34.193s
200             0m27.759s
500             0m31.075s
1000            0m25.451s

NB: The effect of the batch size is limited by the checkpoint configuration (see https://www.enterprisedb.com/blog/basics-tuning-checkpoints), but in this test the container DB was running with a default configuration.

Checklist

  • I have read and understand the contributing guidelines
    - [ ] This PR fixes a defect, and I have provided tests to verify that the fix is effective
  • This PR implements an enhancement, and I have provided tests to verify that it works as intended
    - [ ] This PR introduces changes to the database model, and I have added corresponding update logic
    - [ ] This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

Signed-off-by: Mikael Carneholm <mikael.carneholm.2@wirelesscar.com>
@nscuro nscuro added the enhancement New feature or request label Nov 19, 2024
@nscuro nscuro added this to the 4.13 milestone Nov 19, 2024
@codacy-production
Copy link

codacy-production bot commented Nov 19, 2024

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
+0.07% (target: -1.00%) 92.25% (target: 70.00%)
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (45982a2) 22797 18074 79.28%
Head commit (51fb679) 22926 (+129) 18193 (+119) 79.36% (+0.07%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#4383) 129 119 92.25%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more

@mikael-carneholm-2-wcar
Copy link
Contributor Author

@nscuro Am I expected to mark the conversations as resolved, or is there anything else blocking this PR from getting merged?

Use DbUtil to check DB type, use Project.id instead of uuid in queries and don't delete tags, notifications or policies.

Signed-off-by: Mikael Carneholm <mikael.carneholm.2@wirelesscar.com>
@nscuro nscuro merged commit bff8b05 into DependencyTrack:master Dec 19, 2024
9 checks passed
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 19, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make project life-cycle management more effective

2 participants