Skip to content

Comments

feat: introduces workerpool#306

Merged
cx-rogerio-dalot merged 35 commits intomasterfrom
AST-99998-architecture-refactor
Sep 9, 2025
Merged

feat: introduces workerpool#306
cx-rogerio-dalot merged 35 commits intomasterfrom
AST-99998-architecture-refactor

Conversation

@cx-rogerio-dalot
Copy link
Contributor

@cx-rogerio-dalot cx-rogerio-dalot commented Jul 7, 2025

Closes #

Proposed Changes

  • It introduces a workerpool to optimize the detection which wraps around pond
    • Initializes with a default number of workers calculated by number of CPUs x2
      • Reached that conclusion with the benchmark, see my comment below with the stats.
    • This workerpool can be used for other purposes in next steps of the pipeline (in future PRs)
  • On a future PR we won't wait for the detection to end.
  • Fixed some linter issues

Checklist

  • I covered my changes with tests.
  • I Updated the documentation that is affected by my changes:
    • Change in the CLI arguments
    • Change in the configuration file

I submit this contribution under the Apache-2.0 license.

@cx-rogerio-dalot cx-rogerio-dalot requested a review from a team as a code owner July 7, 2025 16:36
@github-actions
Copy link

github-actions bot commented Jul 7, 2025

Logo
Checkmarx One – Scan Summary & Details01a349f1-0a28-488f-ba64-d3d2c0bc4057

Great job! No new security vulnerabilities introduced in this pull request

@cx-rogerio-dalot cx-rogerio-dalot force-pushed the AST-99998-architecture-refactor branch from b7f3ada to 3a71a7c Compare July 7, 2025 16:46
@cx-rogerio-dalot
Copy link
Contributor Author

cx-rogerio-dalot commented Jul 8, 2025

Benchmark Analysis (10 runs each) on a CPU with 8 cores

Performance Summary

50 Items - Best Performers (by mean)

  1. 32 workers: 84.4ms (CoV: 19.5%)
  2. 16 workers: 96.3ms (CoV: 9.6%)
  3. 64 workers: 96.8ms (CoV: 25.6%)

100 Items - Best Performers (by mean)

  1. 128 workers: 143.8ms (CoV: 13.7%)
  2. 16 workers: 158.1ms (CoV: 16.4%)
  3. 32 workers: 163.0ms (CoV: 22.6%)

500 Items - Best Performers (by mean)

  1. 32 workers: 575.6ms (CoV: 8.1%)
  2. 128 workers: 603.9ms (CoV: 9.1%)
  3. 64 workers: 620.8ms (CoV: 9.8%)

1000 Items - Best Performers (by mean)

  1. 8 workers: 1040.7ms (CoV: 8.2%)
  2. 128 workers: 1041.6ms (CoV: 3.9%)
  3. 64 workers: 1055.6ms (CoV: 6.0%)

10000 Items - Best Performers (by mean)

  1. 16 workers: 8659.0ms (CoV: 5.6%)
  2. 64 workers: 8995.3ms (CoV: 0.4%)
  3. 8 workers: 9259.6ms (CoV: 6.5%)

Key Insights

Best Worker Counts (Overall Average)

  1. 16 workers: 2147.4ms average
  2. 64 workers: 2186.5ms average
  3. 8 workers: 2241.5ms average

@cx-rogerio-dalot cx-rogerio-dalot force-pushed the AST-99998-update-makefile-and-linters branch from 76c6ed0 to 615d684 Compare July 8, 2025 15:48
@cx-rogerio-dalot cx-rogerio-dalot force-pushed the AST-99998-architecture-refactor branch from 8c0ae9e to 71aedac Compare July 8, 2025 16:13
Base automatically changed from AST-99998-update-makefile-and-linters to master July 10, 2025 10:10
@cx-rogerio-dalot cx-rogerio-dalot force-pushed the AST-99998-architecture-refactor branch from 71aedac to 066f87c Compare July 10, 2025 10:22
@github-actions
Copy link

github-actions bot commented Jul 10, 2025

kics-logo

KICS version: v1.7.13

Category Results
HIGH HIGH 0
MEDIUM MEDIUM 0
LOW LOW 0
INFO INFO 0
TRACE TRACE 0
TOTAL TOTAL 0
Metric Values
Files scanned placeholder 13
Files parsed placeholder 13
Files failed to scan placeholder 0
Total executed queries placeholder 53
Queries failed to execute placeholder 0
Execution time placeholder 1

@cx-rogerio-dalot cx-rogerio-dalot force-pushed the AST-99998-architecture-refactor branch from de0fcec to ca72934 Compare July 31, 2025 15:13
@cx-rogerio-dalot cx-rogerio-dalot force-pushed the AST-99998-architecture-refactor branch from d51e096 to 0db287f Compare August 26, 2025 16:44
@cx-rogerio-dalot cx-rogerio-dalot changed the base branch from master to feat/update-gitleaks-version August 26, 2025 16:45
@cx-rogerio-dalot cx-rogerio-dalot changed the base branch from feat/update-gitleaks-version to master August 26, 2025 16:45
@cx-rogerio-dalot cx-rogerio-dalot changed the base branch from master to AST-0000-upgrade-gitleaks August 26, 2025 16:46
@cx-rogerio-dalot cx-rogerio-dalot force-pushed the AST-99998-architecture-refactor branch from 1b14461 to 4f31644 Compare August 26, 2025 16:49
Base automatically changed from AST-0000-upgrade-gitleaks to master August 28, 2025 09:51
@cx-rogerio-dalot cx-rogerio-dalot force-pushed the AST-99998-architecture-refactor branch from 7f667b8 to d183911 Compare August 29, 2025 10:28
cx-rui-gomes
cx-rui-gomes previously approved these changes Sep 2, 2025
@cx-rogerio-dalot cx-rogerio-dalot merged commit 6a44739 into master Sep 9, 2025
14 of 15 checks passed
@cx-rogerio-dalot cx-rogerio-dalot deleted the AST-99998-architecture-refactor branch September 9, 2025 09:57
cx-rogerio-dalot added a commit that referenced this pull request Sep 11, 2025
<!--
Thanks for contributing to 2ms by offering a pull request.
-->

Closes #

**Proposed Changes**

- It introduces a workerpool to optimize the detection which wraps
around [pond](https://github.com/alitto/pond)
- Initializes with a default number of workers calculated by number of
CPUs x2
- Reached that conclusion with the benchmark, see my comment below with
the stats.
- This workerpool can be used for other purposes in next steps of the
pipeline (in future PRs)
- On a future PR we won't wait for the detection to end.
- Fixed some linter issues
<!--
Please describe the big picture of your changes here. If it fixes a bug
or resolves a feature request, be sure to link to that issue.
-->

**Checklist**

- [X] I covered my changes with tests.
- [ ] I Updated the documentation that is affected by my changes:
  - [ ] Change in the CLI arguments
  - [ ] Change in the configuration file

I submit this contribution under the Apache-2.0 license.

---------

Co-authored-by: Rogério Dalot <206845486+cx-rogerio-dalot-x@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants