perf: improve memory consumption in 2ms file walk

[cx-rui-oliveira]

Proposed Changes

• Separate secret detection logic for filesystem plugin
  • Chunking when the processed file is larger than 10 MB (with peeking so as not to lose the context of the secrets);
  • Weighted semaphore to control the use of memory (with a dynamic budget, i.e., dependent on the host's RAM).

Checklist

• I covered my changes with tests.
• I Updated the documentation that is affected by my changes:
  • Change in the CLI arguments
  • Change in the configuration file

[github-actions]

KICS version: v1.7.13

Category Results HIGH 0 MEDIUM 0 LOW 0 INFO 0 TRACE 0 TOTAL 0	Metric Values Files scanned 11 Files parsed 11 Files failed to scan 0 Total executed queries 53 Queries failed to execute 0 Execution time 1

[github-actions]

Checkmarx One – Scan Summary & Details – 1ebce5bf-6ac3-48aa-be12-0196b419ef01

Great job, no security vulnerabilities found in this Pull Request

[cx-rogerio-dalot]

I think we should have a package under engine, something like engine/semaphore. We can have all the logic there that handles everything related to the weights and memory (including the one in cmd/memory).

In cmd we can then handle the DI of the semaphore by injecting it on the engine instance. It is in the end, the semaphore that knows which memory budget to allocate and how to handle the weights, and the engine, that should operate and depend on the semaphore.

WDYT?

[cx-rui-oliveira]

I think we should have a package under engine, something like engine/semaphore. We can have all the logic there that handles everything related to the weights and memory (including the one in cmd/memory).

In cmd we can then handle the DI of the semaphore by injecting it on the engine instance. It is in the end, the semaphore that knows which memory budget to allocate and how to handle the weights, and the engine, that should operate and depend on the semaphore.

WDYT?

It makes perfect sense, it is a cleaner and more modular approach.

[cx-rogerio-dalot]

Awesome work 🎉