Open
Conversation
Implement multiprocessing for rule processing: Refactored process_rules_file to use multiprocessing.Pool to significantly speed up the evaluation of large rulesets. Add top-level worker functions: Extracted the core rule parsing and checking logic into a new _process_rule_task function so it can be safely pickled and distributed across worker processes. Preserve logging across processes: Added a _worker_init function to initialize each worker process with a QueueHandler, successfully routing child process logs back to the main thread's QueueListener (building on the Koen1999#30 logging setup). Batch file reading: Updated the file reading loop in process_rules_file to sequentially parse the file and group multi-line rules into a list of tasks before handing them off to the process pool. Maintain output order: Used pool.map to ensure that the final output report preserves the exact original order of the rules in the file, regardless of which worker finishes first.
Koen1999
requested changes
Mar 9, 2026
Comment on lines
-462
to
-475
| """Processes a rule file and returns a list of rules and their issues. | ||
|
|
||
| Args: | ||
| rules: A path to a Suricata rules file. | ||
| evaluate_disabled: A flag indicating whether disabled rules should be evaluated. | ||
| checkers: The checkers to be used when processing the rule file. | ||
|
|
||
| Returns: | ||
| A list of rules and their issues. | ||
|
|
||
| Raises: | ||
| RuntimeError: If no checkers could be automatically discovered. | ||
|
|
||
| """ |
Owner
There was a problem hiding this comment.
Can you clarify why documentation was removed in several places?
Comment on lines
+568
to
+574
| # Spin up the process pool | ||
| with multiprocessing.Pool( | ||
| initializer=_worker_init, | ||
| initargs=(log_queue,) | ||
| ) as pool: | ||
| # pool.map preserves the original order of the rules in the file | ||
| results = pool.map(_process_rule_task, tasks) |
Owner
There was a problem hiding this comment.
This approach implies processing only begins after the entire file is read. Perhaps it can start earlier and tasks can be added as lines are read?
Comment on lines
-602
to
-616
| """Checks a rule and returns a dictionary containing the rule and a list of issues found. | ||
|
|
||
| Args: | ||
| rule: The rule to be checked. | ||
| checkers: The checkers to be used to check the rule. | ||
| ignore: Regular expressions to match checker codes to ignore | ||
|
|
||
| Returns: | ||
| A list of issues found in the rule. | ||
| Each issue is typed as a `dict`. | ||
|
|
||
| Raises: | ||
| InvalidRuleError: If the rule does not follow the Suricata syntax. | ||
|
|
||
| """ |
Comment on lines
+455
to
+457
| # ---------------------------------------------------------------------- | ||
| # NEW MULTIPROCESSING WORKER FUNCTIONS | ||
| # ---------------------------------------------------------------------- |
Owner
There was a problem hiding this comment.
These comments should be removed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.