Skip to content

Dynamic length of filter pattern #732

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
babenek merged 20 commits into from
Jul 17, 2025
Merged

Dynamic length of filter pattern #732

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
0c82ecf
variable pattern length for the filter
babenek Jul 5, 2025
56be5c4
style
babenek Jul 5, 2025
b3bcdd1
BM
babenek Jul 5, 2025
35f61fe
BM fix
babenek Jul 5, 2025
6345e95
Merge branch 'main' into kewordgo
babenek Jul 7, 2025
bcdab36
Update benchmark.txt
babenek Jul 7, 2025
6fa7fb9
Merge branch 'main' into kewordgo
babenek Jul 10, 2025
2c79708
Merge remote-tracking branch 'origin/kewordgo' into kewordgo
babenek Jul 10, 2025
8433e61
[skip actions] [kewordgo] 2025-07-12T10:37:15+03:00
babenek Jul 12, 2025
f2c1338
refactoring
babenek Jul 12, 2025
a769a9d
Optional config for filter but required for group
babenek Jul 12, 2025
51ac93c
style & linters fix
babenek Jul 12, 2025
13f7bb1
BM scores fix
babenek Jul 12, 2025
2606871
ValueLengthCheck
babenek Jul 13, 2025
43a453a
Config optimization in group
babenek Jul 13, 2025
838e50e
Merge branch 'main' into kewordgo
babenek Jul 13, 2025
f581d82
Dropbox token fix
babenek Jul 14, 2025
2eadae8
Merge remote-tracking branch 'origin/kewordgo' into kewordgo
babenek Jul 14, 2025
cd99e19
Merge branch 'main' into kewordgo
babenek Jul 16, 2025
b640f75
fix
babenek Jul 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions .ci/benchmark.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ FileType FileNumber ValidLines Positives Negatives
.zsh 6 872 12
.zsh-theme 1 97 1
TOTAL: 11478 16703140 16104 50313
credsweeper result_cnt : 15725, lost_cnt : 0, true_cnt : 15469, false_cnt : 256
credsweeper result_cnt : 15812, lost_cnt : 0, true_cnt : 15521, false_cnt : 291
Rules Positives Negatives Reported TP FP TN FN FPR FNR ACC PRC RCL F1
------------------------------ ----------- ----------- ---------- ----- ---- ----- ---- -------- -------- -------- -------- -------- --------
API 246 3361 235 234 1 3360 12 0.000298 0.048780 0.996396 0.995745 0.951220 0.972973
Expand All @@ -242,7 +242,7 @@ AWS Multi 82 10 34 34
AWS S3 Bucket 67 23 92 67 23 0 0 1.000000 0.000000 0.744444 0.744444 1.000000 0.853503
Akamai Credentials 6 2 6 6 0 2 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Atlassian Old PAT token 5 8 11 5 6 2 0 0.750000 0.000000 0.538462 0.454545 1.000000 0.625000
Auth 1094 2837 1064 1055 9 2828 39 0.003172 0.035649 0.987789 0.991541 0.964351 0.977757
Auth 1094 2837 1081 1072 9 2828 22 0.003172 0.020110 0.992114 0.991674 0.979890 0.985747
Azure Access Token 21 0 13 13 0 0 8 0.380952 0.619048 1.000000 0.619048 0.764706
BASE64 Private Key 22 4 22 22 0 4 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
BASE64 encoded PEM Private Key 12 0 12 12 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Expand All @@ -251,7 +251,7 @@ Bearer Authorization 165 0 165 165
Bitbucket Client ID 36 66 42 25 16 50 11 0.242424 0.305556 0.735294 0.609756 0.694444 0.649351
Bitbucket Client Secret 38 105 86 27 58 47 11 0.552381 0.289474 0.517483 0.317647 0.710526 0.439024
CMD ConvertTo-SecureString 13 4 13 13 0 4 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
CMD Password 29 137 27 27 0 137 2 0.000000 0.068966 0.987952 1.000000 0.931034 0.964286
CMD Password 29 137 29 29 0 137 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
CMD Secret 1 17 1 1 0 17 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
CMD Token 6 2 5 5 0 2 1 0.000000 0.166667 0.875000 1.000000 0.833333 0.909091
Credential 99 498 100 99 1 497 0 0.002008 0.000000 0.998325 0.990000 1.000000 0.994975
Expand All @@ -270,21 +270,21 @@ JSON Web Token 148 61 141 141
JWK 55 0 55 55 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Jira / Confluence PAT token 0 4 0 0 4 0 0.000000 1.000000
Jira 2FA 36 2 31 30 1 1 6 0.500000 0.166667 0.815789 0.967742 0.833333 0.895522
Key 4195 16294 4217 4154 63 16231 41 0.003866 0.009774 0.994924 0.985060 0.990226 0.987637
Nonce 115 50 111 111 0 50 4 0.000000 0.034783 0.975758 1.000000 0.965217 0.982301
Key 4195 16294 4257 4168 89 16205 27 0.005462 0.006436 0.994338 0.979093 0.993564 0.986275
Nonce 115 50 113 113 0 50 2 0.000000 0.017391 0.987879 1.000000 0.982609 0.991228
Other 9 7444 0 0 7444 9 0.000000 1.000000 0.998792 0.000000
PEM Private Key 1142 76 1146 1142 4 72 0 0.052632 0.000000 0.996716 0.996510 1.000000 0.998252
Password 2513 9954 2456 2432 24 9930 81 0.002411 0.032232 0.991578 0.990228 0.967768 0.978869
Password 2513 9954 2462 2433 29 9925 80 0.002913 0.031834 0.991257 0.988221 0.968166 0.978090
SQL Password 44 14 41 41 0 14 3 0.000000 0.068182 0.948276 1.000000 0.931818 0.964706
Salesforce Credentials 6 0 5 5 0 0 1 0.166667 0.833333 1.000000 0.833333 0.909091
Salt 83 75 80 80 0 75 3 0.000000 0.036145 0.981013 1.000000 0.963855 0.981595
Secret 1501 2378 1497 1488 9 2369 13 0.003785 0.008661 0.994328 0.993988 0.991339 0.992662
Secret 1501 2378 1500 1491 9 2369 10 0.003785 0.006662 0.995102 0.994000 0.993338 0.993669
Seed 1 6 0 0 6 1 0.000000 1.000000 0.857143 0.000000
Slack Token 4 1 4 4 0 1 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Stripe Credentials 2 0 2 2 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Tencent WeChat API App ID 47 0 47 47 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Token 947 4640 862 859 3 4637 88 0.000647 0.092925 0.983712 0.996520 0.907075 0.949696
Token 947 4640 879 872 7 4633 75 0.001509 0.079197 0.985323 0.992036 0.920803 0.955093
Twilio Credentials 30 39 30 30 0 39 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
URL Credentials 229 361 229 229 0 361 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
UUID 1866 265 1849 1848 1 264 18 0.003774 0.009646 0.991084 0.999459 0.990354 0.994886
16104 50313 15734 15469 256 50057 635 0.005088 0.039431 0.986585 0.983720 0.960569 0.972007
16104 50313 15821 15521 291 50022 583 0.005784 0.036202 0.986841 0.981596 0.963798 0.972616
6 changes: 3 additions & 3 deletions credsweeper/common/keyword_pattern.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,22 @@
class KeywordPattern:
"""Pattern set of keyword types"""
directive = r"(?P<directive>(?:(?:[#%]define|%global)(?:\s|\\t)|\bset))?"
key_left = r"(?:\\[nrt]|%[0-9a-f]{2}|\s)*" \
key_left = r"(?:\\[nrt]|(\\\\*u00|%)[0-9a-f]{2}|\s)*" \
r"(?P<variable>(([`'\"]{1,8}[^:='\"`}<>\\/&?]*|[^:='\"`}<>\s()\\/&?;,%]*)" \
r"(?P<keyword>"
# there will be inserted a keyword
key_right = r")" \
r"[^%:='\"`<>({?!&;\n]*" \
r")" \
r"(&(quot|apos);|%[0-9a-f]{2}|[`'\"])*" \
r"(&(quot|apos);|(\\\\*u00|%)[0-9a-f]{2}|[`'\"])*" \
r")" # <variable>
separator = r"(?(directive)|(\s|\\{1,8}[tnr])*\]?(\s|\\{1,8}[tnr])*)" \
r"(?P<separator>:(\s[a-z]{3,9}[?]?\s)?=|:(?!:)|=(>|&gt;|(\\\\*u00|%)26gt;)|!==|!=|===|==|=~|=" \
r"|(?(directive)(\\t|\s|\((?!\))){1,80}|%3d))" \
r"(\s|\\{1,8}[tnr])*"
# might be curly, square or parenthesis with words before
wrap = r"(?P<wrap>(" \
r"(new(\s|\\{1,8}[tnr]|byte|char|string|\[\]){1,8})?" \
r"((\s|\\{1,8}[tnr]|new|byte|char|string|\[\]){1,8})?" \
r"(?P<get>([_a-z][0-9a-z_.\[\]]*\.)get|(os\.)?getenv)?" \
r"([0-9a-z_.]|::|-(>|&gt;))*" \
r"\s*" \
Expand Down
4 changes: 2 additions & 2 deletions credsweeper/config/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ def __init__(self, config: Dict[str, Any]) -> None:
self.doc: bool = config["doc"]
self.severity: Severity = Severity.get(config.get("severity"))

self.min_keyword_value_length: int = int(config["min_keyword_value_length"])
self.min_pattern_value_length: int = int(config["min_pattern_value_length"])
self.max_url_cred_value_length: int = int(config["max_url_cred_value_length"])
self.max_password_value_length: int = int(config["max_password_value_length"])

# Trim exclude patterns from space like characters
self.exclude_lines = set(line.strip() for line in self.exclude_lines)
Expand Down
2 changes: 1 addition & 1 deletion credsweeper/filters/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
from credsweeper.filters.value_camel_case_check import ValueCamelCaseCheck
from credsweeper.filters.value_couple_keyword_check import ValueCoupleKeywordCheck
from credsweeper.filters.value_dictionary_keyword_check import ValueDictionaryKeywordCheck
from credsweeper.filters.value_dictionary_value_length_check import ValueDictionaryValueLengthCheck
from credsweeper.filters.value_discord_bot_check import ValueDiscordBotCheck
from credsweeper.filters.value_entropy_base32_check import ValueEntropyBase32Check
from credsweeper.filters.value_entropy_base36_check import ValueEntropyBase36Check
Expand All @@ -29,6 +28,7 @@
from credsweeper.filters.value_json_web_key_check import ValueJsonWebKeyCheck
from credsweeper.filters.value_json_web_token_check import ValueJsonWebTokenCheck
from credsweeper.filters.value_last_word_check import ValueLastWordCheck
from credsweeper.filters.value_length_check import ValueLengthCheck
from credsweeper.filters.value_method_check import ValueMethodCheck
from credsweeper.filters.value_not_allowed_pattern_check import ValueNotAllowedPatternCheck
from credsweeper.filters.value_not_part_encoded_check import ValueNotPartEncodedCheck
Expand Down
4 changes: 3 additions & 1 deletion credsweeper/filters/filter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
from abc import abstractmethod, ABC
from typing import Optional

from credsweeper.config.config import Config
from credsweeper.credentials.line_data import LineData
Expand All @@ -9,7 +10,8 @@ class Filter(ABC):
"""Base class for all filters that operates on 'line_data' objects."""

@abstractmethod
def __init__(self, config: Config, *args):
def __init__(self, config: Optional[Config], *args):
"""Config is optional for a filter"""
raise NotImplementedError()

@abstractmethod
Expand Down
53 changes: 22 additions & 31 deletions credsweeper/filters/group/group.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,31 @@ class Group(ABC):
"""Abstract Group class"""

def __init__(self, config: Config, rule_type: GroupType = GroupType.DEFAULT) -> None:
"""Config is required for filter group"""
if rule_type == GroupType.KEYWORD:
self.filters: List[Filter] = self.get_keyword_base_filters(config)
self.__filters = [ #
ValueAllowlistCheck(), #
ValueArrayDictionaryCheck(), #
ValueBlocklistCheck(), #
ValueCamelCaseCheck(), #
ValueFilePathCheck(), #
ValueHexNumberCheck(), #
ValueLastWordCheck(), #
ValueMethodCheck(), #
ValueSimilarityCheck(), #
ValueStringTypeCheck(check_for_literals=config.check_for_literals), #
ValueTokenCheck(), #
]
if not config.doc:
self.__filters.extend([ValuePatternCheck(), ValueNotAllowedPatternCheck()])
elif rule_type == GroupType.PATTERN:
self.filters: List[Filter] = self.get_pattern_base_filters(config)
self.__filters = [ #
LineSpecificKeyCheck(), #
ValuePatternCheck(), #
]
else:
self.filters: List[Filter] = []
# GroupType.DEFAULT
self.__filters = []

@property
def filters(self) -> List[Filter]:
Expand All @@ -40,31 +59,3 @@ def filters(self) -> List[Filter]:
def filters(self, filters: List[Filter]) -> None:
"""property setter"""
self.__filters = filters

@staticmethod
def get_keyword_base_filters(config: Config) -> List[Filter]:
"""returns base filters"""
filters = [ #
ValueAllowlistCheck(),
ValueArrayDictionaryCheck(),
ValueBlocklistCheck(),
ValueCamelCaseCheck(),
ValueFilePathCheck(),
ValueHexNumberCheck(),
ValueLastWordCheck(),
ValueMethodCheck(),
ValueSimilarityCheck(),
ValueStringTypeCheck(config),
ValueTokenCheck(),
]
if not config.doc:
filters.extend([ValuePatternCheck(pattern_len=config.pattern_len), ValueNotAllowedPatternCheck()])
return filters

@staticmethod
def get_pattern_base_filters(config: Config) -> List[Filter]:
"""return base filters for pattern"""
return [ #
LineSpecificKeyCheck(), #
ValuePatternCheck(pattern_len=config.pattern_len), #
]
13 changes: 7 additions & 6 deletions credsweeper/filters/group/password_keyword.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from credsweeper.common.constants import GroupType
from credsweeper.config.config import Config
from credsweeper.filters import ValueDictionaryValueLengthCheck, LineGitBinaryCheck
from credsweeper.filters import ValueLengthCheck, LineGitBinaryCheck
from credsweeper.filters import ValueSplitKeywordCheck
from credsweeper.filters.group.group import Group
from credsweeper.filters.line_uue_part_check import LineUUEPartCheck
Expand All @@ -11,8 +11,9 @@ class PasswordKeyword(Group):

def __init__(self, config: Config) -> None:
super().__init__(config, GroupType.KEYWORD)
self.filters.extend(
[ValueDictionaryValueLengthCheck(),
ValueSplitKeywordCheck(),
LineGitBinaryCheck(),
LineUUEPartCheck()])
self.filters.extend([
ValueLengthCheck(max_len=config.max_password_value_length),
ValueSplitKeywordCheck(),
LineGitBinaryCheck(),
LineUUEPartCheck()
])
2 changes: 1 addition & 1 deletion credsweeper/filters/group/token_pattern.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@ def __init__(self, config: Config) -> None:
ValueCoupleKeywordCheck(),
ValueNumberCheck(),
ValueCamelCaseCheck(),
ValuePatternCheck(pattern_len=config.pattern_len)
ValuePatternCheck(),
]
12 changes: 6 additions & 6 deletions credsweeper/filters/group/url_credentials_group.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
from credsweeper.common.constants import GroupType
from credsweeper.config.config import Config
from credsweeper.filters import (ValueAllowlistCheck, ValueArrayDictionaryCheck, ValueBlocklistCheck,
ValueCamelCaseCheck, ValueDictionaryValueLengthCheck, ValueFilePathCheck,
ValueLastWordCheck, ValueMethodCheck, ValueNotAllowedPatternCheck, ValuePatternCheck,
ValueStringTypeCheck, ValueTokenCheck)
ValueCamelCaseCheck, ValueLengthCheck, ValueFilePathCheck, ValueLastWordCheck,
ValueMethodCheck, ValueNotAllowedPatternCheck, ValuePatternCheck, ValueStringTypeCheck,
ValueTokenCheck)
from credsweeper.filters.group.group import Group


Expand All @@ -25,9 +25,9 @@ def __init__(self, config: Config) -> None:
ValueFilePathCheck(),
ValueLastWordCheck(),
ValueMethodCheck(),
ValueStringTypeCheck(config),
ValueStringTypeCheck(check_for_literals=config.check_for_literals),
ValueNotAllowedPatternCheck(),
ValueTokenCheck(),
ValueDictionaryValueLengthCheck(min_len=4, max_len=80),
ValuePatternCheck(pattern_len=config.pattern_len)
ValueLengthCheck(max_len=config.max_url_cred_value_length),
ValuePatternCheck()
]
2 changes: 1 addition & 1 deletion credsweeper/filters/group/weird_base36_token.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ def __init__(self, config: Config) -> None:
super().__init__(config, GroupType.DEFAULT)
self.filters = [
ValueCoupleKeywordCheck(),
ValuePatternCheck(config),
ValuePatternCheck(),
ValueNumberCheck(),
ValueTokenBase36Check(),
ValueEntropyBase36Check()
Expand Down
2 changes: 1 addition & 1 deletion credsweeper/filters/group/weird_base64_token.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ def __init__(self, config: Config) -> None:
ValueBase64DataCheck(),
ValueTokenBase64Check(),
ValueEntropyBase64Check(),
ValuePatternCheck(config),
ValuePatternCheck(),
ValueNotPartEncodedCheck(),
ValueBase64PartCheck(),
]
3 changes: 2 additions & 1 deletion credsweeper/filters/line_git_binary_check.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import base64
import contextlib
import re
from typing import Optional

from credsweeper.config.config import Config
from credsweeper.credentials.line_data import LineData
Expand All @@ -12,7 +13,7 @@ class LineGitBinaryCheck(Filter):
"""Checks that line is not a part of git binary patch"""
base85string = re.compile(r"^[A-Za-z][0-9A-Za-z!#$%&()*+;<=>?@^_`{|}~-]{6,65}$")

def __init__(self, config: Config = None) -> None:
def __init__(self, config: Optional[Config] = None) -> None:
pass

def run(self, line_data: LineData, target: AnalysisTarget) -> bool:
Expand Down
3 changes: 2 additions & 1 deletion credsweeper/filters/line_specific_key_check.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import re
from typing import Optional

from credsweeper.common.constants import ML_HUNK
from credsweeper.config.config import Config
Expand All @@ -14,7 +15,7 @@ class LineSpecificKeyCheck(Filter):
NOT_ALLOWED = [r"example", r"\benc[\(\[]", r"\btrue\b", r"\bfalse\b"]
NOT_ALLOWED_PATTERN = re.compile(Util.get_regex_combine_or(NOT_ALLOWED), re.IGNORECASE)

def __init__(self, config: Config = None) -> None:
def __init__(self, config: Optional[Config] = None) -> None:
pass

def run(self, line_data: LineData, target: AnalysisTarget) -> bool:
Expand Down
3 changes: 2 additions & 1 deletion credsweeper/filters/line_uue_part_check.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import re
from typing import Optional

from credsweeper.config.config import Config
from credsweeper.credentials.line_data import LineData
Expand All @@ -10,7 +11,7 @@ class LineUUEPartCheck(Filter):
"""Checks that line is not a part of UU encoding only for maximal line"""
uue_string = re.compile(r"^M[!-`]{60}$")

def __init__(self, config: Config = None) -> None:
def __init__(self, config: Optional[Config] = None) -> None:
pass

def run(self, line_data: LineData, target: AnalysisTarget) -> bool:
Expand Down
3 changes: 2 additions & 1 deletion credsweeper/filters/value_allowlist_check.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import re
from typing import Optional

from credsweeper.config.config import Config
from credsweeper.credentials.line_data import LineData
Expand Down Expand Up @@ -40,7 +41,7 @@ class ValueAllowlistCheck(Filter):

ALLOWED_UNQUOTED_PATTERN = re.compile(Util.get_regex_combine_or(ALLOWED_UNQUOTED), flags=re.IGNORECASE)

def __init__(self, config: Config = None) -> None:
def __init__(self, config: Optional[Config] = None) -> None:
pass

def run(self, line_data: LineData, target: AnalysisTarget) -> bool:
Expand Down
Loading
Loading