-
-
Notifications
You must be signed in to change notification settings - Fork 616
Created domain typosquatting checker #1644
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
zbalkan
wants to merge
62
commits into
TechnitiumSoftware:master
Choose a base branch
from
zbalkan:app/typosquattingdetector
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
62 commits
Select commit
Hold shift + click to select a range
9d2ad83
Created domain typosquatting checker
zbalkan 109df58
Added customlist capability
zbalkan 1ef77d6
Final version
zbalkan 3fde001
Foxed performance issue
zbalkan c0eb10b
Added path check
zbalkan 216c9ac
Added exception handling for file parsing
zbalkan 2349023
Disposed the threadlocal normalizer
zbalkan d345ce2
Memory management for detector
zbalkan 9a9fc59
Marked detector as volatile
zbalkan 62c97b4
Reused same httpclient
zbalkan ac3a4b9
Fixed async query
zbalkan d82deb5
Fixed nullability attributes
zbalkan 381049f
Minor fix for null bloomfilter
zbalkan 55be360
Final touches
zbalkan 43e2701
Fixed sync-in-async issue
zbalkan 5ef1b49
Fixed hash-check bug
zbalkan 3be8c44
Fixed timer issue
zbalkan 4e81914
Prioritized user provided domain list
zbalkan 16ee898
Added "m" as a common subdomain name to cleanup
zbalkan a142f23
Optimized fuzzy matching
zbalkan 1ef772e
Minor performance optimizations on hot path
zbalkan ccc064e
Added README
zbalkan 038d255
Updated app description
zbalkan 4340beb
Fixed regions
zbalkan c27794e
Improved null handlng
zbalkan f38c308
Simplified suspicious check
zbalkan ae1d7eb
Simplified Reason enum
zbalkan ea267e0
Added empty or corrupted hash file edge case handling
zbalkan cf26711
Guard clause for httpClient leak edge case
zbalkan 9cc367a
Added default severity.
zbalkan d464196
Updated description
zbalkan 8fb46f3
Made httpclient non-static
zbalkan 6b724e3
Used regular for loop for small instances to minimize overload
zbalkan 94aed97
Removed unused httpclient
zbalkan 8375608
Solving race condition in Parallel.Foreach
zbalkan c57e475
Fixrd concurrency issue in swapping detector
zbalkan b8fbcf9
Volarile _change
zbalkan f4e5968
Concurrency issues, again
zbalkan c111cd1
Fixed regex
zbalkan 79cf25b
Added null check
zbalkan 2af0ae5
Shared lock issue fixed
zbalkan e05fbc3
Normalizatipn function fixed
zbalkan 5b7695e
Simplification
zbalkan ce40194
Optimizations
zbalkan 576be77
Used Lazy static HTTP client issues
zbalkan e82225b
Fixed path issues
zbalkan 2369713
Concurrency is a headache
zbalkan a22f599
Used explicit type instead of var everywhere
zbalkan b934b24
Fixed regex issue
zbalkan c4502fc
Formatting
zbalkan 5ac62ff
Refactor
zbalkan 0ceae8f
Created statepool for allocation issues
zbalkan 24a45a8
Added second-level sharding (length + prefix2), with a prefix1 fallba…
zbalkan 3d2bab5
Optimized score
zbalkan 22f012c
Null check
zbalkan 67fe1b6
Used DomainCache for domain name normalization optimization
zbalkan b37f08a
Optimize Bloom filter shards
zbalkan 78ee5a9
Used bounded statepool to prevent memory allocation issues
zbalkan 7cf90cf
Rolled back accidental changes
zbalkan 95fb88c
Added lock check in Clear
zbalkan 2866ee8
Application config guards added
zbalkan 0f26f9d
Fixed typo
zbalkan File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| /* | ||
| Technitium DNS Server | ||
| Copyright (C) 2025 Shreyas Zare ([email protected]) | ||
| Copyright (C) 2025 Zafer Balkan ([email protected]) | ||
|
|
||
| This program is free software: you can redistribute it and/or modify | ||
| it under the terms of the GNU General Public License as published by | ||
| the Free Software Foundation, either version 3 of the License, or | ||
| (at your option) any later version. | ||
|
|
||
| This program is distributed in the hope that it will be useful, | ||
| but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| GNU General Public License for more details. | ||
|
|
||
| You should have received a copy of the GNU General Public License | ||
| along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
| */ | ||
|
|
||
| using System.ComponentModel.DataAnnotations; | ||
| using System.IO; | ||
| using System.Text.Json.Serialization; | ||
| using System.Text.RegularExpressions; | ||
|
|
||
| namespace TyposquattingDetector | ||
| { | ||
| public sealed partial class App | ||
| { | ||
| private class Config | ||
| { | ||
| [JsonPropertyName("addExtendedDnsError")] | ||
| public bool AddExtendedDnsError { get; set; } = true; | ||
|
|
||
| [JsonPropertyName("allowTxtBlockingReport")] | ||
| public bool AllowTxtBlockingReport { get; set; } = true; | ||
|
|
||
| [JsonPropertyName("disableTlsValidation")] | ||
| public bool DisableTlsValidation { get; set; } = false; | ||
|
|
||
| [JsonPropertyName("enable")] | ||
| public bool Enable { get; set; } = true; | ||
|
|
||
| [JsonPropertyName("fuzzyMatchThreshold")] | ||
| [Range(75, 90, ErrorMessage = "fuzzyMatchThreshold must be between 75 and 90.")] | ||
| [Required(ErrorMessage = "fuzzyMatchThreshold is a required configuration property. The lower threshold means more false positives.")] | ||
| public int FuzzyMatchThreshold { get; set; } = 75; | ||
|
|
||
| [JsonPropertyName("customList")] | ||
| [CustomValidation(typeof(FileContentValidator), nameof(FileContentValidator.ValidateDomainFile))] | ||
| public string? Path { get; set; } | ||
|
|
||
| [JsonPropertyName("updateInterval")] | ||
| [Required(ErrorMessage = "updateInterval is a required configuration property.")] | ||
| [RegularExpression(@"^\d+[mhdw]$", ErrorMessage = "Invalid interval format. Use a number followed by 'm', 'h', or 'd' (e.g., '90m', '2h', '7d').", MatchTimeoutInMilliseconds = 3000)] | ||
| public string UpdateInterval { get; set; } = "30d"; | ||
| } | ||
|
|
||
| public partial class FileContentValidator | ||
| { | ||
| // Optimized Regex: Compiled for performance during "Happy Path" scans | ||
| private static readonly Regex DomainRegex = DomainPattern(); | ||
|
|
||
| public static ValidationResult? ValidateDomainFile(string? path, ValidationContext context) | ||
| { | ||
| // 1. If path is null/empty, we assume validation is not required here | ||
| // (Use [Required] on the property if you want to force a path to be provided) | ||
| if (string.IsNullOrWhiteSpace(path)) return ValidationResult.Success; | ||
|
|
||
| // 2. Existence Check | ||
| if (!File.Exists(path)) | ||
| return new ValidationResult($"File not found: {path}"); | ||
|
|
||
| try | ||
| { | ||
| // 3. Stream through lines | ||
| // If the file is empty, this loop is simply skipped | ||
| foreach (string line in File.ReadLines(path)) | ||
| { | ||
| string trimmedLine = line.Trim(); | ||
|
|
||
| // Skip truly empty lines (whitespace only) | ||
| if (string.IsNullOrEmpty(trimmedLine)) continue; | ||
|
|
||
| // 4. Fail-Fast Logic | ||
| // If any content exists, it MUST follow the domain rules | ||
| if (trimmedLine.Contains('*') || !DomainRegex.IsMatch(trimmedLine)) | ||
| { | ||
| return new ValidationResult($"Invalid content: '{trimmedLine}'. Wildcards are not allowed."); | ||
| } | ||
| } | ||
| } | ||
| catch (IOException ex) | ||
| { | ||
| return new ValidationResult($"File access error: {ex.Message}"); | ||
| } | ||
|
|
||
| // 5. Success Path | ||
| // Reached if the file was empty OR all lines passed validation | ||
| return ValidationResult.Success; | ||
| } | ||
|
|
||
| [GeneratedRegex(@"(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z0-9][a-z0-9-]{0,61}[a-z0-9]", RegexOptions.IgnoreCase | RegexOptions.Compiled, "en-US")] | ||
| private static partial Regex DomainPattern(); | ||
| } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,281 @@ | ||
| /* | ||
| Technitium DNS Server | ||
| Copyright (C) 2025 Shreyas Zare ([email protected]) | ||
| Copyright (C) 2025 Zafer Balkan ([email protected]) | ||
|
|
||
| This program is free software: you can redistribute it and/or modify | ||
| it under the terms of the GNU General Public License as published by | ||
| the Free Software Foundation, either version 3 of the License, or | ||
| (at your option) any later version. | ||
|
|
||
| This program is distributed in the hope that it will be useful, | ||
| but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| GNU General Public License for more details. | ||
|
|
||
| You should have received a copy of the GNU General Public License | ||
| along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
| */ | ||
|
|
||
| using Nager.PublicSuffix; | ||
| using Nager.PublicSuffix.RuleProviders; | ||
| using Nager.PublicSuffix.RuleProviders.CacheProviders; | ||
| using System; | ||
| using System.Collections.Concurrent; | ||
| using System.Net.Http; | ||
| using System.Threading; | ||
|
|
||
| namespace TyposquattingDetector | ||
| { | ||
|
|
||
|
|
||
| /// <summary> | ||
| /// Thread-safe cache for parsed domain information using the SIEVE eviction algorithm. | ||
| /// SIEVE provides better scan resistance than LRU, making it ideal for DNS workloads | ||
| /// where one-time queries (typos, DGA domains) are common. | ||
| /// | ||
| /// Reference: "SIEVE is Simpler than LRU: an Efficient Turn-Key Eviction Algorithm for | ||
| /// Web Caches" (NSDI '24) | ||
| /// </summary> | ||
| internal sealed class DomainCache | ||
| { | ||
| #region variables | ||
|
|
||
| private const int MaxSize = 10000; | ||
| private const int StringPoolMaxSize = 10000; | ||
| private static readonly DomainInfo Empty = new DomainInfo(); | ||
|
|
||
| // ADR: Loading the PSL must not block or fail plugin startup. We defer | ||
| // initialization and make it best-effort to avoid network dependencies. | ||
| private static readonly Lazy<DomainParser?> _parser = new Lazy<DomainParser?>(InitializeParser); | ||
|
|
||
zbalkan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| private static readonly HttpClient _pslHttpClient = new HttpClient(); | ||
|
|
||
| private static readonly Lazy<CachedHttpRuleProvider> _sharedRuleProvider = | ||
| new Lazy<CachedHttpRuleProvider>(static () => | ||
| { | ||
| LocalFileSystemCacheProvider cacheProvider = new LocalFileSystemCacheProvider(); | ||
| CachedHttpRuleProvider rp = new CachedHttpRuleProvider(cacheProvider, _pslHttpClient); | ||
| rp.BuildAsync().GetAwaiter().GetResult(); | ||
| return rp; | ||
| }, isThreadSafe: true); | ||
|
|
||
| private readonly ConcurrentDictionary<string, CacheNode> _cache = | ||
| new ConcurrentDictionary<string, CacheNode>(StringComparer.OrdinalIgnoreCase); | ||
| private readonly ConcurrentDictionary<string, string> _stringPool = | ||
| new ConcurrentDictionary<string, string>(StringComparer.OrdinalIgnoreCase); | ||
| private readonly Lock _evictionLock = new Lock(); | ||
|
|
||
| // SIEVE data structures | ||
| private CacheNode? _head; | ||
| private CacheNode? _tail; | ||
| private CacheNode? _hand; | ||
| #endregion | ||
|
|
||
| #region public | ||
| public DomainInfo GetOrAdd(string domainName) | ||
| { | ||
| if (string.IsNullOrWhiteSpace(domainName)) | ||
| return Empty; | ||
|
|
||
| // Fast path: try cache lookup with original name first (case-insensitive) | ||
| if (_cache.TryGetValue(domainName, out CacheNode? node)) | ||
| { | ||
| node.Visited = true; | ||
| return node.Domain; | ||
| } | ||
|
|
||
| // NormalizeConfig only if needed, using string pool to reduce allocations | ||
| string normalizedName = GetPooledNormalizedName(domainName); | ||
|
|
||
| // Check cache again with normalized name (may differ from original) | ||
| if (!ReferenceEquals(normalizedName, domainName) && | ||
| _cache.TryGetValue(normalizedName, out node)) | ||
| { | ||
| node.Visited = true; | ||
| return node.Domain; | ||
| } | ||
|
|
||
| DomainInfo domain = Parse(domainName); | ||
| AddToCache(normalizedName, domain); | ||
| return domain; | ||
| } | ||
|
|
||
|
|
||
| public void Clear() | ||
| { | ||
| lock (_evictionLock) | ||
| { | ||
| _cache.Clear(); | ||
| _stringPool.Clear(); | ||
| _head = null; | ||
| _tail = null; | ||
| _hand = null; | ||
| } | ||
| } | ||
|
|
||
| #endregion | ||
|
|
||
| #region private | ||
|
|
||
| /// <summary> | ||
| /// Returns a pooled, normalized version of the domain name to reduce allocations. | ||
| /// If the name is already normalized, returns the original string. | ||
| /// </summary> | ||
| private string GetPooledNormalizedName(string name) | ||
| { | ||
| if (!NeedsNormalization(name)) | ||
| return name; | ||
|
|
||
| string normalized = name.ToLowerInvariant().TrimEnd('.'); | ||
|
|
||
| // Try to get from pool, or add if not present | ||
| if (_stringPool.TryGetValue(normalized, out string? pooled)) | ||
| return pooled; | ||
|
|
||
| // Limit pool size to prevent unbounded growth | ||
| if (_stringPool.Count < StringPoolMaxSize) | ||
| { | ||
| _stringPool.TryAdd(normalized, normalized); | ||
| } | ||
|
|
||
| return normalized; | ||
| } | ||
|
|
||
| /// <summary> | ||
| /// Checks if the domain name needs normalization (has uppercase or trailing dot). | ||
| /// </summary> | ||
| private static bool NeedsNormalization(string name) | ||
| { | ||
| if (name.Length > 0 && name[^1] == '.') | ||
| return true; | ||
|
|
||
| foreach (char c in name) | ||
| { | ||
| if (c >= 'A' && c <= 'Z') | ||
| return true; | ||
| } | ||
zbalkan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| return false; | ||
| } | ||
|
|
||
| private static DomainInfo Parse(string name) | ||
| { | ||
| DomainParser? parser = _parser.Value; | ||
| if (parser == null) | ||
| return Empty; | ||
|
|
||
| try | ||
| { | ||
| return parser.Parse(name) ?? Empty; | ||
| } | ||
| catch | ||
| { | ||
| // Parsing errors are intentionally ignored because PSL is optional. | ||
| return Empty; | ||
| } | ||
| } | ||
|
|
||
| private static DomainParser? InitializeParser() | ||
| { | ||
| // ADR: The PSL download via SimpleHttpRuleProvider performs outbound HTTP. | ||
| // Relying on external network connectivity at plugin startup is unsafe in | ||
| // production DNS environments (offline appliances, firewalled networks, | ||
| // corporate proxies). Initialization must never block or fail due to PSL | ||
| // retrieval. We therefore treat PSL availability as optional: | ||
| // - If the download succeeds, domain parsing is enriched. | ||
| // - If it fails, we return null and logging continues without PSL data. | ||
| try | ||
| { | ||
| return new DomainParser(_sharedRuleProvider.Value); | ||
| } | ||
| catch | ||
| { | ||
| return null; | ||
| } | ||
| } | ||
|
|
||
| private void AddToCache(string key, DomainInfo domain) | ||
| { | ||
| lock (_evictionLock) | ||
| { | ||
| if (_cache.ContainsKey(key)) | ||
| return; | ||
|
|
||
| while (_cache.Count >= MaxSize) | ||
| Evict(); | ||
|
|
||
| CacheNode newNode = new CacheNode(key, domain); | ||
| InsertAtHead(newNode); | ||
| _cache[key] = newNode; | ||
| } | ||
| } | ||
|
|
||
| private void InsertAtHead(CacheNode node) | ||
| { | ||
| node.Next = _head; | ||
| node.Prev = null; | ||
|
|
||
| if (_head != null) | ||
| _head.Prev = node; | ||
|
|
||
| _head = node; | ||
|
|
||
| _tail ??= node; | ||
|
|
||
| _hand ??= node; | ||
| } | ||
|
|
||
| private void Evict() | ||
| { | ||
| _hand ??= _tail; | ||
|
|
||
| while (_hand != null) | ||
| { | ||
| if (!_hand.Visited) | ||
| { | ||
| CacheNode victim = _hand; | ||
| _hand = _hand.Prev ?? _tail; | ||
| RemoveNode(victim); | ||
| _cache.TryRemove(victim.Key, out _); | ||
| return; | ||
| } | ||
|
|
||
| _hand.Visited = false; | ||
| _hand = _hand.Prev ?? _tail; | ||
| } | ||
| } | ||
zbalkan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| private void RemoveNode(CacheNode node) | ||
| { | ||
| if (node.Prev != null) | ||
| node.Prev.Next = node.Next; | ||
| else | ||
| _head = node.Next; | ||
|
|
||
| if (node.Next != null) | ||
| node.Next.Prev = node.Prev; | ||
| else | ||
| _tail = node.Prev; | ||
|
|
||
| if (_hand == node) | ||
| _hand = node.Prev ?? _tail; | ||
| } | ||
|
|
||
| private class CacheNode | ||
| { | ||
| public readonly string Key; | ||
| public readonly DomainInfo Domain; | ||
| public volatile bool Visited; | ||
| public CacheNode? Next; | ||
| public CacheNode? Prev; | ||
|
|
||
| public CacheNode(string key, DomainInfo domain) | ||
| { | ||
| Key = key; | ||
| Domain = domain; | ||
| } | ||
| } | ||
| #endregion | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.