v1.1.0

This releases contains primarily:

• Performance improvements on rules that uses regex or hex strings with
  complex alternations (for example, { ( AA ?? BB ) | ( CC ?? DD ) }.
  This can brings up to a 50% reduction to scan duration for rule sets
  that contain such rules.
• A change in some parsing limits that could be used in very complex rules.
  The limits have been raised and are now configurable if needed.
• A compatibility with free-threaded python for boreal-py.
• A way to provide a value for the pe.is_signed symbol, so that this
  value can be computed externally and used in yara rules.

See the complete changelog for details.

boreal-parser

Added

• Allow configuring a few previously hardcoded parsing limits. There were two
  limits used to prevent stack overflows that were hardcoded. Those limits
  have been raised (since they were hit in legitimate rules), and they
  are additionally configurable in boreal, boreal-py and boreal-cli
  if needed. #241

Changed

• The hardcoded parsing limits used to prevent stack overflows have been
  raised. #241
• codespan-reporting upgraded to 0.13.

boreal

Added

• Add a way to provide a value for the pe.is_signed symbol. A new
  PeData object can be used to provide this value before scanning.
  #246
• Expose the new configurable parsing limits in CompilerParams:
  parse_expression_recursion_limit and parse_string_recursion_limit.
  raised. #241

Changed

• Some significant scan duration improvements thanks to changes in the
  atoms extraction logic. Improvements range from 0 to up to 40-50%
  for some rule sets. #245
• Some dependencies upgrade:
  • windows-sys upgraded to 0.61
  • mach2 upgraded to 0.6 (used with process feature on macos targets).
  • tlsh2 upgraded to 1.1.0
  • codespan-reporting upgraded to 0.13
  • object upgraded to 0.37

Fixed

• Fix possible duplicated matches for very specific rules where the
  same literal is repeated in alternatives branches of a regex. This
  could impact rules that depends on a count of the number of matches
  of those strings. #244
• Fix value of process_memory when scanning processes: this value
  was not properly set to true in this context. This could impact
  a few values from the elf, pe, macho and dotnet modules.
  #245

boreal-py

Added

• Add two new parameters that were previously only settable through
  the global configuration (boreal.set_config):
  • max_match_data in the match method.
  • max_strings_per_rule in the compile method.
    This allows setting those parameters without using a global config
    which impacts all invocations. #236
• Expose a few additional compile options that were available in boreal
  but not configurable in boreal-py: max_condition_depth,
  parse_expression_recursion_limit and parse_string_recursion_limit.
  #243

Changed

• Make the library free-threaded. It can now be used with free-threaded python
  without requiring the GIL to be used. #238
• pyo3 upgraded to 0.27

boreal-cli

Added

• Add a few additional options that were previously configurable in boreal
  but not exposed in boreal-cli: #243
  • --match-max-length
  • --max-condition-lepth
  • --disable-includes
    Make the newly configurable parsing limits available in boreal-cli as well.
  • --parse-expression-recursion-limit
  • --parse-string-recursion-limit

v1.0.0

This release marks the first stable release of Boreal as it is now entirely feature complete with YARA:

• All features available in YARA are available in boreal
• A CLI binary is available that is 100% compatible with invocations of the yara CLI binary.
• A new python library has been released, also 100% compatible with the yara python library.

Here is a list of the major features since 0.9.0:

• Python bindings have been added, and are available through the boreal-python pypi package. Those bindings provide a "yara compatibility" mode with full compatibility with the yara python bindings, allowing seamless transition from it.

• The boreal CLI tool has been reworked and completed. It now supports all options from the yara CLI tool (except --atom-quality-table) and provides a "yara compatibility" mode through the use of the yr subcommand, allowing seamless transition from it.

• A scanner can now be serialized into bytes (and deserialized) through the serialize feature. This is the equivalent of the yara save/load API. This however increases the scanner size by a few percent, and has a lot of caveats linked to it. See the Scanner::to_bytes for more details.

• All scanning APIs now have a variant that uses a callback. This callback will be called on specific events: when a rule matches, when a module is imported, etc. scanner::ScanEvent for more details.

• Several more features have been added: modifying the include callback, limiting the number of strings per rule, etc. See the changelog below for details.

• The benchmarks have been reworked: YARA-X have been added to it and benchmarks on the serialize feature have been added.

Since this is the first stable release, several breaking changes have been done to stabilize the API. See the breaking changes list just below.

Breaking changes:

• The Compiler::into_scanner method has been renamed to Compiler::finalize #226.
• The namespace field for rules is now non optional, and the default namespace is named "default". This means that if you previously added rules in the default namespace and rules in a custom namespace named "default", this would now add to the same namespace and may conflict. This aligns the logic on what yara does and simplifies compatibility 4ffca07
• The ScanResult::statistics field is now boxed. This reduces the size of the object greatly.
• The ScanResult::module_values field has been replaced by ScanResult::modules, which also returns a pointer to the modules, allowing access to their static values #225.
• boreal::scanner::MatchedRule has been renamed to boreal::scanner::EvaluatedRule
  979f162.
• boreal::Compiler::default has been removed, use of the CompilerBuilder object is mandatory to customize which modules are enabled 586be27.
• Bump MSRV to 1.74 928e380.
• boreal::scanner::StringMatch::data has changed from a Vec<u8> to a Box<[u8]> to reduce the memory size of this object 928e380.
• boreal::compiler::AddRuleError no longer has a to_short_description method. Instead, this object implements std::fmt::Display which can be used to generate the same short description 6658ebb.

Added

• Added callback based API variants for all Scanner::scan_* methods. For example, Scanner::scan_mem_with_callback, Scanner::scan_process_with_callback. This callback can receive several type of events, and is able to abort the scan during any received event. See boreal::scanner::ScanEvent and boreal::scanner::CallbackEvents for more details on the types of events handled #187.
• Added serialize feature to serialize a Scanner object into bytes which can be deserialized on another computer. See Scanner::to_bytes for more details. #203.
• Added ability to customize include behavior with a callback used during compilation. See Compiler::set_include_callback for more details 637dece.
• Added scan parameters to include not matched rules in results 8a951d8.
• Callback for console module can now be provided in the scanner rather than during compilation 3522484.
• Added Scanner::rules to iterate over the rules contained in a scanner 68ee69b.
• Added max_strings_per_rule compilation parameter to fail compilation if a rule contains too many rules 696ce79.
• Added xor_key field in boreal::scanner::StringMatch to indicate which xor key was used on a given match 7c9fd27.
• Added has_xor_modifier field in boreal::scanner::StringMatches 6853938.
• Implement std::fmt::Display and std::error::Error on boreal::compiler::AddRuleError. This means this is now a real Error object and the AddRuleError::to_short_description method no longer needs to be called to generate a description for the error 6658ebb.

Updated

• update codespan-reporting to 0.12 2b7f394
• update to nom 8.0 d62db91

boreal-cli

See the boreal-cli CHANGELOG file.

v0.9.0

This release brings several memory optimizations and small API improvements.

Memory optimizations comes in two forms:

• Generic optimizations to reduce the memory footprint of compiled rules, useful in all
  cases when the Scanner object is kept for a long time.
• The introduction of a new profile that can be set in the compiler, which will compile
  rules to optimize for memory usage rather than scanning speed.

boreal

Breaking changes:

• A memory pool was introduced to greatly reduce the memory footprint of compiled rules,
  notably when the same meta strings are used in all rules. This introduces two breaking
  changes:

  • The Metadata and MetadataValue objects are no longer re-exported from boreal-parser
    but are new types.
  • To retrieve strings and byte-strings from those objects, the new Scanner::get_bytes_symbol
    and Scanner::get_string_symbol must be used.

• A new CompilerBuilder object is introduced, to be able to configure a Compiler before
  any rule is added.

• Added UnwindSafe and RefUnwindSafe trait bounds on module datas:

  • add UnwindSafe traits to module private datas 43502307
  • add UnwindSafe traits for module user datas 56111d77

• MSRV is bumped from 1.65 to 1.66 825aaab

Added

• Add CompilerBuilder object to add modules and configure compiler profile: 261b11c2
• Add compiler profile to pick between memory usage or scanning speed: #167.
• Add compiler param to disable includes: #170.
• Update compatibility with YARA 4.5.2: #172.

Changed

• Add bytes intern pool to reduce memory consumption: #165.
• Guarantee Scanner is UnwindSafe and RefUnwindSafe: #171.

• Update memory benchmarks 68a1e046

• Update windows-sys dependency to version 0.59 ff996f77
• Update tlsh2 dependency to version 0.4.0 29097dc8

Fixed

• Fix unused warning on statistics in default features config: #168.

boreal-cli

Added

• Added option --profile to select memory or speed profile: c3a89c29.

v0.8.0

This release consists of several changes to make the library easier to use in any context
or target:

• The dependency on OpenSSL (through the authenticode feature) is removed and replaced by pure-Rust dependencies, through the use of two features:

  • The authenticode feature is retained but is now enabled by default. It uses two new dependencies to parse the authenticode signatures.
  • A new authenticode-verify feature is added to handle the pe.is_signed, pe.signatures[*].verified and pe.signatures[*].countersignatures[*].verified fields. See the dedicated documentation for details.

• The patched version of object has been removed, making the use of the library much easier.

Those changes make boreal depend only on Rust libraries (except for the magic feature), which means the library can be used with any targets and is much easier to integrate.

In addition, this release brings full compatibility with YARA 4.5.1.

⚠ Breaking changes

• The authenticode feature has been revamped. It is now split into two features:

  • The authenticode feature, which implements all the pe.signatures field except the ones related to signature verification. This feature is now enabled by default.
  • The authenticode-verify feature, which implements the pe.is_signed and *.verified fields. This feature is disabled by default. See the dedicated documentation for details.

• The Compiler API has been reworked to remove all the ugly workarounds that were needed due to the unsafety brought by the OpenSSL dependency. The Compiler::new_with_pe_signatures and Compiler::new_without_pe_module functions has been removed.

Added

• add authenticode-verify feature for signature verification 9ced02bf.

Changed

• Remove hex dependency bb46e49e
• Remove object patched version #159.
• Replace authenticode-parser dependency with a custom impl f9521c5c
• Remove authenticode-parser dependency and clean API 21c5cd74
• Enable hash dependencies when authenticode feature is enabled b88fedb6

YARA 4.5.1 compatibility:

• only consider valid ascii bytes for pe dll names c219245e.
• add some safety checks in pe module for corrupted values 00235005
• update rva resolution in pe module 66c2d5f4
• list dotnet resources that are not located in the file b2fa436d

Fixed

• limit size of version info key and value in pe module 4a20f5c4
• fix parsing issues in version_info of pe module 8c00218a

v0.7.0

This release adds the last missing modules from YARA: magic, dex and cuckoo.
It also fixes some bugs related to the use of global rules.

Added:

• The magic module is now available behind the magic feature (not enabled by default). #139.
• The dex module is now available behind the object feature (enabled by default). #141.
• The cuckoo module is now available behind the cuckoo feature (not enabled by default). #143, #144.

Fixed:

• Fix evaluation bug when global rules were declared after non-global rules. #146.
  If the global rules had any strings, it would make the evaluation of the rules that followed it invalid.
• Fix application of global rules to namespaces. #147, #149.
  Global rules were applied to all namespaces instead of only their own namespaces.

Changed:

• The type of boreal::module::StaticValue::Function and of the callback declared in the console module has changed from Arc<Box<...>> to Arc<...>. #142.
• Error reporting has been improved on IO error on the rules file. #140.

v0.6.0

This release mainly adds the dotnet module and simplifies a few dependencies.

boreal

Added:

• The dotnet module is now available behind the object feature (enabled by default). #127, #131, #133, #135.

Fixed:

• Fixed compilation when using --no-default-features and other feature combinations. #129, #130.
• Fixed exposure of some optional dependencies as their own features. #128.
• Added CI jobs to ensure common combinations of features compile and run tests properly. #132.

Changed:

• The bitmap dependency has been removed and replaced by an custom implementation for our very limited usecase. #120.
• The windows dependency has been replaced by windows-sys. #137.
• All dependencies have been updated to their latest versions.

Thanks to @demoray for their contributions.

v0.5.0

This release mainly consists of Yara 4.5 compatibility features and fixes:

Added:

YARA 4.5 support:

• New Warning on unknown escape sequences in regexes. See PR #68.
  This warning is more broad than the YARA one from YARA 4.5.
• always expose pe.is_signed 97d1d11
• Do not report strings whose name starts with _ as unused 1a8a8cd
• Add pe.export_details[*].rva field 7597d3f
• math.count and math.percentage now returns an undefined value when given a
  value outside the [0; 255] range. 6a09ed2
• Imported dlls are ignored if the dll name is longer than 255 bytes 28f8626
• Fix endianness issue in macho.magic field, see the Yara fix 50d418d
• filter imported functions with invalid name in pe module 5a0cb4e
• bump limit on number of listed export symbols in pe module to 16384 98032b3

Changed:

• crc32-fast dependency updated to 1.4 f1ae01a
• authenticode-parser dependency updated e68dde7

Fixed:

• Exclude test assets in package 24ca838.
  This avoids having the package be flagged by antiviruses, as unfortunately, some of the binaries copied from the yara repository
  and used for testing seems trigger false positives.

v0.4.0

This release introduces process memory scanning, implemented on Windows, Linux and macOS. In addition, different modes of scanning are available, documenting the exact semantics of scanning a process memory. This allows picking a mode that is less surpresing and faster than the default mode which reproduces YARA's behavior. See FragmentedScanMode for more details, as well as the updated updated benchmarks.

In addition, an API to scan fragmented memory is now available. This is the API which is used during process scanning, and allows custom handling of which memory blocks to scan.

Finally, a few additional features have been added, including an API to mmap files to scan, and the ability to get partial results when the scanning fails, for example due to a timeout.

v0.3.1

Quick release to add a missing feature in boreal: tags and metadatas of matched rules were not available in the scan results.

Detailed changelog:

Boreal:

• Add rule metadata and tags in results of scans. Only the rule name and namespace was listed, which was an oversight.
  In addition, the Metadata and MetadataValue structs from boreal-parser are re-exported, to avoid having to depend on it to
  inspect matched rules metadatas.
  See PR #85.

v0.3.0

This is a huge release containing several months of work, including:

• Full compatibility with Yara 4.3. All the new features from Yara 4.3 are available.

• A complete rewrite of the strings compilation algorithm. Performance has been improved dramatically when using a lot of rules or when using strings of lesser quality. See the updated benchmarks.

• New tools to debug and improve performances of rules scanning, which new flags to display several kind of statistics.

  • Strings statistics can now be computed: how are strings compiled, the quality of the extracted atoms, ...
  • Evaluation duration statistics can now be computed, detailing how long each evaluation step takes. This is only available if the new profiling feature is enabled, to not impact evaluation performance if not set.

• Improved testing on modules and on the boreal-cli binary.

Here are some more details on the new YARA features:

Yara 4.3:

• Negation in hex strings, eg { ~C3 ~?F }.
• New to_string function in math module.
• New string module with to_int and length functions.
• rva field in imported functions in pe module.
• pe.import_rva and pe.delayed_import_rva functions.
• pe.rich_signature.version_data field.
• Iterator on bytes literal, eg for any s in ("foo", "bar"): (...).
• at for expression, eg any of them at 0.
• New functions import_md5 and telfhash in elf module.
• Use of the authenticode-parser lib to parse signatures in pe module. This adds a lot of fields in pe.signatures.

Here are the changes grouped by crate:

Boreal

Added

• Yara 4.3 compatibility. Too many features to list, see above for a short recap of the main new features.
• New profiling feature, needed to compute evaluation statistics.

Changed

• Rewrite of the strings compilation algorithm to significantly improve statistics.
• openssl feature removed, replaced with the authenticode feature.
• Using the pe module with the signatures parsing now requires calling the unsafe function Compiler::new_with_pe_signatures.
• All dependencies updated. regex has been removed in favor of regex-automata.

Fixed

• Improved handling on invalid ranges in '$a in (from..to)' expression.
• Fixed minor differences in edge cases in elf.dynamic_section_entries and elf.number_of_sections (e639df643b05).
• Fixed == operator on boolean values (cec439eee19f).
• Fixed some bugs occuring when using the fullword keyword with both the wide and ascii modifiers, see PR #51.
• Fix compilation of rules following the failed compilation of a rule using a rule dependency. I doubt this actually impacted anyone, see PR #60.
• Change regex behavior to allow non ascii bytes in regexes. See PR #62. A warning has however been added to warn against this situation.
• Fixed string comparison in the pe.imports and pe.(delayed_)import_rva functions to be case-insensitive, See PR #69.

boreal-cli

Added

• New -M flag to a list of available modules.
• New --string-stats flag to display strings' compilation statistics.
• New --scan-stats flag to display evaluation duration statistics.

Changed

• Number of dependencies reduced by removing any use of proc macros.
• boreal updated to 0.3, see boreal changes.

boreal-parser

Added

• Parsing of negation in hex strings, eg { ~C3 ~?F } (9c21fd446).
• Parsing of at for expression, eg any of them at 0 (b26fbc3b6).
• parse_regex and parse_hex_string added to public API (d6a7afc98).

Changed

• Exports of the crate have been entirely reworked. Objects are now nested in relevant modules (3e8682bec).
• Removal of bitflags dependency, rework of VariableModifiers object (05877aae4).
• Regex now accepts non ascii bytes when not in a class. See PR #62.
• AST for bytes and characters in a regex has been updated to provide escaping information and span location. See PR #68.

Fixed

• Some public objects were not properly exposed publicly, this should now be fixed (3e8682bec).