misp-stix 2.4.163 released

v2.4.163 (2022-09-26)

Changes

• [package] New version. [Christian Studer]

Fix

• [stix2 export] Avoiding variables to be referenced before they are declared. [Christian Studer]

• [stix2 export] Fixed the Hash values checking. [Christian Studer]

  • STIX 2 allows some custom Hash types so we don't
    need to consider invalid a hash that is not
    in the list of common supported types

• [stix2 export] Some details fixed on errors handling functions. [Christian Studer]

Other

• Fix; [stix2 export] Added missing check for data fields from attachment attributes. [Christian Studer]

• Wip: [stix2 export] Checking Hash values for object attributes. [Christian Studer]

• Wip: [stix2 export] More Hash values checking. [Christian Studer]

  • We also check now Hash values in the case of a
    conversion as Observable objects

• Wip: [stix2 export] Introducing a hash value checking function to avoid issues with invalid hashes. [Christian Studer]

• Wip: [stix2 import] Added some helpers to parse content in STIX 2 patterns. [Christian Studer]

  • Loading patterns for now

misp-stix 2.4.162 released

v2.4.162 (2022-09-19)

Changes

• [package] Updated to latest version to publish. [Christian Studer]

• [stix2 export] Returning warning as a dictionary of lists instead of sets. [Christian Studer]

• [setup, poetry] Aligning with the package features that are actually used on pypi. [Christian Studer]

• [tests] Ported all STIX 1 export tests to support both JSON & MISP inputs. [Christian Studer]

• [stix2 export] Made the timestamp values checking common to all export classes and moved the test is the values are datetime to this common function. [Christian Studer]

• [tests] Duplicated tests for attributes, objects & galaxies export as STIX 2 to support both JSON & MISP input. [Christian Studer]

• [tests] Tests for interoperability & feeds now support both JSON and MISP inputs. [Christian Studer]

• [stix2 export] Added correct typing to functions receiving attributes, objects or events. [Christian Studer]

  • When the library is used in a python script, we
    can pass directly MISPEvent, MISPAttribute or
    MISPObject objects instead of their JSON format
    It is already working, here we simply fixed the
    functions header with the correct typing

• [doc] add PyPI references. [Alexandre Dulaunoy]

Fix

• [readme] Updated description. [Christian Studer]

• [stix2 export] Added missing use case making available Attributes parsing in some situations while giving the input as file instead of as loaded dict. [Christian Studer]

  • It avoids for instance issues with the command
    line script when giving a file containing an
    attributes collection

• [stix2 export] Fixed edge case when the send-date attribute within an email object is not a correctly formatted datetime value. [Christian Studer]

• [tests] Fixed tests for composite attributes exported as STIX 2 indicator that received a tiny change. [Christian Studer]

• [stix1 export] Fixed composite attribute values parsing to avoid issues with values not formatted the right way. [Christian Studer]

• [stix2 export] Fixed parsing of composite attributes which require some attribute type handling. [Christian Studer]

  • The composite attribute type will indeed always
    have the standard | as separator

• [stix2 export] Handling composite attribute values when they are not formatted as they should be with a | [Christian Studer]

• [stix2 export] Added the missing interoperability parameter in the Relationship object arguments. [Christian Studer]

• [stix2 export] Fixed annotation object export as STIX 2.1 when there is no object reference. [Christian Studer]

• [clean up] Removed debugging print statements. [Christian Studer]

• [tests] Making the datetime to str utility function common to all STIX testing classes. [Christian Studer]

• [stix1 export] Handling the data field while creating an Artifact object. [Christian Studer]

• [stix1 export] Handling some datetime values. [Christian Studer]

• [documentation] Fixed documentation following changes on the lnk objects export to STIX 2.0. [Christian Studer]

• [tests] Fixing some tests triggered by a lot of unit tests to make them work with a MISP input. [Christian Studer]

• [tests] Avoiding issues with the geolocation object & the to_ids value of some asn object attributes. [Christian Studer]

• [stix2 export] Added missing import. [Christian Studer]

• [tests] Better handling of timeline value & the data field. [Christian Studer]

• [stix2 export] Better lnk objects parsing including the timeline attributes export as STIX 2.0 that were missing. [Christian Studer]

• [stix2 export] Correctly handling the timestamp fields and values. [Christian Studer]

• [stix2 export] Handling properly data fields in attributes and object attributes. [Christian Studer]

• [stix2 export] Handling some timestamp values depending whether they are datetime or str. [Christian Studer]

• [requirements] Regenerated the requirements files. [Christian Studer]

• [requirements] Fixed requirements regarding the STIX 2 dependency. [Christian Studer]

• [stix2 export] Fixed timestamp handling when they are already datetime. [Christian Studer]

  • Happens if we give the STIX Parser a MISPEvent
    type input instead of the JSON format that is
    the standard case when used in MISP core

• [stix2 export] Fixed pattern validation to avoid sanitisation for strings to be executed on non string values. [Christian Studer]

• [stix2 export] Fixed custom objects parsing for standalone pe-section objects parsing. [Christian Studer]

Other

• Merge branch 'main' of github.com:MISP/misp-stix. [Christian Studer]

misp-stix 2.4.161 released

v2.4.161 (2022-08-23)

Changes

• [package] Updated library version. [Christian Studer]

• [package] Updated some setup information. [Christian Studer]

• [readme] Updated instructions for pip install. [Christian Studer]

• [poetry] Bumped latest lock file. [Christian Studer]

• [poetry] Added a few information and using the stix2 library package instead of the git dependency. [Christian Studer]

• [poetry] Bumped latest lock file. [Christian Studer]

• [poetry] Updated pyproject file. [Christian Studer]

Fix

• [package] Fixed setup. [Christian Studer]

• [stix2 import] Fixed a couple typo issues. [Christian Studer]

• [poetry] Bumped latest lock file. [Christian Studer]

• [stix2 import] Added missing import. [Christian Studer]

• [stix2 import] Fixed the add_attribute method that was missing the ** prefix that is required when you pass a dict directly to it. [Christian Studer]

Other

• Merge pull request #21 from netantho/patch-1. [Christian Studer]

  Add setuptools as a build-system dependency

• Add setuptools as a build-system dependency. [Anthony VEREZ]

• Wip: [stix2 import] Better handling of external references from attack-pattern objects. [Christian Studer]

  • Instead of having a common parsing function for
    all STIX 2 attack pattern external references,
    we parse those references depending on whether
    it is external STIX data or not, to have 1 very
    specific parsing function for content we know,
    and a more flexible one for external content in
    order to avoid issues with that kind of data

misp-stix 2.4.160 released

Alongside with MISP latest release, we are pleased to announce that misp-stix comes with a few improvements that are available and used on MISP.

This release also includes a few new features, changes and improvements on the library itself.

Changes immediately available on MISP

The MISP objects conversion mapping to STIX 2 has been updated to support the following templates:

• http-request
• netflow
• sigma (only STIX 2.1, which supports multiple patterning languages such as yara or suricata that are already included)

With the implementation of the conversion for these object templates, we also added the related tests and updated the documentation.

Improvements on the STIX -> MISP import feature

The STIX 2 -> MISP import feature has been substantially improved to complete the support of STIX content that is produced with this library.
Then we should now have a STIX 2 -> MISP mapping similar to the MISP -> STIX 2 one and be able to import back to MISP what has been exported as STIX 2.
(documentation will be also available soon)

(WiP) The conversion of STIX 2 content from external sources has been improved and now supports most of the SDOs.
There is nonetheless a not negligible amount of work needed to "fully" support the conversion of STIX patterns and Cyber Observable objects into the appropriate MISP data structure (Attribute, Objects, ...). Soon we will rework and improve the mapping for these STIX features so STIX -> MISP import feature can be used on MISP and replace the old built-in code 🤞

Additional features

Single MISP attributes parsing & incremental conversion

A parse_misp_attribute method has been added to handle the conversion to STIX of single MISP attributes (this feature is different from the already implemented parse_misp_attributes method that is used to convert MISP Attributes collections).

Alongside with the ability to parse single attributes independently, we improved the ability to parse MISP data incrementally and fetch the conversion results.
As a result, we can now use the main parsing functions that handle MISP data as many times as needed and store the converted STIX data in one single Bundle more easily than before.
For example:

from misp_stix_converter import MISPtoSTIX21Parser
parser21 = MISPtoSTIX21Parser()
for event in whatever_process_returning_MISP_events():
    parser.parse_misp_event(event)

The STIX objects are available then with:

parser.stix_objects # if you want to simply look the list of objects
# OR
parser.fetch_stix_objects # to extract the STIX objects you just generated from the conversion of MISP events

If you want to get those objects within a fancy STIX Bundle:

parser.bundle # extracts the STIX objects like `fetch_stix_object` and puts them in a STIX Bundle

This feature works with all the supported MISP data structures conversion (Events, Attributes, ...) and does not interfere with the collections handling features that do the same work for you in a single callable function.

This feature has been initiated from a request in #16 by @mavam

---

Changelog available here: https://github.com/MISP/misp-stix/commits/v2.4.160

misp-stix 2.4.159 released

v2.4.159 (2022-05-30)

Changes

• [poetry] Updated poetry config file & lock file to the latest. [Christian Studer]

• [tests] Changed samples used for email objects import from STIX 2 Observable objects. [Christian Studer]

• [tests] Updated tests for attributes export as STIX1 URI objects or STIX2 URL objects. [chrisr3d]

• [tests] Added more attributes types to be converted as STIX URL / URI objects. [chrisr3d]

• [stix2 import] Added a reusable function to fetch observable objects. [chrisr3d]

• [tests] Added more hash attribute types to be tested & fixed the tests for thoses attributes export as STIX 1 at the same time. [chrisr3d]

• [stix2 export] Added link attribute from the news-agency object to the list of contact information fields within the STIX 2 Identity object. [chrisr3d]

• [stix2 import] Enhanced the vulnerability object import mapping. [chrisr3d]

• Tests, documentation] Modifying the documentation to keep the shortened data values even if we use the actual files in tests. [chrisr3d]

• [tests] Using the actual attachment files to declare tests samples. [chrisr3d]

• [tests] Preparing some features to be reused with more inheritance from parent classes. [chrisr3d]

• [stix2 export] Updated the employee object export as STIX 2 mapping. [chrisr3d]

  • Now includes the recently added full-name
    object relation

• [tests] Deduplication of test code for attack-pattern object tests & for some multiple assertion statements. [chrisr3d]

• [tests] Preparing some features to be reused with more inheritance from parent classes. [chrisr3d]

• [tests] Updated tests for attack-pattern objects export as STIX 2.0 & 2.1. [chrisr3d]

• [documentation] Re-generated the full documentation with the updated mapping. [chrisr3d]

• [tests, documentation] Populating the automated documentation from attributes & objects export as STIX 2.0 tests. [chrisr3d]

• [documentation] Used the automated documentation update from tests to regenerate the objects export as STIX 2.1 mapping. [chrisr3d]

• [documentation] Used the automated documentation update from tests to regenerate the attributes export as STIX 2.1 mapping. [chrisr3d]

• [documentation] The misp objects mapping to stix21 summary is sanitized. [chrisr3d]

• [documentation] Re-generated the full documentation with the updated mapping. [chrisr3d]

• [tests, documentation] Populating the automated documentation from attributes & objects export as STIX 2.0 tests. [chrisr3d]

• [documentation] Used the automated documentation update from tests to regenerate the objects export as STIX 2.1 mapping. [chrisr3d]

• [documentation] Used the automated documentation update from tests to regenerate the attributes export as STIX 2.1 mapping. [chrisr3d]

• [documentation] The misp objects mapping to stix21 summary is sanitized. [chrisr3d]

• [stix2 import] Made some loading functions specific to each subclass. [chrisr3d]

• [stix2 import] Merged common grouping and report parsing process into on function. [chrisr3d]

  • Obviously kept separated what is different
    between groupings and reports

• [stix2 import] Better marking refs & labels parsing within Grouping & Report objects. [chrisr3d]

• [stix2 export] Only a quick and non critical change on STIX objects labels. [chrisr3d]

  • Labels generated from the conversion of a MISP
    object to a STIX 2 objects now have the label
    field matching the MISP object meta-category
    field, where the category field is specific to
    MISP attributes

• [stix2 export] Just a tiny change to prioritise the object name label. [chrisr3d]

• [tests] Better testing of observable objects ids. [chrisr3d]

• [stix2 export] Added more detail in the converted Artifact objects when they come from the conversion of malware-sample attributes. [chrisr3d]

  • Supported for both malware-sample single
    attributes and object attributes within file
    objects
  • Simply added details like the mime type, and for
    STIX 2.1, which supports additional fields
    compared to STIX 2.0, also the encryption
    algorithm and the decryption key fields

• [stix2 export] Using the github-user object parsing function as generic parsing function for other user/account objects. [chrisr3d]

  • Like we use a generic function to parse standard
    user & account objects, we now have the same
    generic function for user & account objects that
    have attachment attributes

• [stix2 export] More generic account objects parsing. [chrisr3d]

Fix

• [readme] Updated test commands. [Christian Studer]

• [stix import] Removed unused import. [Christian Studer]

• [cleanup] Some clean up and typing fixed. [Christian Studer]

• [github actions] Added recursive submodules checkout. [Christian Studer]

• [poetry] Fixed non existing dependency version. [Christian Studer]

• [poetry] Updated dependency version. [Christian Studer]

• [poetry] Added missing codecov dependency that was removed by error. [Christian Studer]

• [github actions] Typo. [Christian Studer]

• [misp-stix] Typo. [Christian Studer]

• [misp-stix] Fixed a few typos and variable name issues. [Christian Studer]

• [tests] Fixed tests for email objects import from indicator objects following the recent changes on the related mapping & parsing. [Christian Studer]

• [stix2 import] Fixed email objects mapping & parsing for indicator objects. [Christian Studer]

• [documentation] Updated mapping documentation auto-generated with the recent changes on email objects export tests. [Christian Studer]

• [tests] Fixed email objects export tests. [Christian Studer]

• [stix2 export] Fixed user-account objects export to indicator where characters were not escaped. [Christian Studer]

• [stix2 import] Added missing Observed Data object in the STIX 2.1 email samples. [Christian Studer]

• [tests] Removed print used for debugging. [Christian Studer]

• [tests] Fixed space missing to make pep8 happy. [Christian Studer]

• [tests] Added tests for the content_disposition fields within the email-message objects body_multipart. [Christian Studer]

• [stix2 export] Exporting content disposition in the body_multipart field within email-message objects while exporting email objects as indicator, to keep the object_relation field. [Christian Studer]

• [documentation] Fixed documentation auto-generation by checking the Observed Data version. [Christian Studer]

• [documentation] Regenerated documentation with the recent changes on documentation mapping. [Christian Studer]

• [documentation] Updated documentation mapping for domain-ip objects export as STIX 2 Indicators. [Christian Studer]

• [tests] Fixed tests for domain-ip objects export as STIX2 Indicators. [Christian Studer]

• [stix2 export] Fixed domain-ip objects export as Indicator to avoid confusions. [Christian Studer]

  • When domain and hostname attributes are both
    present, we want to avoid confusions between the
    domain attribute and the hostname attribute

• [stix2 import] Fixed the twitter-account object mapping. [Christian Studer]

• [tests] Added missing credential objects checking functions. [Christian Studer]

• [tests, documentation] Added the missing mapping documentation autogeneration functions. [Christian Studer]

• [misp_stix_converter] A few debugging message fixed. [Christian Studer]

• Fix: [readme] More verbose command-line usage example to please @adulau. [Christian Studer]

• [setup] Updated supported python versions. [Christian Studer]

• [poetry] Updated poetry.lock. [Christian Studer]

• [setup] Updated setup & poetry config files. [Christian Studer]

• [documentation] Regenerated documentation to include the recent updates to the documentation mapping. [Christian Studer]

• [tests] Fixed variable name typo. [chrisr3d]

• [stix2 import] Fixed twitter account object mapping. [chrisr3d]

• [documentation] The MISP objects export as STIX 2 documentation mapping has been regenerated with the recent changes on the user & account object samples. [chrisr3d]

• [documentation] The link attributes export as STIX 2 documentation has been fixed with the documentation auto-regeneration. [chrisr3d]

• [tests] Fixed tests for user & account objects export as STIX 2. [chrisr3d]

• [stix2 export] Fixed some user & account objects mapping as STIX 2. [chrisr3d]

• [stix2 import] Made pep8 more happy with some code style fixed. [chrisr3d]

• [tests] In STIX 2 samples: getting the data fields by base64-encoding the related files instead of copy-pasting the base64-encoded string. [chrisr3d]

• [stix2 import] Skipping timeline fields parsing for observed_data objects when the first_observed and last_observed values are the same as modified [chrisr3d]

• [stix2 import] Avoiding to raise the unknown STIX object exception with a test against a list of observable object types. [chrisr3d]

• [documentation] Updated attributes export as STIX 2 mapping. [chrisr3d]

• [tests] Fixed wrong category for the link attribute export. [chrisr3d]

• [tests] Just a quick function name fix. [chrisr3d]

• [tests] Removed unused variable in some MISP to STIX 1 export features tests. [chrisr3d]

• [documentation] Attributes export as STIX 2 documentation updated following the recent changes on tests. [chrisr3d]

• [stix2 export] Fixed hash attribute types mapping with the filename|telfhash type that does not exist. [chrisr3d]

• [tests] For tests using loops over attributes and stix objects, we assert the number of converted attributes first to make sure we do not loop over an empty list (which does not raise any assertion error) [chrisr3d]

• [stix2 export] Simplified the pe-section hash attributes handling with only the supported hash types, and no longer the full list of existing hash ty...

misp-stix initial release (v2.4.149)

misp-stix initial release

What's Changed

• Adds fix for 'parse_misp_attribute' object reference error by @cr-fp in #8
• Use https for submodule by @JakubOnderka in #9

New Contributors

• @cr-fp made their first contribution in #8
• @JakubOnderka made their first contribution in #9

Full Changelog: https://github.com/MISP/misp-stix/commits/v2.4.149