misp-stix 2.4.159 released

adulau released this 26 Sep 12:28

· 1901 commits to main since this release

95c141f

v2.4.159 (2022-05-30)

Changes

[poetry] Updated poetry config file & lock file to the latest. [Christian Studer]
[tests] Changed samples used for email objects import from STIX 2 Observable objects. [Christian Studer]
[tests] Updated tests for attributes export as STIX1 URI objects or STIX2 URL objects. [chrisr3d]
[tests] Added more attributes types to be converted as STIX URL / URI objects. [chrisr3d]
[stix2 import] Added a reusable function to fetch observable objects. [chrisr3d]
[tests] Added more hash attribute types to be tested & fixed the tests for thoses attributes export as STIX 1 at the same time. [chrisr3d]
[stix2 export] Added link attribute from the news-agency object to the list of contact information fields within the STIX 2 Identity object. [chrisr3d]
[stix2 import] Enhanced the vulnerability object import mapping. [chrisr3d]
Tests, documentation] Modifying the documentation to keep the shortened data values even if we use the actual files in tests. [chrisr3d]
[tests] Using the actual attachment files to declare tests samples. [chrisr3d]
[tests] Preparing some features to be reused with more inheritance from parent classes. [chrisr3d]
[stix2 export] Updated the employee object export as STIX 2 mapping. [chrisr3d]
- Now includes the recently added full-name
  object relation
[tests] Deduplication of test code for attack-pattern object tests & for some multiple assertion statements. [chrisr3d]
[tests] Preparing some features to be reused with more inheritance from parent classes. [chrisr3d]
[tests] Updated tests for attack-pattern objects export as STIX 2.0 & 2.1. [chrisr3d]
[documentation] Re-generated the full documentation with the updated mapping. [chrisr3d]
[tests, documentation] Populating the automated documentation from attributes & objects export as STIX 2.0 tests. [chrisr3d]
[documentation] Used the automated documentation update from tests to regenerate the objects export as STIX 2.1 mapping. [chrisr3d]
[documentation] Used the automated documentation update from tests to regenerate the attributes export as STIX 2.1 mapping. [chrisr3d]
[documentation] The misp objects mapping to stix21 summary is sanitized. [chrisr3d]
[documentation] Re-generated the full documentation with the updated mapping. [chrisr3d]
[tests, documentation] Populating the automated documentation from attributes & objects export as STIX 2.0 tests. [chrisr3d]
[documentation] Used the automated documentation update from tests to regenerate the objects export as STIX 2.1 mapping. [chrisr3d]
[documentation] Used the automated documentation update from tests to regenerate the attributes export as STIX 2.1 mapping. [chrisr3d]
[documentation] The misp objects mapping to stix21 summary is sanitized. [chrisr3d]
[stix2 import] Made some loading functions specific to each subclass. [chrisr3d]
[stix2 import] Merged common grouping and report parsing process into on function. [chrisr3d]
- Obviously kept separated what is different
  between groupings and reports
[stix2 import] Better marking refs & labels parsing within Grouping & Report objects. [chrisr3d]
[stix2 export] Only a quick and non critical change on STIX objects labels. [chrisr3d]
- Labels generated from the conversion of a MISP
  object to a STIX 2 objects now have the label
  field matching the MISP object meta-category
  field, where the category field is specific to
  MISP attributes
[stix2 export] Just a tiny change to prioritise the object name label. [chrisr3d]
[tests] Better testing of observable objects ids. [chrisr3d]
[stix2 export] Added more detail in the converted Artifact objects when they come from the conversion of malware-sample attributes. [chrisr3d]
- Supported for both malware-sample single
  attributes and object attributes within file
  objects
- Simply added details like the mime type, and for
  STIX 2.1, which supports additional fields
  compared to STIX 2.0, also the encryption
  algorithm and the decryption key fields
[stix2 export] Using the github-user object parsing function as generic parsing function for other user/account objects. [chrisr3d]
- Like we use a generic function to parse standard
  user & account objects, we now have the same
  generic function for user & account objects that
  have attachment attributes
[stix2 export] More generic account objects parsing. [chrisr3d]

Fix

[readme] Updated test commands. [Christian Studer]
[stix import] Removed unused import. [Christian Studer]
[cleanup] Some clean up and typing fixed. [Christian Studer]
[github actions] Added recursive submodules checkout. [Christian Studer]
[poetry] Fixed non existing dependency version. [Christian Studer]
[poetry] Updated dependency version. [Christian Studer]
[poetry] Added missing codecov dependency that was removed by error. [Christian Studer]
[github actions] Typo. [Christian Studer]
[misp-stix] Typo. [Christian Studer]
[misp-stix] Fixed a few typos and variable name issues. [Christian Studer]
[tests] Fixed tests for email objects import from indicator objects following the recent changes on the related mapping & parsing. [Christian Studer]
[stix2 import] Fixed email objects mapping & parsing for indicator objects. [Christian Studer]
[documentation] Updated mapping documentation auto-generated with the recent changes on email objects export tests. [Christian Studer]
[tests] Fixed email objects export tests. [Christian Studer]
[stix2 export] Fixed user-account objects export to indicator where characters were not escaped. [Christian Studer]
[stix2 import] Added missing Observed Data object in the STIX 2.1 email samples. [Christian Studer]
[tests] Removed print used for debugging. [Christian Studer]
[tests] Fixed space missing to make pep8 happy. [Christian Studer]
[tests] Added tests for the content_disposition fields within the email-message objects body_multipart. [Christian Studer]
[stix2 export] Exporting content disposition in the body_multipart field within email-message objects while exporting email objects as indicator, to keep the object_relation field. [Christian Studer]
[documentation] Fixed documentation auto-generation by checking the Observed Data version. [Christian Studer]
[documentation] Regenerated documentation with the recent changes on documentation mapping. [Christian Studer]
[documentation] Updated documentation mapping for domain-ip objects export as STIX 2 Indicators. [Christian Studer]
[tests] Fixed tests for domain-ip objects export as STIX2 Indicators. [Christian Studer]
[stix2 export] Fixed domain-ip objects export as Indicator to avoid confusions. [Christian Studer]
- When domain and hostname attributes are both
  present, we want to avoid confusions between the
  domain attribute and the hostname attribute
[stix2 import] Fixed the twitter-account object mapping. [Christian Studer]
[tests] Added missing credential objects checking functions. [Christian Studer]
[tests, documentation] Added the missing mapping documentation autogeneration functions. [Christian Studer]
[misp_stix_converter] A few debugging message fixed. [Christian Studer]
Fix: [readme] More verbose command-line usage example to please @adulau. [Christian Studer]
[setup] Updated supported python versions. [Christian Studer]
[poetry] Updated poetry.lock. [Christian Studer]
[setup] Updated setup & poetry config files. [Christian Studer]
[documentation] Regenerated documentation to include the recent updates to the documentation mapping. [Christian Studer]
[tests] Fixed variable name typo. [chrisr3d]
[stix2 import] Fixed twitter account object mapping. [chrisr3d]
[documentation] The MISP objects export as STIX 2 documentation mapping has been regenerated with the recent changes on the user & account object samples. [chrisr3d]
[documentation] The link attributes export as STIX 2 documentation has been fixed with the documentation auto-regeneration. [chrisr3d]
[tests] Fixed tests for user & account objects export as STIX 2. [chrisr3d]
[stix2 export] Fixed some user & account objects mapping as STIX 2. [chrisr3d]
[stix2 import] Made pep8 more happy with some code style fixed. [chrisr3d]
[tests] In STIX 2 samples: getting the data fields by base64-encoding the related files instead of copy-pasting the base64-encoded string. [chrisr3d]
[stix2 import] Skipping timeline fields parsing for observed_data objects when the first_observed and last_observed values are the same as modified [chrisr3d]
[stix2 import] Avoiding to raise the unknown STIX object exception with a test against a list of observable object types. [chrisr3d]
[documentation] Updated attributes export as STIX 2 mapping. [chrisr3d]
[tests] Fixed wrong category for the link attribute export. [chrisr3d]
[tests] Just a quick function name fix. [chrisr3d]
[tests] Removed unused variable in some MISP to STIX 1 export features tests. [chrisr3d]
[documentation] Attributes export as STIX 2 documentation updated following the recent changes on tests. [chrisr3d]
[stix2 export] Fixed hash attribute types mapping with the filename|telfhash type that does not exist. [chrisr3d]
[tests] For tests using loops over attributes and stix objects, we assert the number of converted attributes first to make sure we do not loop over an empty list (which does not raise any assertion error) [chrisr3d]
[stix2 export] Simplified the pe-section hash attributes handling with only the supported hash types, and no longer the full list of existing hash types. [chrisr3d]
[documentation] Fixed documentation with non existing attribute type removed. [chrisr3d]
[tests] Fixed hash attributes tests since filename|telfhash is not an existing MISP attribute type. [chrisr3d]
[tests] Better automation on tests for multiple single attributes export. [chrisr3d]
[stix2 export] Enhanced the list of supported hash attribute types to be exported. [chrisr3d]
[tests] Removed utility function that had already been moved in the parent class. [chrisr3d]
[documentation] Documentation regenerated. [chrisr3d]
[stix2 import] Added missing imports. [chrisr3d]
[documentation] Objects documentation mapping fixed. [chrisr3d]
[documentation] Attributes documentation mapping fixed. [chrisr3d]
[tests, documentation] Fixed automatic documentation generation from import tests. [chrisr3d]
[stix2 import] Fixed timeline fields parsing for indicator objects. [chrisr3d]
[tests] Fixed tests for suricata objects export as STIX 2.1 and added more attributes to the suricata & yara test object samples to be tested. [chrisr3d]
[stix2 export] Fixed the suricata object export as STIX 2.1 mapping. [chrisr3d]
[stix2 import] Fixed patterning language objects parsing for external STIX content. [chrisr3d]
[stix2 import] Fixed STIX 2.1 Location objects import as geolocation objects. [chrisr3d]
[tests] Fixed the geolocation object export tests following the recent changes on this object's mapping. [chrisr3d]
[stix2 export] Fixed geolocation object export mapping. [chrisr3d]
[tests] Fixed tests for news-agency objects export as STIX 2.0 & 2.1 following the changes on the contact information field for this object. [chrisr3d]
[tests] A few changes in the test function names & added unit tests for the MISP object names. [chrisr3d]
[stix2 import] Fixed the STIX 2 Vulnerability object parsing. [chrisr3d]
[tests] Fixed tests for employee objects import from STIX 2 Identity objects, following the recent changes on the contact_information field handling. [chrisr3d]
[stix2 import] Fixed the Identity object error message. [chrisr3d]
[stix2 import] Fixed contact information field handling in the STIX 2 Identity object import as MISP employee object. [chrisr3d]
[tests] Fixed documentation auto-generation from tests for user account objects. [chrisr3d]
[stix2 export] Better patterns escaping. [chrisr3d]
[tests] Better patterns escaping tests. [chrisr3d]
[tests] Fixed tests for legal-entity export as STIX 2.0 & 2.1. [chrisr3d]
[stix2 export] Fixed the legal-entity objects export as STIX 2 mapping, with the website attribute now being part of the contact information mapping for this object. [chrisr3d]
[stix2 export] Fixed employee objects export as STIX 2 mapping, with the email-address attribute being now part of the contact information mapping for this object. [chrisr3d]
[stix2 export] Added missing specific mapping list for employee objects export as STIX 2.0 & 2.1. [chrisr3d]
[stix2 export] Fixed employee object export of the contact information STIX 2 field. [chrisr3d]
[stix2 import] Fixed a variable name. [chrisr3d]
[stix2 import] Better handling of STIX objects loaded in a dict with a used flag. [chrisr3d]
[tests] Putting the AttackPattern objects checking function at the right place. [chrisr3d]
- In this case, this is a testing function for
  specific STIX 2 objects generated from MISP
[stix2 import] Avoiding any issue with the type feature in mappings. [chrisr3d]
- Making sure it is not considered as the type
  feature of a python method
- Declaring dictionaries and passing them to the
  Mapping class when needed
[tests] Enhanced course-of-action objects export tests. [chrisr3d]
[stix2 import] Added force_timestamps parameter at the creation of MISP events and objects to make sure the timestamps will be preserved once ingested in MISP format. [chrisr3d]
[stix2 export] Fixed attack-pattern export as STIX 1 tests following the recent changes on the sample objects. [chrisr3d]
[stix2 import] Removed unused imports. [chrisr3d]
[tests] Function name typo. [chrisr3d]
[tests] Fixed some tests function names. [chrisr3d]
- Wrong test function name makes the test to be
  skipped. Must start with test
[stix2 import] A few quick fixes. [chrisr3d]
[stix2 import] Clarification on the Unknown STIX object type exception handling. [chrisr3d]
[stix2 import] Added some missing loading functions (mapping + actual function) [chrisr3d]
[stix2 import] Fixed Vulnerability objects parsing. [chrisr3d]
[stix2 import] A few variable names and copy paste issues fixed. [chrisr3d]
[documentation] Making sure we don't face any path issue in case the documentation generation is ran from another path. [chrisr3d]
[documentation] Updated summary. [chrisr3d]
[documentation, tests] Some typos which generated a broken documentation update. [chrisr3d]
[tests] Just a quick summary update. [chrisr3d]
[tests] A few copy paste and variable name issues. [chrisr3d]
[tests] Reusing declared variables. [chrisr3d]
[tests] Removed or used unused variables. [chrisr3d]
[tests] Reusing existing variable. [chrisr3d]
[tests] Fixed undefined variable name. [chrisr3d]
[documentation, tests] Sanitized the automated documentation generation from the tests. [chrisr3d]
[documentation, tests] Stripped data fields values to make them more convenient to be used in a documentation. [chrisr3d]
[documentation, tests] Forcing some summary definition in the objects documentation. [chrisr3d]
[tests] Better variables handling in some attributes export tests. [chrisr3d]
[tests] Fixed variable name. [chrisr3d]
[documentation, tests] Fixed the mac-address Observed Data documentation automation. [chrisr3d]
[tests] Removed test print. [chrisr3d]
[stix2 export] Fixed the suricata object mapping. [chrisr3d]
[stix2 export] Using the parent class property to get the identity_id since the "private" attribute is not known by the children classes. [chrisr3d]
[git] Fixed gitmodules file. [chrisr3d]
[tests] Quick grouping features testing simplification. [chrisr3d]
[stix2 export] Fixed cti library path following the recent path changes for this git submodule. [chrisr3d]
[stix2 export] Simplified one tmp variable that was not necessary. [chrisr3d]
[stix2 export] Fixed typo with Sighting fields. [chrisr3d]
[documentation] Making sure we don't face any path issue in case the documentation generation is ran from another path. [chrisr3d]
[documentation] Updated summary. [chrisr3d]
[documentation, tests] Some typos which generated a broken documentation update. [chrisr3d]
[tests] Just a quick summary update. [chrisr3d]
[tests] A few copy paste and variable name issues. [chrisr3d]
[tests] Reusing declared variables. [chrisr3d]
[tests] Removed or used unused variables. [chrisr3d]
[tests] Reusing existing variable. [chrisr3d]
[tests] Fixed undefined variable name. [chrisr3d]
[documentation, tests] Sanitized the automated documentation generation from the tests. [chrisr3d]
[documentation, tests] Stripped data fields values to make them more convenient to be used in a documentation. [chrisr3d]
[documentation, tests] Forcing some summary definition in the objects documentation. [chrisr3d]
[tests] Better variables handling in some attributes export tests. [chrisr3d]
[tests] Fixed variable name. [chrisr3d]
[documentation, tests] Fixed the mac-address Observed Data documentation automation. [chrisr3d]
[tests] Removed test print. [chrisr3d]
[stix2 export] Fixed the suricata object mapping. [chrisr3d]
[stix2 export] Using the parent class property to get the identity_id since the "private" attribute is not known by the children classes. [chrisr3d]
[stix2 import] A few changes on the single_event parameter and the number of report or grouping objects. [chrisr3d]
[git] Fixed gitmodules file. [chrisr3d]
[tests] Quick grouping features testing simplification. [chrisr3d]
[stix2 export] Fixed cti library path following the recent path changes for this git submodule. [chrisr3d]
[stix2 export] Fixed typo with Sighting fields. [chrisr3d]
[stix2 import] Clarification on various mapping variable names. [chrisr3d]
- Making sure we know whether we deal with an
  attribute or object mapping
- Making sure we differenciate MISP features and
  STIX objects mapping
[stix2 import] Added missing Location object import. [chrisr3d]
[stix2 import] Changed the pattern type exception catching to an error instead of a warning since we cannot call the stix2-pattern object creation function in this case. [chrisr3d]
[stix2 import] Typo. [chrisr3d]
[stix2 export] Simplified one tmp variable that was not necessary. [chrisr3d]
[stix2 import] Quick fix on vulnerability object parameter that is a ref and not the vulnerability object directly. [chrisr3d]
[stix2 import] Making the MISP object creation function an attribute of the parent class, available for both children classes. [chrisr3d]
[stix2 import] A few errors fixed, like a missing import or a wrong variable name etc. [chrisr3d]
[stix2 import] Made the list of unsupported pattern separation key words a property of the external STIX files parsing mapping. [chrisr3d]
[stix2 import] This typing variable is now going to be needed in the parent class. [chrisr3d]
[stix2 import] Better separation in catching exceptions while looping over report or grouping object_refs. [chrisr3d]
[stix2 import] Fixed a few variable names issues. [chrisr3d]
[stix2 import] Fixed function name change that was missing. [chrisr3d]
[stix1 export] Better errors handling for objects to parse as the same improvement has been made to STIX2 recently. [chrisr3d]
[stix1 export] Better errors handling for objects to parse as the same improvement has been made to STIX2 recently. [chrisr3d]
[stix export] Enhanced handling of MISP object which encountered a parsing issue. [chrisr3d]
- Avoiding those objects to be skipped
- They're exported as custom objects instead
[stix2 export] Enhanced the pattern values sanitisation. [chrisr3d]
- Generalised the sanitisation made on registry
  key values to all the pattern since they may
  contain characted like % and \ which are
  particularly tricky to handle in STIX patterns
[stix2 export] Enhanced the pattern values sanitisation. [chrisr3d]
- Generalised the sanitisation made on registry
  key values to all the pattern since they may
  contain characted like % and \ which are
  particularly tricky to handle in STIX patterns
[stix2 export] Better exceptions catching while handling MISP objects to parse. [chrisr3d]
- Most of the objects are parsed on the go and
  directly converted into a STIX object, but some
  objects have specific relations that require
  special care. It is the case for file objects
  with pe and pe-section objects. Since they are
  exported into a single STIX file object with an
  extension, we need to store them until we are
  sure all MISP objects have been handled (parsed
  or stored) and we do have all the referenced
  objects to start the special parsing. Then they
  are parsed together using the ObjectReference
  field of each one of them. For this specific use
  case, we were missing some exception catching
  since they're out of the standard objects
  resolving loop
[tests] Making sure the recent changes on STIX objects labels don't break the tests. [chrisr3d]
[stix2 import] Updated the stix2_to_misp helper function. [chrisr3d]
- We already wrote previously a skeleton for this
  function to take a filename using its name and
  to call the parsing function which takes the
  STIX2 bundle object. We simply updated it with
  the recent STIX2 to MISP parsing features
  development
[stix2 import] Variable names typo. [chrisr3d]
[stix2 import] Wrong variable name. [chrisr3d]
[tests] Fixed tests on labels. [chrisr3d]
[stix2 export] Better markings handling to avoid issues with unrecognised tlp tags. [chrisr3d]
[stix2 import] Syntax fixed. [chrisr3d]
[stix2 export] Better markings handling to avoid issues with unrecognised tlp tags. [chrisr3d]
[stix1 export] Transforming into upper case TLP tags only. [chrisr3d]
- TLP tags that are not parsed as TLPMarkings are
  then exported as SimpleMarking with no uppercase
  conversion, which keeps the tag as is
- It also avoids the .upper() for every test ran
  on each tag, and limits this conversion into
  uppercase only when needed
[stix1 export] Transforming into upper case TLP tags only. [chrisr3d]
- TLP tags that are not parsed as TLPMarkings are
  then exported as SimpleMarking with no uppercase
  conversion, which keeps the tag as is
- It also avoids the .upper() for every test ran
  on each tag, and limits this conversion into
  uppercase only when needed
[stix1 export] Fixed tags parsing to avoid issues with TLP tags. [chrisr3d]
- Parsing as TLPMarking only the supported TLP tags
- The other ones are exported as SimpleMarkings
[stix1 export] Fixed tags parsing to avoid issues with TLP tags. [chrisr3d]
- Parsing as TLPMarking only the supported TLP tags
- The other ones are exported as SimpleMarkings
[tests] Fixed orgname testing in every different test. [chrisr3d]
- The orgname value used to define the information
  source and reporter identity remains the same
- The orgname value used to define every STIX
  object id is correctly sanitized
[stix1 export] Fixed missing import and typo. [chrisr3d]
[stix1 export] Fixed STIX objects ID identifier. [chrisr3d]
- Making sure the orgname used is sanitised and
  does not contain any space
[stix1 framing] Fixed STIX 1 XML Header framing. [chrisr3d]
[stix2 export] Making sure observable object ids are correctly parsed. [chrisr3d]
- Making also sure those ids are correctly
  fetched if there are event reports, so they are
  correctly referenced in the object_refs field
[stix2 export] Better handling of object ids used in the object_refs field within the Note objects generated from the event reports parsing. [chrisr3d]
[stix2 export] Fixed lnk object parsing. [chrisr3d]
- The uuid fields list was missing the
  malware-sample attribute
- Differenciation between the uuid fields and the
  path fields
  - uuid fields are the attributes that are
    exported in a different observable object than
    the main one resulting from the conversion of
    most of the object attributes
  - path fields are the attributes that are
    exported as directory objects and referenced
    by the main file object with the
    directory_ref field
[stix2 export] Making parent-pid attribute prioritary over parent-command-line to define which attribute uuid is used to define the parent process id while parsing process objects. [chrisr3d]
[tests] Fixed tests for legal-entity objects export. [chrisr3d]
- Added the attribute that was missing, following
  the recent fix on this object mapping
[stix2 export] Fixed legal-entity object mapping. [chrisr3d]
[stix2 export] Making sure we want the uuid of an object attribute before actually getting it. [chrisr3d]
[stix2 export] Fixed image object export, especially as STIX 2.1 which was missing some attribute uuids. [chrisr3d]
[stix2 export] Quick change on file observable objects parsing to prepare future updates on event reports handling. [chrisr3d]
[stix2 export] Fixed email object attributes parsing. [chrisr3d]
- In the parent STIX 2 parsing class, we cannot
  hardcode object_relation fields that are only
  supported in either STIX 2.0 or STIX 2.1.
  In this case, the message-id attribute is only
  supported in STIX 2.1, and we reach a KeyError
  exception if we try to get the STIX 2.0 mapping
  for this object_relation in STIX 2.0
[stix2 export] Fixed message-id attribute from email object export as STIX 2.1. [chrisr3d]
[stix2 export] Better domain|ip objects parsing to make sure the DomainName objects have the correct id field. [chrisr3d]
[tests] Removed empty line. [chrisr3d]
[stix2 export] Fixed lnk object mapping. [chrisr3d]
- Removed the unsupported fields in the main class
  mapping since they are specific to STIX 2.1 only
- Removed the duplicated mappings that are no
  longer needed in the subclasses since the
  mapping is single and the specific fields are
  handled in another mapping structure
[stix export] Removed unused imports. [chrisr3d]
[stix2 export] Removed unused import. [chrisr3d]
[stix2 export] Quick typo & empty line issues fixed. [chrisr3d]
[tests] Added missing legal-entity test object that is necessary for the related tests. [chrisr3d]
[tests] Fixed tests for malware-sample attributes & object attributes tests following the recent updates on the conversion of this type of attribute. [chrisr3d]
[stix2 export] Added missing created_by_ref field in Note & Location objects. [chrisr3d]
[stix2 export] Fixed copy paste issue in variable name. [chrisr3d]
[tests] Added missing cpe-asset metadata values. [chrisr3d]
[stix2 export] Better handling of custom features with potential data field in STIX objects or Observable objects. [chrisr3d]
[tests] Testing the location object id with the grouping refs. [chrisr3d]
[tests] Fixed tests for objects which recently got there STIX conversion to contain a to_ids tag. [chrisr3d]
[stix2 export] Added the global to_ids tag fetched from object attributes even in STIX objects that are not dependant from this tag. [chrisr3d]
- As opposed to Indicator & Observable objects
  which are directly depending on the to_ids
  value, other objects were not getting the value
  as additional tag value. As it does not cost
  much more to at least get the info whether there
  was a to_ids flag in the object attributes, we
  add this tag in some objects that were missing it
[tests] Testing precisely the observable ids within observable compositions while exporting MISP into STIX 1. [chrisr3d]
[tests] Changed ids of observable objects within observable composition objects to comply with the recent changes on observable ids in that specific case. [chrisr3d]
[tests] Properly testing the observable features in the case of an export of a domain|ip attribute. [chrisr3d]
- Compared to before, when the observable object
  id was set with the domain|ip attribute uuid, we
  replaced it with a v5 uuid defined with the
  attribute uuid, and the corresponding value. We
  now test the resulting observable ids based on
  these v5 uuids

Other

Fix; [github actions] Added missing pytest dependency for github actions. [Christian Studer]
Add: [github actions] Added workflow. [Christian Studer]
Wip: [tests] Tests for email objects import from STIX 2 Observable objects. [Christian Studer]
Fix; [stix2 export] Better email objects export handling. [Christian Studer]
- Enhanced parsing of email addresses and the
  related display names for both indicator and
  observable objects
- Better definition of the email-message refs
  within the pattern
Merge branch 'main' of github.com:MISP/misp-stix. [Christian Studer]
Wip: [stix2 import] Importing email objects from STIX 2 Observable objects. [Christian Studer]
Wip: [tests] Tests for email objects import from Indicators. [Christian Studer]
Wip: [stix2 import] Importing email objects from Indicators. [Christian Studer]
- Observable parsing in progress
- Improvement & fixes might also come for both
  email objects export and then import (as a
  consequence to support the same mapping in both
  directions)
Wip: [tests] Added indicator & observable samples to be imported as email objects. [Christian Studer]
Wip: [tests] Tests for domain-ip import from STIX 2 Indicator & Observable objects. [Christian Studer]
Wip: [stix2 import] Importing domain-ip objects from STIX 2 Indicator & Observable objects. [Christian Studer]
Wip: [tests] Added tests for user-account objects import from STIX 2 Indicator & Observable objects. [Christian Studer]
Wip: [stix2 import] Importing user-account objects from STIX 2 Indicator & Observable objects. [Christian Studer]
Wip: [tests] Added tests for credential objects import from STIX 2 Indicator & Observable objects. [Christian Studer]
Wip: [stix2 import] Importing credential objects from STIX 2 Indicator & Observable objects. [Christian Studer]
Add: [readme] Added Usage examples for the command-line usage. [Christian Studer]
Add: [setup] Made the python library executable. [Christian Studer]
- Supported now: Export only
- Reusing helpers that were already available if
  the library is imported in a python script
Wip: [tests] Tests for user & account objects with attachments import from STIX 2 Indicator & Observable objects. [chrisr3d]
Wip: [stix2 import] Importing user & account objects which can contain attachments from STIX 2 Indicator & Observable objects. [chrisr3d]
Wip: [tests] Tests for user & account objects import from STIX 2 Indicator & Observable objects. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix. [chrisr3d]
Wip: [stix2 import] Importing user & account objects from STIX 2 Indicator & Observable objects. [chrisr3d]

-> User & account objects that have no attachement
attribute with a data field
Wip: [tests] Fixed STIX 2 samples for import tests, following the recent fixes on user & account objects mapping. [chrisr3d]
Wip: [stix2 import] Changed user account objects import parsing mapping. [chrisr3d]
Wip: [tests] Added samples for user account objects import. [chrisr3d]
Wip: [tests] Tests for cpe-asset objects import from STIX 2 Indicator & Observable objects. [chrisr3d]
Wip: [stix2 import] Importing cpe-asset objects from STIX 2 Indicator & Observables objects. [chrisr3d]
Wip: [tests] Tests for asn objects import from STIX 2 Indicator & Observable objects. [chrisr3d]
Wip: [stix2 import] Started importing MISP objects from Indicator & Observable objects with the asn object. [chrisr3d]
Wip: [tests] Tests for the recently added attribute types import from STIX 2. [chrisr3d]
Wip: [stix2 import] Completing the attributes import mapping with the missing attribute types. [chrisr3d]
- All the attribute types that are supported in
  the MISP -> STIX 2 export mapping should now be
  supported in the STIX 2 -> MISP import mapping
Wip: [tests] Tests for filename attributes import from STIX 2 Indicator & Observable objects. [chrisr3d]
Wip: [stix2 import] Importing filename attributes from STIX 2 Indicator & Observable objects. [chrisr3d]
Add: [tests, documentation] Some STIX 2 import documentation generated from the tests. [chrisr3d]
Wip: [tests] Tests for email attributes import from STIX 2 & split internal STIX 2 sub-classes. [chrisr3d]
- Separating STIX 2.0 & STIX 2.1 testing classes
  to avoid mixing up with the documentation
  variables that are not reset to empty when the
  tests from 2 different unittest classes are
  declared in the same file
Wip: [stix2 import] Importing email attributes and better attributes mapping. [chrisr3d]
- Split indicator & obsevrable mappings to be able
  to regroup specific parsing functions that are
  the same
Wip: [tests] Tests for URL Indicator & Observable objects import as MISP attributes. [chrisr3d]
Wip: [stix2 import] Importing URL Indicator & Observable objects to attributes. [chrisr3d]
Wip: [tests] Tests for the attributes import from Indicator & Observable objects we just added. [chrisr3d]
Wip; [stix2 import] Added more attributes parsing from Indicator & Observable objects. [chrisr3d]
- Adding step by step functions that are already
  (or not) in the STIX 2 to MISP mapping
Wip: [tests] Tests for x509 fingerprint attributes import from STIX 2 Indicator & Observable objects. [chrisr3d]
Wip: [stix2 import] Importing x509 fingerprint attributes from STIX 2 Indicator & Observable objects. [chrisr3d]
Wip: [tests] Tests for ip & ip|port attributes import from STIX 2 Indicator & Observable objects. [chrisr3d]
Wip: [stix2 import] Importing ip & ip|port attributes from STIX 2 Indicator & Observable objects. [chrisr3d]
Wip: [tests] Tests for hash attributes import from STIX 2.0 & 2.1 Observable & Indicator objects. [chrisr3d]
Wip: [tests] Added test samples for hash attributes import from Observable and Indicator objects. [chrisr3d]
Wip: [stix2 import] Added the missing hash attribute types to the STIX 2 to MISP mapping. [chrisr3d]
Add: [documentation] Hash attribute types recently added in the test samples have their documentation auto-generated also. [chrisr3d]
Merge branch 'dev' of github.com:MISP/misp-stix into main. [chrisr3d]
Wip: [tests] Tests for patterning language attributes & objects export from STIX 2.1 Indicator objects. [chrisr3d]
Wip: [stix2 import] Importing patterning language attributes & objects from STIX 2.1 Indicator objects. [chrisr3d]
Wip: [tests] Tests for geolocation objects import from STIX 2.1 Location objects. [chrisr3d]
Wip: [tests] Tests for script objects import from STIX 2 Malware & Tool objects. [chrisr3d]
Wip: [stix2 import] Importing script objects from STIX 2 Malware & Tool objects. [chrisr3d]
Wip: [tests] Tests for campaign-name attributes import from STIX 2 Campaign objects. [chrisr3d]
Wip: [stix2 import] Importing campaign-name attributes from STIX 2 Campaign objects. [chrisr3d]
Wip: [tests] Tests for news-agency & organization objects import from STIX 2 Identity objects. [chrisr3d]
Wip: [stix2 import] Importingnews-agency & organization object from STIX 2 Identity object re-using the Identity object parsing function. [chrisr3d]
Wip: [tests] Tests for vulnerability attributes & objects import from STIX 2 Vulnerability objects. [chrisr3d]
Wip: [tests] Tests for legal-entity objects import from STIX 2 Identity objects. [chrisr3d]
Wip: [stix2 import] Importing legal-entity objects from STIX 2 Identity objects. [chrisr3d]
Fix; [tests] Fixed tests for the employee objects export as STIX 2.0 & 2.1. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Wip: [tests] Tests for employee objects import from STIX 2.0 & 2.1 Identity objects. [chrisr3d]
Wip: [stix2 import] Importing employee objects previously exported as STIX 2 Identity objects. [chrisr3d]
Wip: [tests] Tests for CourseOfAction STIX 2 objects import. [chrisr3d]
Wip: [stix2 import] Importing CourseOfAction STIX 2 objects. [chrisr3d]
Wip: [tests] Added testing classes for STIX 2 import, starting with attack-pattern objects. [chrisr3d]
Wip: [tests] Already made some test features available in parent classes that will be reachable for import tests. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Add: [tests] Added some attack-pattern object attributes to be exported as STIX custom fields in the Attack Pattern object. [chrisr3d]
Wip: [stix2 import] Parsing STIX 2.0 & 2.1 Attack Pattern objects. [chrisr3d]
Wip: [stix2 import] Updated the STIX 2 objects mapping handling. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Add: [documentation] MISP objects export as STIX 2.0 & 2.1 mappings are automatically updated with the recent changes on tests. [chrisr3d]
Add: [tests] Added tests for script objects export as STIX 2.0 & 2.1. [chrisr3d]
Add: [stix2 export] Added script objects to the export as STIX 2.0 & 2.1 export mapping. [chrisr3d]
Wip: [documentation] Updated documentation has been regenerated. [chrisr3d]
Wip: [documentation] Replaced the attributes & objects export as STIX 2.0 & 2.1 summaries with the formatting headers so they are generated from the recently added summary mappings. [chrisr3d]
Wip: [documentation] Added the auto generation of the attributes & objects export as STIX 2.0 & 2.1 mapping summary. [chrisr3d]
Add: [documentation] Added the attributes & objects export as STIX 2.0 summary autogenerated with tests. [chrisr3d]
Wip: [documentation] Updated the MISP objects export as STIX 2.0 documentation using the documentation automated update from tests. [chrisr3d]
Wip: [documentation] Updated the attributes export to STIX 2.0 documentation regenerated with the tests automated documentation update. [chrisr3d]
Wip: [documentation, tests] Updated the automated documentation generation to support STIX 2.0. [chrisr3d]
Fix; [tests] Removed or used unused variables. [chrisr3d]
Iadd: [documentation] Added summary mapping for attributes & objects export as STIX 2.1. [chrisr3d]
Wip: [documentation, tests] Populating the objects documentation while running STIX 2.1 tests. [chrisr3d]
Wip: [documentation, tests] Outsourced the documentation update process to an external class and script. [chrisr3d]
Wip: [documentation, tests] Testing if the attributes conversion as STIX 2.1 mapping from documentation if different from the mapping built from tests before replacing it. [chrisr3d]
Wip: [documentation, tests] Replacing attribute to STIX 2.1 mapping with the samples used in tests. [chrisr3d]
Wip: [tests] Initiated an automated way to check if the mapping documentation is up-to-date using the tests. [chrisr3d]
- Started with the tests for attributes export as STIX 2.1
Add: [tests] Added tests for patterning language objects export as STIX 2.1. [chrisr3d]
Add: [tests] Test samples for objects converted into indicator with a specific pattern type. [chrisr3d]
Add: [stix2 export] Added suricata & yara to the list of supported MISP object templates for export as STIX 2.1. [chrisr3d]
Add: [submodules] Sub-moduled misp-galaxy. [chrisr3d]
Add: [git] Added tmp dir & a gitignore file that contains the tmp dir for now. [chrisr3d]
Add: [documentation] MISP objects export as STIX 2.0 & 2.1 mappings are automatically updated with the recent changes on tests. [chrisr3d]
Add: [tests] Added tests for script objects export as STIX 2.0 & 2.1. [chrisr3d]
Add: [stix2 export] Added script objects to the export as STIX 2.0 & 2.1 export mapping. [chrisr3d]
Wip: [documentation] Updated documentation has been regenerated. [chrisr3d]
Wip: [documentation] Replaced the attributes & objects export as STIX 2.0 & 2.1 summaries with the formatting headers so they are generated from the recently added summary mappings. [chrisr3d]
Wip: [documentation] Added the auto generation of the attributes & objects export as STIX 2.0 & 2.1 mapping summary. [chrisr3d]
Add: [documentation] Added the attributes & objects export as STIX 2.0 summary autogenerated with tests. [chrisr3d]
Wip: [documentation] Updated the MISP objects export as STIX 2.0 documentation using the documentation automated update from tests. [chrisr3d]
Wip: [documentation] Updated the attributes export to STIX 2.0 documentation regenerated with the tests automated documentation update. [chrisr3d]
Wip: [documentation, tests] Updated the automated documentation generation to support STIX 2.0. [chrisr3d]
Fix; [tests] Removed or used unused variables. [chrisr3d]
Iadd: [documentation] Added summary mapping for attributes & objects export as STIX 2.1. [chrisr3d]
Wip: [documentation, tests] Populating the objects documentation while running STIX 2.1 tests. [chrisr3d]
Wip: [documentation, tests] Outsourced the documentation update process to an external class and script. [chrisr3d]
Wip: [documentation, tests] Testing if the attributes conversion as STIX 2.1 mapping from documentation if different from the mapping built from tests before replacing it. [chrisr3d]
Wip: [documentation, tests] Replacing attribute to STIX 2.1 mapping with the samples used in tests. [chrisr3d]
Wip: [tests] Initiated an automated way to check if the mapping documentation is up-to-date using the tests. [chrisr3d]
- Started with the tests for attributes export as STIX 2.1
Add: [tests] Added tests for patterning language objects export as STIX 2.1. [chrisr3d]
Add: [tests] Test samples for objects converted into indicator with a specific pattern type. [chrisr3d]
Add: [stix2 export] Added suricata & yara to the list of supported MISP object templates for export as STIX 2.1. [chrisr3d]
Wip: [stix2 import] Enhanced complex patterns exclusion. [chrisr3d]
Wip: [stix2 import] Function to handle the import case for various STIX objects to convert: either as MISP attribute or MISP object. [chrisr3d]
Wip: [stix2 import] Parsing external STIX patterns that are not stix patterns. [chrisr3d]
Wip: [stix2 import] Added STIX 2.1 pattern types parsing for internal indicators with a pattern type that is not stix. [chrisr3d]
Wip; [stix2 import] Parsing Location objects. [chrisr3d]
Wip: [stix2 import] Parsing external STIX 2 Vulnerability objects. [chrisr3d]
Wip: [stix2 import] Parsing MISP generated STIX 2 Vulnerability objects. [chrisr3d]
Wip: [stix2 import] Handling the synonyms to tag names mapping. [chrisr3d]
- Synonyms are the different names of threat actors,
  courses of action, attack patterns and other
  STIX objects converted as MISP Galaxy clusters
- In order to avoid looping over galaxy clusters,
  and to avoid parsing multiple times the same
  galaxy cluster, we load this mapping once to
  provide the association of all the known galaxy
  cluster names and the related tag names
Merge branch 'dev' of github.com:MISP/misp-stix into dev. [chrisr3d]
Add: [submodules] Sub-moduled misp-galaxy. [chrisr3d]
Add: [git] Added tmp dir & a gitignore file that contains the tmp dir for now. [chrisr3d]
Wip: [stix2 import] Better pattern type handling & redirection to the stix2-pattern object creation in case of parsing exception. [chrisr3d]
Wip: [stix2 import] Some pieces of documentation for the main parsing function used for external STIX 2. [chrisr3d]
Wip: [stix2 import] Considering the possibility some producers of STIX data still use the deprecated objects field instead of object_refs [chrisr3d]
Wip: [stix2 import] Added a first version of observable & pattern mappings for STIX objects from external STIX files. [chrisr3d]
Wip: [stix2 import] Added missing Exceptions. [chrisr3d]
Wip: [stix2 import] More observable mapping skeleton. [chrisr3d]
Wip: [stix2 import] Skeleton for external STIX files parsing. [chrisr3d]
Wip: [stix2 import] Added a few pattern parsing functions to initiate the concept. [chrisr3d]
Wip: [stix2 import] More logical observable mapping functions. [chrisr3d]
Wip: [stix2 import] Added indicators parsing & better exceptions catching for observed data and indicator objects. [chrisr3d]
Wip: [stix2 import] Parsing STIX objects timeline fields. [chrisr3d]
Merge branch 'dev' of github.com:MISP/misp-stix into dev. [chrisr3d]
Wip: [stix2 import] Better was to fetch STIX object to be parsed, once they are all loaded. [chrisr3d]
Wip: [stix2 import] Better separation between objects loading & parsing. [chrisr3d]
Wip: [stix2 import] More steps for single reports parsing. [chrisr3d]
Merge branch 'dev' of github.com:MISP/misp-stix into dev. [chrisr3d]
Wip: [stix2 import] A few steps forward to the stix objects parsing from bundle. [chrisr3d]
Wip: [stix2 import] Starting with some observable objects parsing functions. [chrisr3d]
Wip: [stix2 import] STIX2 observable objects mapping for STIX content from MISP. [chrisr3d]
Wip: [stix2 import] Added some observable parsing processing. [chrisr3d]
- We'll continue with the observable mapping and
  the different related functions needed to get
  convert the observable objects into MISP
  attributes or objects
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Wip: [stix2 import] Populating STIX2 parsing functions. [chrisr3d]
- Started with the Custom objects which are the
  most straight forward ones :)
Merge branch 'dev' of github.com:MISP/misp-stix into dev. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Wip: [stix2 import] Adding library imports & changes concerning the STIX2 import features. [chrisr3d]
Wip: [stix2 import] We continue building the stix2 import skeleton. [chrisr3d]
Wip: [stix2 import] Main STIX2 objects parsing functions mapping. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Set theme jekyll-theme-cayman. [Alexandre Dulaunoy]
Set theme jekyll-theme-cayman. [Alexandre Dulaunoy]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Merge branch 'dev' of github.com:MISP/misp-stix into dev. [chrisr3d]
Wip: [stix2 import] We start the STIX2 import. [chrisr3d]
- From pseudo-code draft & ideas in mind
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Fix; [tests] Testing STIX 2.1 objects ids correctly. [chrisr3d]
- Some needed attribute uuids added
- We added several tests for the ids of different
  objects as well as observable objects
Wip: [stix import] First skeletton premise of the STIX to MISP import feature. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Add: [tests] Tests for android-app objects export as STIX 2.0 & 2.1. [chrisr3d]
Add: [stix2 export] Added android-app object to the list of supported object templates export as STIX 2.0 & 2.1. [chrisr3d]
Merge branch 'dev' of github.com:MISP/misp-stix into dev. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Merge branch 'main' of github.com:MISP/misp-stix into dev. [chrisr3d]
Add: [tests] Tests for lnk objects export as STIX 2.0 & 2.1. [chrisr3d]
Add: [stix2 export] Added lnk objects to the list of mapped object templates export as STIX 2.0 & 2.1. [chrisr3d]
Add: [tests] Tests for image objects export as STIX 2.0 & 2.1. [chrisr3d]
Add: [stix2 export] Added image objects to the list of supported object templates export as STIX 2.0 & 2.1. [chrisr3d]
Add: [tests] Added tests for legal-entity objects export as STIX 2.0 & 2.1. [chrisr3d]
Add: [stix2 export] Added legal-entity objects in the list of supported object templates export as STIX 2.0 & 2.1. [chrisr3d]
Add: [tests] Tests for news-agency & organization objects export as STIX 2.0 & 2.1. [chrisr3d]
Add: [stix2 export] Added news-agency & organization objects to the list of supported object templates export as STIX 2.0 & 2.1. [chrisr3d]
Add: [tests] Added missing test the identity_class field within an Identity STIX object exported from an employee MISP object. [chrisr3d]
Add: [tests] Added tests for employee objects export as STIX 2.0 & 2.1. [chrisr3d]
Add: [stix2 export] Added employee objects to the list of supported objects export as STIX 2.0 & 2.1. [chrisr3d]
Add: [tests] Added tests for the parler-account & reddit-account objects. [chrisr3d]
- To be tested with the github-user object
  using the account objects with attachment
  attributes parsing function
Add: [stix2 export] Added parler-account & reddit-account to the list of supported objects export as STIX 2.0 & 2.1. [chrisr3d]
- Rusing the account objects with at least one
  potential attachment attribute parsing function
  that has been made generic and that already
  supports github-user objects
Add: [tests] Added tests for telegram-account objects export as STIX 2.0 & 2.1 to the existing tests for account objects. [chrisr3d]
Add: [stix2 export] Added telegram-account objects to the list of supported objects export as STIX 2.0 & 2.1. [chrisr3d]
- Reusing the account objects parsing function
Add: [tests] Tests for cpe-asset objects export as STIX 2.0 & 2.1. [chrisr3d]
Add: [stix2 export] Added cpe-asset to the list of mapped object templates export as STIX 2.0 & 2.1. [chrisr3d]
Add: [tests] Added test for annotation objects export as STIX 2.1. [chrisr3d]
Add: [stix2 export] Added annotation objects to the list of supported object export as STIX 2.1. [chrisr3d]
- Annotation objects are exported as STIX 2.1 Note
  objects which appeared only in 2.1
- The process of parsing those objects is pretty
  similar to the pe & pe-section objects parsing,
  we need to parse first all the attributes and
  objects referenced by the annotation in order to
  get then their exact STIX object id once they
  are already converted, otherwise we would have
  the referenced_uuid value only and we would
  miss the STIX object type to build the object_ref
  id value: {type}--{uuid}
Add: [tests] Added tests for github-user objects export as STIX 2.0 & 2.1. [chrisr3d]
Add: [stix2 export] Added github-user to the list of supported objects export as STIX 2.0 & 2.1. [chrisr3d]
- As gitlab-user is already supported, there was
  no reason to skip this template, but it required
  some additional attention since there is an
  attribute with a potential data field
Add: [tests] Added tests for gitlab-user objects export as STIX 2.0 & 2.1. [chrisr3d]
Add: [tests] Added tests for github-username attributes export as STIX 2.0 & 2.1. [chrisr3d]
Add: [stix2 export] Added github-username attribute type to the list of supported types exported as STIX 2.0 & 2.1. [chrisr3d]
- As a side note: this attribute export as STIX 2.0
  observed data object is not supported due to the
  user_id field requirement that is effective
  in STIX 2.0, which is no longer the case in 2.1
  where it is optional
Add: [stix2 export] Added gitlab-user object template to the list of supported objects export as STIX 2.0 & 2.1. [chrisr3d]
- Using the most recent changes on the account
  objects parsing that made the function also
  available for this object template (in addition
  to the account objects already supported)
Add: [tests] Added tests for sigma, snort & yara attributes export as STIX 2.1. [chrisr3d]
Add: [stix2 export] Exporting sigma, snort & yara attributes in STIX 2.1 since Indicators support multiple pattern types in STIX 2.1. [chrisr3d]

Contributors

adulau

Assets 2