SO Elastic Integration w/ OpenCTI for enrichment.

[Slipperyclock]

Hello SO community, I am looking to get some help or at least pointed in the right direction to get this moving. I have been using SO since ...wow 16.04, GEEZ we came a long way. Good work you guys. I digress, I am looking to setup threat intel enrichment with either MISP or OpenCTI intel. Right now I have an OpenCTI server up, my SO instance is ingesting the data and it shows up in a number of locations. After reading and watching some videos, I am a bit confused on the difference and coordination between a .ds and logs-. First all, I have read https://glue.ghost.io/leveraging-threat-intel-for-event-enrichment-in-security-onion/, #12665, https://www.elastic.co/guide/en/integrations/current/ti_opencti.html and watched https://www.youtube.com/watch?v=4TwstDJBiVw&list=PLljFlTO9rB168zCHTwjrxUJPvc-o8RtwO [Enrich Your Data and Your Life SO 2023]. After all of that I have come up with the following.

Setup: SO 2.4.120 Distributed.

1. Create the enrichment Policy, 2x one for IP and other for Domain/hostname.

• This execution works as expected and returns successful.
• I did notice a discrepancy where the video which is newer than the article from Wes, uses .ds, where as Wes uses the so-threatintel-* index.

PUT /_enrich/policy/opencti-ip-enrichment-policy
{
 "match":{
 "indices":".ds-logs-ti_opencti.indicator-default-*",
 "match_field":"threat.indicator.ip",
 "enrich_fields":[
  "threat.indicator.provider",
  "threat.indicator.type",
  "threat.indicator.confidence",
  "opencti.indicator.score",
  "tags"
  ]
 }
}

PUT /_enrich/policy/opencti-domain-enrichment-policy
{
 "match":{
 "indices":".ds-logs-ti_opencti.indicator-default-*",
 "match_field":"threat.indicator.url.domain",
 "enrich_fields":[
  "threat.indicator.provider",
  "threat.indicator.type",
  "threat.indicator.confidence",
  "opencti.indicator.score",
  "tags"
  ]
 }
}

2. Execute Policy

POST /_enrich/policy/opencti-ip-enrichment-policy/_execute
POST /_enrich/policy/opencti-domain-enrichment-policy/_execute

• When I execute these it says the match does not work and I get the below error for both "threat.indicator.ip" and "threat.indicator.url.domain".

{
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "Match field 'threat.indicator.url.domain' doesn't have a correct mapping type for policy type 'match'"
      }
    ],
    "type": "exception",
    "reason": "Match field 'threat.indicator.url.domain' doesn't have a correct mapping type for policy type 'match'"
  },
  "status": 500
}

• If I recreate the policies and use 'logs-ti_opencti_latest.dest_indicator-' instead of 'ds-logs-ti_opencti.indicator-default-' the command executes successfully.

3. Check that the enrich index is created.

• This does return results and shows Yellow to Green.
  grep opencti-domain-enrichment-policy /opt/so/log/elasticsearch/securityonion.log

• When attempt the check from Wes' documentation nothing shows up, but if I go into the UI and follow though stack management, I can see .enrich- in "Index Management" > Indicies "Check Included hidden indicies".
  sudo so-elasticsearch-query _cat/indices | grep enrich

4 Create Ingest Pipeline

• I then create this file.
• The video doesn't reference any action to be taken other than restart elasticsearch. Wes' documentation says you need to the final pipeline for the indexes you want to enrich.

{ 
 "description" : "Threat Enrichment",
 "processors" : [
  { "enrich":{ "description": "Enrich IP address info with threat intel indicators", "policy_name": "opencti-ip-enrichment-policy", "target_field": "enrichment", "field": "destination.ip", "ignore_failure": true }},
  { "enrich":{ "description": "Enrich IP address info with threat intel indicators", "policy_name": "opencti-ip-enrichment-policy", "target_field": "enrichment", "field": "destination.address", "ignore_failure": true }},  
  { "enrich":{ "description": "Enrich DNS queries info with threat intel indicators", "policy_name": "opencti-domain-enrichment-policy", "target_field": "enrichment", "field": "dns.query.name", "ignore_failure": true }},
  { "enrich":{ "description": "Enrich SSL host info with threat intel indicators", "policy_name": "opencti-domain-enrichment-policy", "target_field": "enrichment", "field": "ssl.server_name", "ignore_failure": true }},
  { "enrich":{ "description": "Enrich HTTP traffic info with threat intel indicators", "policy_name": "opencti-domain-enrichment-policy", "target_field": "enrichment", "field": "http.virtual_host", "ignore_failure": true }}
 ]
}

5. Add to the final pipelines.

• Since the file referenced is no longer available '/opt/so/saltstack/local/pillar/global.sls', I did it in "Administration > Configuration > Elasticsearch" and added to the import template then reviewed where the change was made and found it in "/opt/so/saltstack/local/pillar/elasticsearch/soc_elasticsearch.sls" and added another setting for 'so-zeek'.

elasticsearch:
  index_settings:
    so-import:
      index_template:
        template:
          settings:
            index:
              final_pipeline: enrich.misp
              number_of_shards: 1
      warm: 7
      close: 45
      delete: 365

My setup

    index_settings:
        so-import:
            index_template:
                template:
                    settings:
                        index:
                            final_pipeline: threat.enrich
        so-logs-pfsense_x_log:
            index_template:
                template:
                    settings:
                        index:
                            final_pipeline: threat.enrich
        so-zeek:
            index_template:
                template:
                    settings:
                        index:
                            final_pipeline: threat.enrich

When that is all done i reboot and look in the index "logs-ti_opencti_latest.dest_indicator-*" index for an IP and host. I test with a ping or a lookup. I see the Zeek data but nothing ever get attached to enrich the data. Since then, I made a grave mistake in one of my instances. I thought maybe if i delete the OpenCTI Index Template and Component Template, that might fix the match error. It did not. I assumed if I removed and reinstalled the Fleet integration it would put them back and it didn't.

1. Is what I am trying to accomplish, enrich my Zeek data with threat intel indicators possible in SO 2.4.120+?
2. What am I missing or doing wrong here?
3. Is there a way to get the Templates back?

Thanks as always all.

[reyesj2]

The .ds indices are part of a datastream. Just think of it as part of an overall index. Try using the datastream name in your enrich policy like

PUT /_enrich/policy/opencti-ip-enrichment-policy
{
 "match":{
 "indices":"logs-ti_opencti.indicator-default-*",
 "match_field":"threat.indicator.ip",
 "enrich_fields":[
  "threat.indicator.provider",
  "threat.indicator.type",
  "threat.indicator.confidence",
  "opencti.indicator.score",
  "tags"
  ]
 }
}

Execute the policy and you should have an enrich index created, make sure it has docs in it not just empty (if its empty make sure the policy executed with an OK status)

If you set ignore failure to false do you get errors showing the enrich pipeline is being run, but just data is malformed in someway?

To get the templates back for opencti if you go to the integrations page and click settings there is a 'reinstall' option

[Slipperyclock]

Thank you for the advice. So I went back and put the policies in place as below. When I go to execute the polices I get the same error. Thanks for the heads up on "reinstall" that restored everything.

Policy creation: both return OK.

PUT /_enrich/policy/opencti-ip-enrichment-policy
{
 "match":{
 "indices":"logs-ti_opencti.indicator-default-*",
 "match_field":"threat.indicator.ip",
 "enrich_fields":[
  "threat.indicator.provider",
  "threat.indicator.type",
  "threat.indicator.confidence",
  "opencti.indicator.score",
  "tags"
  ]
 }
}

PUT /_enrich/policy/opencti-domain-enrichment-policy
{
 "match":{
 "indices":"logs-ti_opencti.indicator-default-*",
 "match_field":"threat.indicator.url.domain",
 "enrich_fields":[
  "threat.indicator.provider",
  "threat.indicator.type",
  "threat.indicator.confidence",
  "opencti.indicator.score",
  "tags"
  ]
 }
}

Execution of Policy

POST /_enrich/policy/opencti-ip-enrichment-policy/_execute

POST /_enrich/policy/opencti-domain-enrichment-policy/_execute

[Slipperyclock]

For a follow up I did the same settings for AbuseCH and it worked as expected for Domains. It seems to be an issue with the way SO does the mappings for threat.indicator.ip and threat.indicator.url.domain for opencti integration. I tested this and the intel data does get enriched in Zeek data.

AbuseCH Policy

PUT /_enrich/policy/ti-abusech-url-domain-policy
{
 "match":{
 "indices":".ds-logs-ti_abusech.url-*",
 "match_field":"threat.indicator.url.domain",
 "enrich_fields":[
  "threat.indicator.provider",
  "threat.indicator.type",
  "threat.indicator.url.original",
  "threat.indicator.reference",
  "tags"
  ]
 }
}

Execution (Worked as expected)

POST /_enrich/policy/ti-abusech-url-domain-policy/_execute

Pipeline

{ 
 "description" : "Threat Enrichment",
 "processors" : [
  { "enrich":{ "description": "AbuseCH Enrich DNS queries info with threat intel indicators", "policy_name": "ti-abusech-url-domain-policy", "target_field": "enrichment", "field": "dns.query.name", "ignore_failure": true }},
  { "enrich":{ "description": "AbuseCH Enrich SSL host info with threat intel indicators", "policy_name": "ti-abusech-url-domain-policy", "target_field": "enrichment", "field": "ssl.server_name", "ignore_failure": true }},
  { "enrich":{ "description": "AbuseCH Enrich Meraki log info with threat intel indicators", "policy_name": "ti-abusech-url-domain-policy", "target_field": "enrichment", "field": "url.domain", "ignore_failure": true }},  
  { "enrich":{ "description": "AbuseCH Enrich HTTP traffic info with threat intel indicators", "policy_name": "ti-abusech-url-domain-policy", "target_field": "enrichment", "field": "http.virtual_host", "ignore_failure": true }}
 ]
}

Final Pipeline Settings

        so-logs-cisco_meraki_x_log:
            index_template:
                template:
                    settings:
                        index:
                            final_pipeline: threat.enrich
        so-zeek:
            index_template:
                template:
                    settings:
                        index:
                            final_pipeline: threat.enrich

[ColeVan]

I'm facing the same issues when trying to create a OpenCTI enrichment policy. Seems to be a mapping error that I am unable to pinpoint at the moment.

[reyesj2]

I went through setting up the demo version of opencti and got data ingested and was able to reproduce the issue above.

After messing with it a bit I reviewed the opencti integration mappings. We don't modify the field mappings that come with integrations, we just create any needed index templates.

The fix I was able to do was create a custom component template with the specific mapping needed. Enrich policies for "match" need keyword or constant_keyword mapping. Here are the steps you should be able to follow to fix.

Here is a dev tools request you can make to create the component template I used

PUT _component_template/logs-ti_opencti.indicator@custom
{
  "template": {
    "mappings": {
      "properties": {
        "threat": {
          "type": "object",
          "properties": {
            "indicator": {
              "type": "object",
              "properties": {
                "ip": {
                  "type": "keyword"
                }
              }
            }
          }
        }
      }
    }
  }
}

Not sure why all fields for the integration aren't included in the logs-ti_opencti.indicator@package component template, but this explicity adds a 'keyword' mapping for threat.indicator.ip. If you need to edit or make changes in the future you can find it in Stack Management -> index management -> component templates -> logs-ti_opencti.indicator@custom

Then create the enrich policy

PUT /_enrich/policy/opencti-ip-enrichment-policy
{
    "match": {
        "indices": "logs-ti_opencti.indicator-default*",
        "match_field": "threat.indicator.ip",
        "enrich_fields": [
          "threat.indicator.provider",
          "threat.indicator.type",
          "threat.indicator.confidence",
          "opencti.indicator.score",
          "tags"
          ]
    }
}

Execute it

POST /_enrich/policy/opencti-ip-enrichment-policy/_execute

Should now get

{
  "status": {
    "phase": "COMPLETE"
  }
}

If not you may need to just clear the old opencti data

DELETE _data_stream/logs-ti_opencti.indicator-default

Once done you should see the .enrich index for opencti

GET _cat/indices/.enrich-opencti-ip-enrichment-policy*
green open .enrich-opencti-ip-enrichment-policy-1742680390171 Douvpu_dS16u-M8W2XUujQ 1 2 18050 0   2.4mb 836.8kb 836.8kb