Deleting overrides -> A detection with this public Id already exists. Please choose a different public Id.

[ejgh-oe]

Version

3.1.0

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

32

RAM

512GB

Storage for /

4TB

Storage for /nsm

4TB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hi,
Just wanted to delete a bunch of overrides for Suricata rules via the GUI, but whenever I tried to change any rule/override it resulted in an error message "A detection with this public Id already exists. Please choose a different public Id."

Here's an example:

Tried to modify (i.e. delete the override) on Rule 2001581 since it refers to a variable $DA_NODES that's no longer defined. First I deleted the corresponding variable definition via Administration > Configuration > Adv > Suricata > Advanced and then tried to remove the rule override mentioning that variable by clicking on the "x"

Instead of the override being deleted I got an error message like

Please note that this happens with every signature I try to delete the tuning from, not just this one :-(

BTW, /opt/so/log/suricata/suricata.log on the sensor just shows entries like the ones below, but no other clue relating to the error message A detection with this public Id already exists. Please choose a different public Id.:

[11 - Suricata-Main] 2026-08-05 15:34:24 Error: rule-vars: Variable "DA_NODES" is not defined in configuration file
[11 - Suricata-Main] 2026-08-05 15:34:24 Error: threshold-config: failed to parse $DA_NODES

Any ideas as to what's wrong here and how I can delete those overrides?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[octothorp88]

You have a couple of options. You can run either of the following scripts to remove the detection and have 0 tunings or you can run the second option and remove the detection with the tunings.

Just make sure and replace the so_detection.publicId to match your detection id.

Stay in Detections with 0 tunings

  /sbin/so-elasticsearch-query '*so-detection/_update_by_query' -d '{
    "query": {
      "match": {
        "so_detection.publicId": "2066594"
      }
    },
    "script": {
      "source": "ctx._source[\"so_detection.overrides\"] = []",
      "lang": "painless"
    }
  }'

Remove from Detections

  /sbin/so-elasticsearch-query '*so-detection/_update_by_query' -d '{
    "query": {
      "match": {
        "so_detection.publicId": "2066594"
      }
    },
    "script": {
      "source": "ctx._source.so_detection.remove(\"overrides\")",
      "lang": "painless"
    }
  }'

[ejgh-oe]

So, just to be clear: Do both of the options you mentioned work only on the detection with overrides, i.e. the ones defined in the Detections tab - or do they work on all detections globally?

In case the work globally, i.e. on all detections/signatures - how would I get a signature back in case it gets deleted inadvertently?

[cm-ops]

Would you go into Kibana > Dev Tools and try this command, it will look for duplicate SIDs.

GET so-detection/_search 
{
  "size": 0,
  "aggs": {
    "duplicate_publicIds": {
      "terms": {
        "field": "so_detection.publicId",
        "min_doc_count": 2,
        "size": 10000
      }
    }
  }
}

[ejgh-oe]

Hi,

Would you go into Kibana > Dev Tools and try this command, it will look for duplicate SIDs.

I'm attaching the output (.json) in this posting
kibana-dev-output-2026-08-14.json

Wrt the original error, i.e. not being able to delete "tuning"(s) via the Detections interface: As mentioned in my original post there's one thing in common with all the overrides I cannot delete: The all refer to variables that are not longer defined.
So for example the override for rule 2001581 refers to a variable $DA_NODES that is no longer defined:

Could this be the underlying problem?

Additionally please note that in addition to the overrides via the Detection interface we've also got custom suricata rules that go a file in /nsm/rules/custom-local-repos/local-suricata being referenced by soc > config > server > modules > suricataengine > rulesetSources > default

[cm-ops]

          "key": "2001581",
          "doc_count": 3
        },

It looks like you have 3 of each SID in your so-detection index. How many overrides do you have currently configured.

For the above SID, would you also run this in Kibana and let me know what the _id values are for that index.

GET so-detection/_search
{
    "query": {
        "match": {
          "so_detection.publicId": "2001581"
        }
    }
}

[ejgh-oe]

It looks like you have 3 of each SID in your so-detection index. How many overrides do you have currently configured.

There's just one override for that SID defined via the detections interface - nothing else:

So I really wonder why this and all other SIDs I've got overrides configured end up three times in the so-detection index.

GET so-detection/_search

{
"query": {
"match": {
"so_detection.publicId": "2001581"
}
}
}

I've attached the results here.

[cm-ops]

I didn't see the results of the above, how many rules in Detections have overrides? I know this one does. If you do not have a lot of overrides, it might be easier to remove the so-detection and so-detectionhistory indices and reimport your rules.

[ejgh-oe]

I didn't see the results of the above, ...

Sorry - my bad - forgot to attach the log - attaching it this time.

so-detection-search-result.json

how many rules in Detections have overrides? I know this one does. If you do not have a lot of overrides, it might be easier to remove the so-detection and so-detectionhistory indices and reimport your rules.

I've got 31 rules with overrides configured via the Detections interface in total.

Speaking of

... easier to remove the so-detection and so-detectionhistory indices and reimport your rules...

• will removing those indices only affect just the rules overrides configured via the Detections interface or anything else? Reason I'm asking: In addition to the 31 rules with overrides I've got a lot of fine grained custom rules (SID > 9000000) with custom triggers/messages stored in a file under /nsm/rules/custom-local-repos/local-suricata. Losing them would be a real nightmare to say the least.

• In case removing the so-detection and so-detectionhistory indices only affects the overrides via the GUI and not the custom defined rules, how should I go about deleting these indices and reimport the rules? (sorry I'm not so deep into elastic-stuff...)

[cm-ops]

Okay, yeah reimport will wipe out all the overrides, so might not be an option.

For 2001581, try this:

POST /so-detection/_delete_by_query
{
  "query": {
    "ids": {
      "values": ["9e0da5ed-6ed9-4c6b-b2a3-2817fabf5875", "OZu43JUBJqwPDyQ55hU7"]
    }
  }
}

Then try to remove your override again.

[ejgh-oe]

Good news: Running

{
  "query": {
    "ids": {
      "values": ["9e0da5ed-6ed9-4c6b-b2a3-2817fabf5875", "OZu43JUBJqwPDyQ55hU7"]
    }
  }
}

resulted in

  "took": 14,
  "timed_out": false,
  "total": 2,
  "deleted": 2,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1,
  "throttled_until_millis": 0,
  "failures": []
}

Tried to delete the override -> and yeah, this time it worked :-)

So just for my understanding, what's the magic behind and how can I get rid of the other "blocking overrides" too? (i.e. where do the values like ..."9e0da5ed-6ed9-4c6b-b2a3-2817fabf5875", "OZu43JUBJqwPDyQ55hU7" come from so I can set up the _delete_by_query statements for the remaining blocking overrides?

[cm-ops]

Those were the _id field values for the documents for "so_detection.publicId": "2001581". You have three (3) copies of every SID in your so-detection index. So you would need to query the SID with:

GET so-detection/_search
{
    "query": {
        "match": {
          "so_detection.publicId": "2001581"
        }
    }
}

Find the three _id field values and remove the two that have no override configured with that last command to delete by query using those IDs.

[ejgh-oe]

Thanks for the clarification - makes sense.

So in order to get rid of the other Detections with overrides that I cannot delete what would be the best approach?

I thought about something like:
For every detection with override that I cannot delete via the detections interface: find the corresponding SID run

GET so-detection/_search
{
    "query": {
        "match": {
          "so_detection.publicId": "<SID>"
        }
    }
}

grab the "_id"s values from the output of the above that have no override

delete them using

{
  "query": {
    "ids": {
      "values": ["<ID1>", "<ID2>"]
    }
  }
}

where ID1 and ID2 come from the output of

GET so-detection/_search
{
    "query": {
        "match": {
          "so_detection.publicId":
...

Would this be the correct approach?

[cm-ops]

It would, without reimporting the rules, which with a lot of overrides would wipe them out. If they are all Suricata, you might be able to reimport and use https://docs.securityonion.net/en/3/main/so-detections-overrides-import/ perhaps.

Do you have any custom sigma or yara rules?

[ejgh-oe]

Do you have any custom sigma or yara rules?
No, just custom suricata rules (SID > 9000000) and rules with overrides via the Detections interface.

So in fact I just want to remove all the overrides in the Detections interface reverting the corresponding rules back to their default behavior.

[cm-ops]

To set everything back to default and reimport the rules you would do the following:

Remove the so-detection and so-detectionhistory indices - sudo so-elasticsearch-query so-detection -XDELETE (same for detectionhistory)

Remove all the contents from ls -l /opt/so/conf/soc/fingerprints/ should be a combination of .state and .fingerprint files.

Restart SOC with so-soc-restart, then in SOC > Detections synchronize each ruleset. You should see the rules reimport in the SOC log /opt/so/log/soc/sensoroni-server.log Sigma (Elastalert) and Yara (Strelka) go pretty quick, Suricata rules take a little bit of time.

[xuxu298]

Worth separating two states before you clean these up, because your own log already says they disagree. threshold-config: failed to parse $DA_NODES is the sensor reporting it could not parse that entry, so whatever that tuning was meant to do, it was not doing it on that node regardless of what the GUI listed.

Cheap way to tell which one the script above actually changed: pull the Suricata startup block out of /opt/so/log/suricata/suricata.log before and after, and compare the rules-successfully-loaded and failed counts plus the threshold-config lines. Counts unchanged means you fixed bookkeeping in the index. Counts changed means the sensor ruleset moved.

That still only tells you the rule loaded, not that it fires. Replaying traffic that should hit 2001581 is the only check I trust for that. On the stacks I have measured, threshold and suppression state are where the console and the engine drift furthest apart.