Type conflict for client.ip and client.geo.location

[mckenziemmack]

Version

2.4.110

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

64

RAM

251G

Storage for /

558G

Storage for /nsm

38T

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hello,

I'm encountering type conflicts from my Okta integration. Specifically, the fields client.ip (desired type: ip, received type: keyword) and client.geo.location (desired type: geo_point, received type: object) are causing mapping issues.

I am aware of the known issues, and I see that there is now the field okta.client.ip that I assume might have been created in relation to this conflict, but that doesn't solve my issues with client.ip. I have another service that uses client.ip, so this is greatly affecting me. I've tried reindexing, but that just made the Okta indice I reindexed have client.ip as text.

I would appreciate any advise on how to address these type conflicts and ensure proper data mapping between my services and Okta?

Thank you!

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Which indices are in conflict? If you check in Kibana > Stack Management > Data Views and select logs-* In the Field Type box hit the dropdown and select conflict. If you hit the Conflict hyperlink it should tell you which indices are what data type.

[mckenziemmack]

Here's a screenshot from Data Views of the conflicting fields:

Infoblox/Palo Alto/Zeek are all of the same type for client.ip, but Okta is causing the conflict.

And similarly with client.geo.location.

[reyesj2]

Try this, go to Kibana -> stack management -> index management -> index templates and search for okta

Click on edit next to the so-logs-okta template

find component templates and search for okta-mappings. Then hit the + and add it to the list on the left-hand side.

Once done you can hit next until you get the last page and save the template. You will need to re-reindex the okta index, but it should have the correct mappings afterwards! This fix should be incorporated into the next Security Onion release aswell

[mckenziemmack]

Thank you for the easy to follow instructions. However, I tried re-indexing in Dev Tools and received an error.

It did however create a new index with the client.ip as type text.
GET /_cat/indices? gave me:

green open .ds-logs-okta.system-default-2024.12.13-000006                     GJmgfiTLTwuKiVVWyl1A7g 1 0    954175     0   1.6gb   1.6gb   1.6gb
green open .ds-logs-okta.system-default-2024.12.13-000006-1                   gn1Fjtc5SwqfKho1WnvXjw 1 1    954175     0   3.9gb     2gb     2gb

[reyesj2]

Did you run the re-index command multiple times? It looks larger than the original

What is the output of

GET _data_stream/logs-okta.system-default 

Likely the output will be similar to

{
  "data_streams": [
    {
      "name": "logs-elastic_agent.osquerybeat-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-logs-elastic_agent.osquerybeat-default-2025.01.13-000001",
          "index_uuid": "pOYzWVLUTcGp8-Jic28bjQ",
          "prefer_ilm": true,
          "ilm_policy": "so-logs-elastic_agent.osquerybeat-logs",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 1,
      "status": "GREEN",
      "template": "so-logs-elastic_agent.osquerybeat",
      "ilm_policy": "so-logs-elastic_agent.osquerybeat-logs",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}

Showing that the only "index" is .ds-logs-elastic_agent.osquerybeat-default-2025.01.13-000001

If that is the case for you (only 1 index not showing the second .ds-logs-okta.system-default-2024.12.13-000006-1 you manually created then you can delete that second index .ds-logs-okta.system-default-2024.12.13-000006-1 which is duplicated data from .ds-logs-okta.system-default-2024.12.13-000006 (original with incorrect mappings)

Then you can try rolling over the okta datastream to ensure new data splits off and starts writing to a new index that is using the correct index template with proper mappings

POST logs-okta.system-default/_rollover

If you then re-ran the command

GET _data_stream/logs-okta.system-default 

you should see 2 indices listed similar to

{
  "data_streams": [
    {
      "name": "logs-system.syslog-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-logs-system.syslog-default-2025.01.13-000001",
          "index_uuid": "0lTRLNvLTo6xXGkZElcMIg",
          "prefer_ilm": true,
          "ilm_policy": "so-logs-system.syslog-logs",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-logs-system.syslog-default-2025.01.14-000002",
          "index_uuid": "5p8VVQh_QhaGI7TFtQxxLA",
          "prefer_ilm": true,
          "ilm_policy": "so-logs-system.syslog-logs",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 2,
      "status": "GREEN",
      "template": "so-logs-system.syslog",
      "ilm_policy": "so-logs-system.syslog-logs",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}

The old (original) index .ds-logs-okta.system-default-2024.12.13-000006 then needs to be reindex for the mappings to be updated and clear up the type conflicts

Once you run the reindex, do not re-run it again. You may timeout from dev tools, but you can later check the progress of the reindex by running

GET _tasks?actions=*reindex&detailed

Once it shows completed you can close the original index (instead of deleting) so we can verify that the type conflict is gone

POST .ds-logs-okta.system-default-2024.12.13-000006/_close

[reyesj2]

Created issue #14106