mapping conflict under Kibana / stack management / index patterns / *:so-*

[jchenturbo]

While working on our standalone instance of SO 2.3.61.. i noticed this error under Kibana / stack management / index patterns / :so-

Time field: '@timestamp'
Mapping conflict

A field is defined as several types (string, integer, etc) across the indices that match this pattern. You may still be able to use these conflict fields in parts of Kibana, but they will be unavailable for functions that require Kibana to know their type. Correcting this issue will require reindexing your data.

I wasn't sure if we caused it so we deployed another instance fresh from the ISO and see the same warning..

Is this something to be concerned about? how can we fix it?

[rwaight]

Hey @jchenturbo, would you be able to find which fields have conflicts by looking in the All field types drop-down menu and select any entries that have multiple values (i.e., text, long) and providing that in a screenshot?

I found that process.pid has a mapping conflict in SO 2.3.61 also. I'm assuming we will need to add mappings to Elasticsearch, but it appears there are dashboard dependencies that do not exist in the mapping.

[jchenturbo]

yes.... process.pid is the one w/ the mapping conflict on my instances.

[weslambert]

I'm pretty sure this is because of the so-ossec indices (text), but can you please run the following and provide the output?

sudo so-elasticsearch-query */_mapping/field/process.pid

[weslambert]

Hey @jchenturbo, would you be able to find which fields have conflicts by looking in the All field types drop-down menu and select any entries that have multiple values (i.e., text, long) and providing that in a screenshot?

I found that process.pid has a mapping conflict in SO 2.3.61 also. I'm assuming we will need to add mappings to Elasticsearch, but it appears there are dashboard dependencies that do not exist in the mapping.

Most of the fields that are "missing" from the dashboard visualizations are due to the fact that the Field Caps API is now used and the mappings are refreshed (non-existent if you don't have events that contain the fields, because the mappings are derived from Elasticsearch). We used to pre-load them and you would not see the errors you see now, but now that the auto-refresh occurs, you see that kind of behavior.

[jchenturbo]

{"so-zeek-2021.08.22":{"mappings":{}},"so-zeek-2021.08.23":{"mappings":{}},"so-elasticsearch-2021.08.12":{"mappings":{}},"so-elasticse arch-2021.08.11":{"mappings":{}},"so-kibana-2021.08.20":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type" :"long"}}}}},"elastalert_status":{"mappings":{}},"so-elasticsearch-2021.08.10":{"mappings":{}},"so-elasticsearch-2021.08.16":{"mapping s":{}},"so-elasticsearch-2021.08.15":{"mappings":{}},"so-ossec-2021.08.09":{"mappings":{"process.pid":{"full_name":"process.pid","mapp ing":{"pid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":32765},"security":{"type":"text","analyzer":"es_securi ty_analyzer"}}}}}}},"so-elasticsearch-2021.08.14":{"mappings":{}},"so-elasticsearch-2021.08.13":{"mappings":{}},"so-elasticsearch-2021 .08.19":{"mappings":{}},"so-elasticsearch-2021.08.18":{"mappings":{}},"so-elasticsearch-2021.08.17":{"mappings":{}},"so-kibana-2021.08 .18":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"long"}}}}},"so-osquery-2021.08.09":{"mappings":{} },"so-kibana-2021.08.19":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"long"}}}}},".apm-custom-link" :{"mappings":{}},"so-kibana-2021.08.12":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"long"}}}}},"so -kibana-2021.08.13":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"long"}}}}},"so-kibana-2021.08.10": {"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"long"}}}}},"so-kibana-2021.08.11":{"mappings":{"proces s.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"long"}}}}},"elastalert_status_error":{"mappings":{}},"so-kibana-2021.08.16 ":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"long"}}}}},"so-kibana-2021.08.17":{"mappings":{"proc ess.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"long"}}}}},"so-kibana-2021.08.14":{"mappings":{"process.pid":{"full_name ":"process.pid","mapping":{"pid":{"type":"long"}}}}},"so-kibana-2021.08.15":{"mappings":{"process.pid":{"full_name":"process.pid","map ping":{"pid":{"type":"long"}}}}},".apm-agent-configuration":{"mappings":{}},"so-elasticsearch-2021.08.09":{"mappings":{}},".tasks":{"m appings":{}},".kibana-event-log-7.13.4-000001":{"mappings":{}},"so-kibana-2021.08.23":{"mappings":{"process.pid":{"full_name":"process .pid","mapping":{"pid":{"type":"long"}}}}},"so-kibana-2021.08.21":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pi d":{"type":"long"}}}}},"so-kibana-2021.08.22":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"long"}}} }},".kibana_task_manager_7.13.4_001":{"mappings":{}},"so-ossec-2021.08.22":{"mappings":{"process.pid":{"full_name":"process.pid","mapp ing":{"pid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":32765},"security":{"type":"text","analyzer":"es_securi ty_analyzer"}}}}}}},"so-ossec-2021.08.23":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"text","field s":{"keyword":{"type":"keyword","ignore_above":32765},"security":{"type":"text","analyzer":"es_security_analyzer"}}}}}}},"so-ossec-202 1.08.20":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"text","fields":{"keyword":{"type":"keyword"," ignore_above":32765},"security":{"type":"text","analyzer":"es_security_analyzer"}}}}}}},"so-ossec-2021.08.21":{"mappings":{"process.pi d":{"full_name":"process.pid","mapping":{"pid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":32765},"security":{ "type":"text","analyzer":"es_security_analyzer"}}}}}}},"so-zeek-2021.08.09":{"mappings":{}},"elastalert_status_silence":{"mappings":{} },".kibana_7.13.4_001":{"mappings":{}},"so-zeek-2021.08.10":{"mappings":{}},"so-ossec-2021.08.11":{"mappings":{"process.pid":{"full_na me":"process.pid","mapping":{"pid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":32765},"security":{"type":"text ","analyzer":"es_security_analyzer"}}}}}}},"so-zeek-2021.08.11":{"mappings":{}},"so-ossec-2021.08.12":{"mappings":{"process.pid":{"ful l_name":"process.pid","mapping":{"pid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":32765},"security":{"type":" text","analyzer":"es_security_analyzer"}}}}}}},"so-zeek-2021.08.12":{"mappings":{}},"so-zeek-2021.08.13":{"mappings":{}},"so-ossec-202 1.08.10":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"text","fields":{"keyword":{"type":"keyword"," ignore_above":32765},"security":{"type":"text","analyzer":"es_security_analyzer"}}}}}}},"so-zeek-2021.08.14":{"mappings":{}},"so-elast icsearch-2021.08.23":{"mappings":{}},"so-zeek-2021.08.15":{"mappings":{}},"elastalert_status_status":{"mappings":{}},"so-elasticsearch -2021.08.22":{"mappings":{}},"so-zeek-2021.08.16":{"mappings":{}},"so-elasticsearch-2021.08.21":{"mappings":{}},"so-zeek-2021.08.17":{ "mappings":{}},"elastalert_status_past":{"mappings":{}},"so-elasticsearch-2021.08.20":{"mappings":{}},"so-zeek-2021.08.18":{"mappings" :{}},"so-ossec-2021.08.19":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"text","fields":{"keyword":{ "type":"keyword","ignore_above":32765},"security":{"type":"text","analyzer":"es_security_analyzer"}}}}}}},"so-zeek-2021.08.19":{"mappi ngs":{}},"so-ossec-2021.08.17":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"text","fields":{"keywor d":{"type":"keyword","ignore_above":32765},"security":{"type":"text","analyzer":"es_security_analyzer"}}}}}}},"so-ossec-2021.08.18":{" mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_abov e":32765},"security":{"type":"text","analyzer":"es_security_analyzer"}}}}}}},"so-ossec-2021.08.15":{"mappings":{"process.pid":{"full_n ame":"process.pid","mapping":{"pid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":32765},"security":{"type":"tex t","analyzer":"es_security_analyzer"}}}}}}},"so-ossec-2021.08.16":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pi d":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":32765},"security":{"type":"text","analyzer":"es_security_analyz er"}}}}}}},"so-ossec-2021.08.13":{"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"text","fields":{"keyw ord":{"type":"keyword","ignore_above":32765},"security":{"type":"text","analyzer":"es_security_analyzer"}}}}}}},"so-ossec-2021.08.14": {"mappings":{"process.pid":{"full_name":"process.pid","mapping":{"pid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_ab ove":32765},"security":{"type":"text","analyzer":"es_security_analyzer"}}}}}}},"so-kibana-2021.08.09":{"mappings":{"process.pid":{"ful l_name":"process.pid","mapping":{"pid":{"type":"long"}}}}},".security-7":{"mappings":{}},".async-search":{"mappings":{}},"so-zeek-2021