Could not locate that index-pattern-field (id: "FIELD_NAME")

[tcritch05]

Hello!

I'm working on standing up a Security Onion 2 environment (converting from SO1 with a fresh reinstall) and I didn't notice any issues during the installation. At the moment, I have a manager online and 1 forwarding sensors connected to it. I can confirm that the sensor is checking into the manager and that the sensor is seeing logs. However in Kibana/Elastic, none of the tables are populating and only display "Could not locate that index-pattern-field (id: "FIELD_NAME")"

Examples:
Could not locate that index-pattern-field (id: @timestamp)
Could not locate that index-pattern-field (id: event.category.keyword)
Could not locate that index-pattern-field (id: event.dataset.keyword)
Could not locate that index-pattern-field (id: total_bytes)
Could not locate that index-pattern-field (id: source_ip)

Any ideas how to troubleshoot this?

[facyber]

Just to confirm that I also experience similar issues ever since I updated my setup to 2.3.40 (manager + heavy node), only I see error in Osquery dashboard.

Could not locate that index-pattern-field (id: result.name.keyword)

[tcritch05]

Good to know! I'm currently running 2.3.40 as well.

[tcritch05]

When comparing my production Security Onion deployment with my personal lab, it appears that I've lost all of the Kibana Index Patterns (https:///kibana/app/management/kibana/indexPatterns). If I click into each pattern, I only have 5 entries.

Any idea how to restore the patterns?

[defensivedepth]

@facyber The osquery Kibana error is a different issue - Please run sudo so-kibana-config-load

Also - do you have any new osquery data coming in?

[facyber]

I can see that data is coming to the manager with tcpdump in port 8090, but error is still present in Kibana. Now I noticed its also issue for Security Onion - Files dashboard too.

If I compare before and now Index patterns, I see additional two patterns logs-* and metrics-* but the rest are same. Do I also need to restart Elasticsearch maybe or some other service?

[defensivedepth]

@facyber Based on the other Discussion, that error is to be expected right now. Once you start having osquery data in your system, that error will go away.

[defensivedepth]

@tcritch05 What version of SO2 are you on?

Please run sudo so-kibana-config-load and see if that makes a difference.

[tcritch05]

I'm running 2.3.40 on CentOS 7. Running so-kibana-config-load didn't seem to make any changes.

[tcritch05]

Here is what all the dashboards look like:

If you look at Kibana Index Patterns, I'm seeing patterns:

Inside the SO-* pattern, you can see we are missing everything.

I believe I'm looking in the right spot, but it appears that all the patterns are present:
/opt/so/conf/elasticsearch/ingest/

[defensivedepth]

@tcritch05 You say you have a Manager & sensor - do you have a search node? That is where the data is actually stored. You could also do a Manager + Search node if you want to combine the node types.

[tcritch05]

I think that might be my issue, I only selected Manager and not Manager + Search Node. I don't have a Search Node. Time to reimage!

[defensivedepth]

Great! Have fun! :)