No v2.4 Kibana dashboards for v2.3 data

[petiepooo]

Version

2.4.40

Installation Method

Network installation on Ubuntu

Description

upgrading

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

32

RAM

128G

Storage for /

250G

Storage for /nsm

2T

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I followed the 2.3 -> 2.4 upgrade instructions from here (loosely; I adapted them for Ubuntu). The upgrade was generally successful, but there are some things I noticed once complete.

FWIW, this is one instance where Ubuntu shines, as all it took was a do-release-upgrade. No need to export the /nsm LV; just a little cleanup of the so-* files, installtmp dir, intca.crt file, and soremote user. I'll post a more detailed process later, if people are interested. I know Ubuntu is not officially supported, but I'm glad it's no longer blocked as an option. The observations below, though, are not related to the underlying OS.

1. The old data is still present and can be retrieved in the Alerts and Hunts sections, which is fantastic.
2. The old data is not viewable in Kibana dashboards, as the default dashboards query only the new indices.
3. There are no alternative dashboards for viewing v2.3 data from before the upgrade.
4. The z16.04 dashboards are surprisingly still present, but do not display the 2.3 data.
5. Even on the new data, collected after the upgrade, there is no link to pivot from Kibana to Hunt, eg., to retrieve a PCAP.
6. I have not run the system long enough to see if the old data ages out, now that curator is deprecated.

Devs, please consider the following:

1. replacing Kibana's z16.04 dashboards with z2.3 dashboards
2. adding the ability to pivot from the new dashboards to Hunt, and/or directly to PCAP
3. recommending how users clear out old v2.3 indices now that curator is gone

Perhaps there is a way to add z2.3 dashboards only if needed, perhaps by running a salt state, and deleting them once they're no longer needed. Or optionally loading them based on a setting in the minion statefile.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[petiepooo]

In looking into this further, it seems the z16.04 Kibana dashboards from 2.3.280 are sticking around post-upgrade. That may also be why pivot links aren't working. Is there a simple way to remove all dashboards and visualizations supplied by so-setup without deleting any other saved objects we may have created? Perhaps a tool that will take a saved objects ndjson file, build a heirarchy of objects and a graph of dependencies, then list all dependencies below a specific object to feed to curl?

It also looks like most of the 2.4 dashboards and visualizations have the same id values (guids?) as the equivalent 2.3 objects. That would prevent one from just changing labels and merging the two together. Could one simply modify the 2.4 index patterns so the saved objects pull up both old and new data, or have fields changed in the new release and index style?

For deleting old 2.3 indices first, so-elasticsearch-indices-delete-delete still lists them and orders them correctly by date, but with the datastream logic in place for 2.4 indices, I think it ignores them, or tries to delete the nonexistent matching datastream. A quick check for a 2.3-style name first (if it starts with 'so-' then just delete it) would allow the script to support both legacy and new index styles.

Anyone else trying to handle a migration from 2.3 to 2.4 without losing continuity of data?

[DebianGuru]

I'm in the exact samb boat. Did you find any answers? I actually tried the 2.3 > 2.4 upgrade in test VMs before the real thing and I had no such problems.

[petiepooo]

I'm working on a script that essentially takes the default saved_objects.ndjson file from 2.3, strips out all the z16.04 dashboards and visualizations, and loops through the remaining dashboards and visualizations, changing each object's ID along with all references to it in the other saved objects. It also changes the dashboard titles, eg prepending then with z2.3 so the names don't collide with existing 2.4 objects.

The plan then is to upgrade, make sure the new 2.4 saved objects are loaded correctly, then import the z2.3 saved objects. Changed IDs mean they won't collide with the 2.4 objects on which they're based. I may need to rewrite the pivot URL on the 2.3 dashboards to go straight to pcap instead of to the SOC pages, too, as I don't know what it would take to get the SOC alert and hunt pages to pull up records from 2.3...

You're running on Ubuntu too, @DebianGuru? And you were not able to access zeek/suricata logs create before the upgrade once your upgrade was complete?

I did notice the events were stored in a data stream index, which is new to 2.4. It may be as simple as changing the index patterns so both old and new indices match, but I can't say for sure that Elasticsearch even allows mixing those two in a single query.

[DebianGuru]

I'm using the SO iso which is based on Oracle Linux.

And you were not able to access zeek/suricata logs create before the upgrade once your upgrade was complete?

Yes. I upgraded on 3/7/2024. All data prior to that date does not show in Kibana the SOC alerts tab. "Dashboards" seems to work. As I said, I had done this same process on a VM before I tried on our production box and it actually did work? That was SO 2.3.?? (Whatever the last release of 2.3 was) on Ubuntu -> 2.4. 40 on Oracle (the SO iso).

For whatever reason, fleet did not install by default in 2.4 so I had to run the install manually, and all the extra indexes and such were created. For more details on my upgrade woes look at: https://github.com/Security-Onion-Solutions/securityonion/discussions/12531