add new Capture-ID message consolidation feature

[danieldonoghue]

This change allows the UI to group and filter the Flow tab such that all messages with the same fingerprint but from different Capture IDs are presented as one (with a drop down to access the individual messages).

This provides for clarity and a simplified sequence chart as multiple capture IDs merely indicate multiple capture clients seeing the same message and not message duplication that requires investigation.

To use this feature, you need to generate a fingerprint for your messages. This can be done using lua in heplify-server:

function checkRAW()
	local protoType = GetHEPProtoType()

	-- Check if we have SIP type 
	if protoType ~= 1 then
		return
	end

	-- original SIP message Payload
	local raw = GetRawMessage()

	-- get source and destination from HEP protocol
	local srcIP, srcPort = GetHEPSrcIP(), GetHEPSrcPort()
	local dstIP, dstPort = GetHEPDstIP(), GetHEPDstPort()

	-- create a fingerprint for the sip message
	local hashBody = string.format("%s:%s-%s:%s-%s", srcIP, srcPort, dstIP, dstPort, raw)
	local sum = HashString("sha256", hashBody)
	SetSIPHeader("fingerprint", sum)

	return
end

the feature will only be available in the filter tab if the data in the flow tab actually contains a fingerprint.

[CLAassistant]

All committers have signed the CLA.

[lmangani]

Interesting idea @danieldonoghue thanks for proposing it! I wonder if it's really necessary to calculate all fingerprints rather than doing this only at the API/UI level when fulfilling requests? Essentially why not just calculating fingerprints on served data since for the vast majority of packets a fingerprint won't be used?

[danieldonoghue]

Interesting idea @danieldonoghue thanks for proposing it! I wonder if it's really necessary to calculate all fingerprints rather than doing this only at the API/UI level when fulfilling requests? Essentially why not just calculating fingerprints on served data since for the vast majority of packets a fingerprint won't be used?

hi @lmangani,

I had a couple of thoughts on that..
in the first instance, that was exactly what I wanted to do but I figured this was more configurable, in that you could have a fingerprint for any message type you like, including custom traffic.

doing it this way also meant I could keep the code change entirely contained to the UI and not have to worry about changing startup configuration or the api in any way.

of course, there's nothing to stop somebody iterating on this later, if they want to.

I can say that this feature has already helped me in my daily work to notice a few irregularities in some of our SIP traffic :)

[danieldonoghue]

oops, nearly forgot, having the fingerprint as a field in the message details is also helpful for debugging purposes :)

and doing it as part of the API request would slow down request handling as it would have to be done every time the messages are read, which can be often if you're doing different queries on the data.