How do I bring back the Sankey Diagram?

[UMHB-InfoSec]

SO 1.x had squert and its sankey diagram. While not perfect, it did make an excellent visual aid for me to explain what is going on on the network to my management team. But I can't find it in SO 2.x.

I know squert is gone, but according to my google-fu kibana also has the ability to generate sankey diagrams. However, when I try to follow any documentation I can find online the sankey option is missing in kibana. Am I doing something wrong? How can I install / activate / whatever the sankey option?

[weslambert]

You can create a Vega visualization in Kibana. Here's some code I slightly updated (for SO2) from @dlee35 's work on a Sankey diagram for 16.04:

{
  $schema: https://vega.github.io/schema/vega/v4.json
  data: [
    {
      name: rawData
      url: {
        %context%: true
        %timefield%: @timestamp
        index: *:so-*
        body: {
          size: 0
          aggs: {
            table: {
              composite: {
                size: 10000
                sources: [
                  {
                    stk1: {
                      terms: {
                        field: source.ip
                      }
                    }
                  }
                  {
                    stk2: {
                      terms: {
                        field: destination.ip
                      }
                    }
                  }
                ]
              }
            }
          }
        }
      }
      format: {
        property: aggregations.table.buckets
      }
      transform: [
        {
          type: formula
          expr: datum.key.stk1
          as: stk1
        }
        {
          type: formula
          expr: datum.key.stk2
          as: stk2
        }
        {
          type: formula
          expr: datum.doc_count
          as: size
        }
      ]
    }
    {
      name: nodes
      source: rawData
      transform: [
        {
          type: filter
          expr: !groupSelector || groupSelector.stk1 == datum.stk1 || groupSelector.stk2 == datum.stk2
        }
        {
          type: formula
          expr: datum.stk1+datum.stk2
          as: key
        }
        {
          type: fold
          fields: [
            stk1
            stk2
          ]
          as: [
            stack
            grpId
          ]
        }
        {
          type: formula
          expr: datum.stack == 'stk1' ? datum.stk1+' '+datum.stk2 : datum.stk2+' '+datum.stk1
          as: sortField
        }
        {
          type: stack
          groupby: [
            stack
          ]
          sort: {
            field: sortField
            order: descending
          }
          field: size
        }
        {
          type: formula
          expr: (datum.y0+datum.y1)/2
          as: yc
        }
      ]
    }
    {
      name: groups
      source: nodes
      transform: [
        {
          type: aggregate
          groupby: [
            stack
            grpId
          ]
          fields: [
            size
          ]
          ops: [
            sum
          ]
          as: [
            total
          ]
        }
        {
          type: stack
          groupby: [
            stack
          ]
          sort: {
            field: grpId
            order: descending
          }
          field: total
        }
        {
          type: formula
          expr: scale('y', datum.y0)
          as: scaledY0
        }
        {
          type: formula
          expr: scale('y', datum.y1)
          as: scaledY1
        }
        {
          type: formula
          expr: datum.stack == 'stk1'
          as: rightLabel
        }
        {
          type: formula
          expr: datum.total/domain('y')[1]
          as: percentage
        }
      ]
    }
    {
      name: destinationNodes
      source: nodes
      transform: [
        {
          type: filter
          expr: datum.stack == 'stk2'
        }
      ]
    }
    {
      name: edges
      source: nodes
      transform: [
        {
          type: filter
          expr: datum.stack == 'stk1'
        }
        {
          type: lookup
          from: destinationNodes
          key: key
          fields: [
            key
          ]
          as: [
            target
          ]
        }
        {
          type: linkpath
          orient: horizontal
          shape: diagonal
          sourceY: {
            expr: scale('y', datum.yc)
          }
          sourceX: {
            expr: scale('x', 'stk1') + bandwidth('x')
          }
          targetY: {
            expr: scale('y', datum.target.yc)
          }
          targetX: {
            expr: scale('x', 'stk2')
          }
        }
        {
          type: formula
          expr: range('y')[0]-scale('y', datum.size)
          as: strokeWidth
        }
        {
          type: formula
          expr: datum.size/domain('y')[1]
          as: percentage
        }
      ]
    }
  ]
  scales: [
    {
      name: x
      type: band
      range: width
      domain: [
        stk1
        stk2
      ]
      paddingOuter: 0.05
      paddingInner: 0.95
    }
    {
      name: y
      type: linear
      range: height
      domain: {
        data: nodes
        field: y1
      }
    }
    {
      name: color
      type: ordinal
      range: category
      domain: {
        data: rawData
        fields: [
          stk1
          stk2
        ]
      }
    }
    {
      name: stackNames
      type: ordinal
      range: [
        Source
        Destination
      ]
      domain: [
        stk1
        stk2
      ]
    }
  ]
  axes: [
    {
      orient: bottom
      scale: x
      encode: {
        labels: {
          update: {
            text: {
              scale: stackNames
              field: value
            }
          }
        }
      }
    }
  ]
  marks: [
    {
      type: path
      name: edgeMark
      from: {
        data: edges
      }
      clip: true
      encode: {
        update: {
          stroke: [
            {
              test: groupSelector && groupSelector.stack=='stk1'
              scale: color
              field: stk2
            }
            {
              scale: color
              field: stk1
            }
          ]
          strokeWidth: {
            field: strokeWidth
          }
          path: {
            field: path
          }
          strokeOpacity: {
            signal: !groupSelector && (groupHover.stk1 == datum.stk1 || groupHover.stk2 == datum.stk2) ? 0.9 : 0.3
          }
          zindex: {
            signal: !groupSelector && (groupHover.stk1 == datum.stk1 || groupHover.stk2 == datum.stk2) ? 1 : 0
          }
          tooltip: {
            signal: datum.stk1 + ' → ' + datum.stk2 + '    ' + format(datum.size, ',.0f') + '   (' + format(datum.percentage, '.1%') + ')'
          }
        }
        hover: {
          strokeOpacity: {
            value: 1
          }
        }
      }
    }
    {
      type: rect
      name: groupMark
      from: {
        data: groups
      }
      encode: {
        enter: {
          fill: {
            scale: color
            field: grpId
          }
          width: {
            scale: x
            band: 1
          }
        }
        update: {
          x: {
            scale: x
            field: stack
          }
          y: {
            field: scaledY0
          }
          y2: {
            field: scaledY1
          }
          fillOpacity: {
            value: 0.6
          }
          tooltip: {
            signal: datum.grpId + '   ' + format(datum.total, ',.0f') + '   (' + format(datum.percentage, '.1%') + ')'
          }
        }
        hover: {
          fillOpacity: {
            value: 1
          }
        }
      }
    }
    {
      type: text
      from: {
        data: groups
      }
      interactive: false
      encode: {
        update: {
          x: {
            signal: scale('x', datum.stack) + (datum.rightLabel ? bandwidth('x') + 8 : -8)
          }
          yc: {
            signal: (datum.scaledY0 + datum.scaledY1)/2
          }
          align: {
            signal: datum.rightLabel ? 'left' : 'right'
          }
          baseline: {
            value: middle
          }
          fontWeight: {
            value: bold
          }
          text: {
            signal: abs(datum.scaledY0-datum.scaledY1) > 13 ? datum.grpId : ''
          }
        }
      }
    }
    {
      type: group
      data: [
        {
          name: dataForShowAll
          values: [
            {
            }
          ]
          transform: [
            {
              type: filter
              expr: groupSelector
            }
          ]
        }
      ]
      encode: {
        enter: {
          xc: {
            signal: width/2
          }
          y: {
            value: 30
          }
          width: {
            value: 80
          }
          height: {
            value: 30
          }
        }
      }
      marks: [
        {
          type: group
          name: groupReset
          from: {
            data: dataForShowAll
          }
          encode: {
            enter: {
              cornerRadius: {
                value: 6
              }
              fill: {
                value: "#f5f5f5"
              }
              stroke: {
                value: "#c1c1c1"
              }
              strokeWidth: {
                value: 2
              }
              height: {
                field: {
                  group: height
                }
              }
              width: {
                field: {
                  group: width
                }
              }
            }
            update: {
              opacity: {
                value: 1
              }
            }
            hover: {
              opacity: {
                value: 0.7
              }
            }
          }
          marks: [
            {
              type: text
              interactive: false
              encode: {
                enter: {
                  xc: {
                    field: {
                      group: width
                    }
                    mult: 0.5
                  }
                  yc: {
                    field: {
                      group: height
                    }
                    mult: 0.5
                    offset: 2
                  }
                  align: {
                    value: center
                  }
                  baseline: {
                    value: middle
                  }
                  fontWeight: {
                    value: bold
                  }
                  text: {
                    value: Show All
                  }
                }
              }
            }
          ]
        }
      ]
    }
  ]
  signals: [
    {
      name: groupHover
      value: {
      }
      on: [
        {
          events: @groupMark:mouseover
          update: "{stk1:datum.stack=='stk1' && datum.grpId, stk2:datum.stack=='stk2' && datum.grpId}"
        }
        {
          events: mouseout
          update: "{}"
        }
      ]
    }
    {
      name: groupSelector
      value: false
      on: [
        {
          events: @groupMark:click!
          update: "{stack:datum.stack, stk1:datum.stack=='stk1' && datum.grpId, stk2:datum.stack=='stk2' && datum.grpId}"
        }
        {
          events: [
            {
              type: click
              markname: groupReset
            }
            {
              type: dblclick
            }
          ]
          update: "false"
        }
      ]
    }
  ]
}

This will give you something like the following (snipped):

[UMHB-InfoSec]

@weslambert thank you for that!

I don't know anything about vega but I am willing to learn, any suggestions where to get started?

How hard do you think it would be to get the sankey settings out of squert in SO 16.04 and get them into kibana in SO 2? I know the drill down stuff would not be there, but the overall design on that diagram is what I am looking to reproduce.

A specific limitation of the diagram above that I am trying to find a way around is the number of entries used to generate the graph.

table: {
   composite: {
      size: 10000
         sources: [

From my child-like tinkering so far the 'size' setting seems to be the number of log entries ES will return to create the data set for the graph. Ideally that number would be all the entries in the desired time period but if I change it to anything much higher it just generates an error.

I suspect there is some max value for 'size' but I haven't been able to find it in the vega docs, but then again ... I don't know enough yet to even read the docs effectively.

I know in the larger scheme of things this seems like a low-priority item, and it probably is. But the diagram in squert was an excellent communication tool for me to show my management team.

"That big red line, yeah that was where the student tried to port scan our web server. We talked to him, he says he's sorry."

It was also really cool to see during our last internal pen-test, the attacker lit up like a Christmas tree!

Would Security Onion Solutions be interested in allowing me to sponsor the development of this feature?

[dougburks]

I happened to run across this old discussion and thought I'd update it for anybody else who happens to find it later. Security Onion 2.3.130 added a new Dashboards app which includes many prebuilt dashboards with Sankey diagrams. It also makes it super easy for you to build your own Sankey diagrams!
https://blog.securityonion.net/2022/06/security-onion-23130-now-available.html
https://docs.securityonion.net/en/2.3/dashboards.html