Possible solution for hierarchical design

[stefan-ma22]

Hi, after last update I would like to point out one possible solution for hierarchical design but I would need your help.
I found really good tutorial here:
https://www.alyne.com/en/blog/tech-talk/using-webcola-and-d3-js-to-create-hierarchical-layout/

Here are the steps I took:
1.Changed diagram.ts
a) import * as _ from 'lodash';
b)

initCola(): any { // eslint-disable-line @typescript-eslint/no-explicit-any return cola.d3adaptor() .avoidOverlaps(true) .handleDisconnected(false) .symmetricDiffLinkLengths(100) .linkDistance(100) .size([this.options.width, this.options.height]); }

c) inside render function I added this
`const levelGroups = _.groupBy(nodes, "level");
console.log(levelGroups)
for (const level of Object.keys(levelGroups)) {
const nodeGroup = levelGroups[level];
const constraint = {
type: "alignment",
axis: "y",
offsets: [],
};
let prevNodeId = "none";
for (const node of nodeGroup) {
constraint.offsets.push({
node: _.findIndex(nodes, (d) => d.name === node.name),
offset: 0,
});

              if (prevNodeId !== "none") {
                console.log(_.findIndex(nodes, (d) => d.name === prevNodeId));
                console.log(_.findIndex(nodes, (d) => d.name === node.name));
                
                constraints.push({
                  axis: "x",
                  left: _.findIndex(nodes, (d) => d.name === prevNodeId),
                  right:_.findIndex(nodes, (d) => d.name === node.name),
                  gap: 50,
                });
              }
      
              prevNodeId = node.name;
              console.log(prevNodeId);
            }
      
            constraints.push(constraint);
        }
        
this.cola.nodes(nodes)
            .links(links)
            **.constraints(constraints)**
            .groups(groups);
        this.setDistance(this.cola);`

d) in node.ts I added level property
in NodeDataType - level: number,
in Node Class - public level: number;
in constructor - this.level = data.level;

e) in package.json in dependencies I added
"@types/lodash": "^4.14.168",
"lodash": "^4.17.20",

JSON here - you can see I added level property:
{ "nodes": [ { "level":1, "name": "testPERMANENT-10.0.1.1", "meta": { "tenant-ip": "10.0.1.1" }, "icon": "/static/images/router.png" }, { "level":1, "name": "testPERMANENT-10.0.1.2", "meta": { "tenant-ip": "10.0.1.2" }, "icon": "/static/images/router.png" }, { "level":2, "name": "INTERNET", "icon": "/static/images/router.png" }, { "level":3, "name": "AZ1-testpermanent-123-muc-vpn1", "meta": { "name": "vpn1" }, "icon": "/static/images/router.png" }, { "level":3, "name": "AZ2-testpermanent-123-muc-vpn2", "meta": { "name": "vpn2" }, "icon": "/static/images/router.png" }, { "level":4, "name": "AZ1-testpermanent-muc-rs1", "meta": { "name": "rs1" }, "icon": "/static/images/router.png" }, { "level":4, "name": "AZ1-testpermanent-muc-rs2", "meta": { "name": "rs2" }, "icon": "/static/images/router.png" }, { "level":4, "name": "AZ2-testpermanent-muc-rs3", "meta": { "name": "rs3" }, "icon": "/static/images/router.png" }, { "level":4, "name": "AZ2-testpermanent-muc-rs4", "meta": { "name": "rs4" }, "icon": "/static/images/router.png" }, { "level":5, "name": "10.0.0.0/24" }, { "level":5, "name": "10.0.1.0/24" }, { "level":6, "name": "POD1-FCIPOCLEAF101", "meta": { "name": "FCIPOCLEAF101" }, "icon": "/static/images/router.png" }, { "level":7, "name": "POD1-FCIPOCLEAF102", "meta": { "name": "FCIPOCLEAF102" }, "icon": "/static/images/router.png" }, { "level":6, "name": "POD2-FCIPOCLEAF202", "meta": { "name": "FCIPOCLEAF202" }, "icon": "/static/images/router.png" }, { "level":7, "name": "POD2-FCIPOCLEAF201", "meta": { "name": "FCIPOCLEAF201" }, "icon": "/static/images/router.png" } ], "links": [ { "source": "10.0.0.0/24", "target": "AZ1-testpermanent-muc-rs1" }, { "source": "10.0.0.0/24", "target": "AZ1-testpermanent-muc-rs2" }, { "source": "10.0.0.0/24", "target": "POD1-FCIPOCLEAF102" }, { "source": "10.0.0.0/24", "target": "POD1-FCIPOCLEAF101" }, { "source": "10.0.0.0/24", "target": "POD2-FCIPOCLEAF201" }, { "source": "10.0.0.0/24", "target": "POD2-FCIPOCLEAF202" }, { "source": "10.0.1.0/24", "target": "AZ2-testpermanent-muc-rs3" }, { "source": "10.0.1.0/24", "target": "AZ2-testpermanent-muc-rs4" }, { "source": "10.0.1.0/24", "target": "POD1-FCIPOCLEAF102" }, { "source": "10.0.1.0/24", "target": "POD1-FCIPOCLEAF101" }, { "source": "10.0.1.0/24", "target": "POD2-FCIPOCLEAF201" }, { "source": "10.0.1.0/24", "target": "POD2-FCIPOCLEAF202" }, { "source": "INTERNET", "target": "AZ1-testpermanent-123-muc-vpn1" }, { "source": "INTERNET", "target": "AZ2-testpermanent-123-muc-vpn2" }, { "source": "INTERNET", "target": "testPERMANENT-10.0.1.1" }, { "source": "INTERNET", "target": "testPERMANENT-10.0.1.2" }, { "source": "10.0.0.0/24", "target": "AZ1-testpermanent-123-muc-vpn1" }, { "source": "10.0.1.0/24", "target": "AZ2-testpermanent-123-muc-vpn2" } ] }

I think it works but not 100% because in the link I provided the generated graph is topdown but for example here LEVEL 1 is displayed at the bottom. And for example if I have groups, one node is displayed at the same level even though its one level lower. So I think it has to do something with groups and constraints.

JSON to image below:
vpns are level 1
internet is level 2
etc. but if you look at LEAFS which are aligned with RS this should not be like that because they have different levels but grouping is a problem maybe

you can see without groups it works better but still not great

[stefan-ma22]

Added also step where nodes are sorted by levels when initiating simulation in diagram.ts

let preSortedNodes = data.nodes ? _.sortBy(data.nodes, "level") : [];
const nodes = preSortedNodes.map((n, i) => new Node(n, i, this.options.meta, this.options.color, this.options.tooltip !== undefined));

[stefan-ma22]

Further testing is showing me, its not reliable that the topology will show top down every time ... just small change will result in complete rebuild of topology, totally ignoring top down layout and inverting it or moving left to right, etc.

[codeout]

Thanks for the steps explained. Let me try to reproduce in my environment and investigate why inet-henge ignores levels defined in json.

[stefan-ma22]

Thanks for the steps explained. Let me try to reproduce in my environment and investigate why inet-henge ignores levels defined in json.

Thanks, its not ignoring levels completely because If you drag one node, you are automatically moving all other nodes at that level, levels are just not aligned vertically. I tried different approaches and not sure what is causing it (maybe translate function ?) because I dont understand what can be difference between the approach I posted in link and my adjusted changed.
E.g. I added new node above VPN nodes with LEVEL 5 all the way at the top and it messed up completely and showed first half of levels at the bottom.
Keep me updated, I will try work on this too :)

[stefan-ma22]

Any update ?
I was looking at this, tried many different things and I found one solution. SetCola js
Look at this - https://uwdata.github.io/setcola/ and choose kruger-foodWeb its perfectly aligned and top down and

I have created CodePen with WebCola + SetCola here https://codepen.io/marhyno/pen/xxRrLYZ (actual code start from LINE 782) and its working perfectly fine items are aligned top down and next to each other and you can create connections between the lowest level to the top without topology to change design. Can you look at it ? :)

[codeout]

I got the topology you mentioned finally.

source code: 598ba2f...4850f7f

A problem I found is that some of the constraints don't work, even they look valid to me. I'm looking into webcola now.

0: {type: "alignment", axis: "y", offsets: Array(2)}
1: {type: "alignment", axis: "y", offsets: Array(1)}
2: {type: "alignment", axis: "y", offsets: Array(2)}
3: {type: "alignment", axis: "y", offsets: Array(4)}
4: {type: "alignment", axis: "y", offsets: Array(2)}
5: {type: "alignment", axis: "y", offsets: Array(2)}
6: {type: "alignment", axis: "y", offsets: Array(2)}
7: {axis: "x", left: 0, right: 1, gap: 50}
8: {axis: "x", left: 3, right: 4, gap: 50}
9: {axis: "x", left: 5, right: 6, gap: 50}
10: {axis: "x", left: 6, right: 7, gap: 50}
11: {axis: "x", left: 7, right: 8, gap: 50}
12: {axis: "x", left: 9, right: 10, gap: 50}
13: {axis: "x", left: 11, right: 13, gap: 50}
14: {axis: "x", left: 12, right: 14, gap: 50}
15: {axis: "y", left: 0, right: 2, gap: 50, left_name: "testPERMANENT-10.0.1.1", …}
16: {axis: "y", left: 2, right: 3, gap: 50, left_name: "INTERNET", …}
17: {axis: "y", left: 3, right: 5, gap: 50, left_name: "AZ1-testpermanent-123-muc-vpn1", …}

// Something wrong
18: {axis: "y", left: 5, right: 9, gap: 50, left_name: "AZ1-testpermanent-muc-rs1", …}
19: {axis: "y", left: 9, right: 11, gap: 50, left_name: "10.0.0.0/24", …}
20: {axis: "y", left: 11, right: 12, gap: 50, left_name: "POD1-FCIPOCLEAF101", …}

[stefan-ma22]

Hi, thank you very much, we found something like when it reaches certain level e.g. level 5 or 6 it stops using levels, and we dont know why. Bu thanks, maybe setcola would be helpful

[codeout]

So do you mean SetCola ( without inet-henge ) works for you? Or inet-henge can work along with SetCola? I'm just curious.

[stefan-ma22]

Inet henge can work along set cola, if you look at the codepen I created, you can see that WebCola works with SetCola so I think it can work together with inet henge :)

The only thing which might cause problems is the setPosition function and translates, because if you look at the codepen, I did not use translate and nodes are perfectly aligned horizontally and vertically

[codeout]

Sounds great 🎉 But yes, setPosition, translate or something like auto-layout stuff may conflict with SetCola.
I'll read your codepen anyways. Thanks!

[stefan-ma22]

Sounds great 🎉 But yes, setPosition, translate or something like auto-layout stuff may conflict with SetCola.
I'll read your codepen anyways. Thanks!

Thanks, as always I will work on this too and provide results if any :)

[stefan-ma22]

So far I have tried to remove .attr('transform', (d) => d.transform()); from group, node and link
Added setCola to diagram.ts
Added preprocessing by setCola in render function:

	var result = setcola
            .nodes(data.nodes)        // Set the graph nodes
            .links(data.links)       // Set the graph links
            .constraints(data.constraints)  // Set the constraints
            .gap(100)
            .layout();     

            const nodes = result.nodes ?
            result.nodes.map((n, i) => new Node(n, i, this.options.meta, this.options.color, this.options.tooltip !== undefined)) : [];
            const links = result.links ?
            result.links.map((l, i) => new Link(l, i, this.options.meta, this.getLinkWidth)) : [];
            const groups = Group.divide(nodes, this.options.groupPattern, this.options.color);
            const tooltips = nodes.map((n) => new Tooltip(n, this.options.tooltip));

The json I use - added constraints

{
  "nodes": [
  ],
  "links": [
  ],
  "constraints":[
    {
      "name": "leveled",
      "sets": {"partition": "level"},
      "forEach": [
        { "constraint": "order", "axis": "x", "by": "level", "gap": 150 },
        { "constraint": "align", "axis": "x" }
     ]
    },
    {
      "sets": ["leveled"],
      "forEach": [{ 
        "constraint": "order",
        "axis": "y", 
        "by": "level",
        "gap": 100,
        "order": ["carnivore", "herbivore", "plant"]
      }]
    }
  ]
}

But so far the generated topology looks like this

here

My latest update is here in this render part:

• if I use result.nodes (just result from SetCola without sending them to the Node.ts class) paths are rendered correctly but nodes are not on correct place

	this.cola.nodes(result.nodes)
                .links(links)
                .groups(groups)
                .constraints(result.constraints);
            this.setDistance(this.cola);

Here

[stefan-ma22]

Anything new ? :)

[stefan-ma22]

Hi, coming to you with working solution I think :)

After days of digging into cola, webcola, setcola, I finally managed to get it working.
Magic lied in this line - this.cola.start(this.options.initialTicks,0, 0, 0);

The start() method now includes up to three integer arguments. First number will initially apply iterations of layout with no constraints, then iterations with only structural (user-specified) constraints and then iterations of layout with all constraints including anti-overlap constraints. Specifying such a schedule is useful to allow the graph to untangle before making it relatively "rigid" with constraints. So basically if I understand it right, the higher number the more space it occupies without "bumping" to each other

After adding setcola, including constraints into json, topology still looked somehow wrong. I compared it with the example I provided in codepen and found out ticks are the problem. If you leave it like above at 0,0,0 it generates random dynamic view. If you specify higher number, look below I put there 50,50,50 it generates nice looking perfectly aligned horizontal and vertical view. So I think you can include option into Diagram class if user wants to have hierarchical design, he needs to add TRUE so the ticks are different, and in json he must include constraints according to setCola page or look at the example at the bottom of this comment. Also as pointed in the earlier comment node.ts must also include level property to read it from json file. At the end of try catch part there is jQuery loop to remove additional nodes which are created by setcola and for end user are useless so I remove them.

var result = setcola
            .nodes(data.nodes)        // Set the graph nodes
            .links(data.links)       // Set the graph links
            .constraints(data.constraints)  // Set the constraints
            .gap(100)
            .layout();

            const nodes = result.nodes ?
            result.nodes.map((n, i) => new Node(n, i, this.options.meta, this.options.color, this.options.tooltip !== undefined)) : [];

            const links = result.links ?
            result.links.map((l, i) => new Link(l, i, this.options.meta, this.getLinkWidth)) : [];
            
            const groups = Group.divide(nodes, this.options.groupPattern, this.options.color);
            const tooltips = nodes.map((n) => new Tooltip(n, this.options.tooltip));   

            this.cola.nodes(nodes)
                .links(links)
                .groups(groups)
                .constraints(result.constraints);
            this.setDistance(this.cola);

            // Start to update Link.source and Link.target with Node object after
            // initial layout iterations without any constraints.
            this.cola.start(50, 50, 50);

            const groupLayer = this.svg.append('g').attr('id', 'groups');
            const linkLayer = this.svg.append('g').attr('id', 'links');
            const nodeLayer = this.svg.append('g').attr('id', 'nodes');
            const linkLabelLayer = this.svg.append('g').attr('id', 'link-labels');
            const tooltipLayer = this.svg.append('g').attr('id', 'tooltips');

            const [link, path, label] = Link.render(linkLayer, linkLabelLayer, links);

            const group = Group.render(groupLayer, groups).call(
                this.cola.drag()
                    .on('dragstart', this.dragstartCallback)
                    .on('drag', () => {
                        if (this.options.bundle) {
                            Link.shiftBundle(link, path, label);
                        }
                    })
            );

            const node = Node.render(nodeLayer, nodes).call(
                this.cola.drag()
                    .on('dragstart', this.dragstartCallback)
                    .on('drag', () => {
                        if (this.options.bundle) {
                            Link.shiftBundle(link, path, label);
                        }

                        Tooltip.followNode(tooltip);
                    })
            );

            // without path calculation
            this.configureTick(group, node, link);

            this.positionCache = PositionCache.load(data, this.options.groupPattern);
            if (this.options.positionCache && this.positionCache) {
                // NOTE: Evaluate only when positionCache: true or 'fixed', and
                //       when the stored position cache matches pair of given data and pop
                Group.setPosition(group, this.positionCache.group);
                Node.setPosition(node, this.positionCache.node);
                Link.setPosition(link, this.positionCache.link);
            } else {
                this.ticksForward();
                this.positionCache = new PositionCache(data, this.options.groupPattern);
                this.savePosition(group, node, link);
            }

            this.hideLoadMessage();

            // render path
            this.configureTick(group, node, link, path, label);

            if (this.options.bundle) {
                Link.shiftBundle(link, path, label);
            }

            path.attr('d', (d) => d.d()); // make sure path calculation is done
            this.freeze(node);

            const tooltip = Tooltip.render(tooltipLayer, tooltips);

            this.dispatch.rendered();

            // NOTE: This is an experimental option
            if (this.options.positionCache === 'fixed') {
                this.cola.on('end', () => {
                    this.savePosition(group, node, link);
                });
            }

            $('.node.rect').each(function(){
                console.log($(this));
                if ($(this).attr("id").indexOf("_set0_") > -1){
                    $(this).remove();
                }
            });

Add constrains from the code below in your json file which is loaded by inet-henge. These constraints are aligning items from top to bottom based on their level, levels are aligned ascended

{
  "nodes": [
      {
          "level": 4,
          "name": "AZ1-testpermanent-muc-rs1",
          "meta": {
              "name": "rs1",
              "hostname": "testpermanent-muc-rs1"
          },
          "icon": "/static/images/ios-xe.png"
      },
      {
          "level": 5,
          "name": "AZ1-testpermanent-muc-rs2",
          "meta": {
              "name": "rs2",
              "hostname": "testpermanent-muc-rs2"
          },
          "icon": "/static/images/ios-xe.png"
      },
      {
          "level": 4,
          "name": "AZ2-testpermanent-muc-rs3",
          "meta": {
              "name": "rs3",
              "hostname": "testpermanent-muc-rs3"
          },
          "icon": "/static/images/ios-xe.png"
      },
      {
          "level": 5,
          "name": "AZ2-testpermanent-muc-rs4",
          "meta": {
              "name": "rs4",
              "hostname": "testpermanent-muc-rs4"
          },
          "icon": "/static/images/ios-xe.png"
      },
      {
          "level": 8,
          "name": "ACI-FCIPOCLEAF101",
          "group": "POD1",
          "meta": {
              "capabilities": ""
          },
          "icon": "/static/images/nexus9k.png"
      },
      {
          "level": 8,
          "name": "ACI-FCIPOCLEAF102",
          "group": "POD1",
          "meta": {
              "capabilities": ""
          },
          "icon": "/static/images/nexus9k.png"
      },
      {
          "level": 8,
          "name": "ACI-FCIPOCLEAF202",
          "group": "POD2",
          "meta": {
              "capabilities": ""
          },
          "icon": "/static/images/nexus9k.png"
      },
      {
          "level": 8,
          "name": "ACI-FCIPOCLEAF201",
          "group": "POD2",
          "meta": {
              "capabilities": ""
          },
          "icon": "/static/images/nexus9k.png"
      },
      {
          "level": 7,
          "name": "10.0.0.0/24"
      },
      {
          "level": 7,
          "name": "10.0.1.0/24"
      },
      {
          "level": 3,
          "name": "AZ1-testpermanent-123-muc-vpn1",
          "meta": {
              "name": "vpn1",
              "hostname": "testpermanent-123-muc-vpn1"
          },
          "icon": "/static/images/vpn.png"
      },
      {
          "level": 3,
          "name": "AZ2-testpermanent-123-muc-vpn2",
          "meta": {
              "name": "vpn2",
              "hostname": "testpermanent-123-muc-vpn2"
          },
          "icon": "/static/images/vpn.png"
      },
      {
          "level": 1,
          "name": "testPERMANENT-10.0.1.1",
          "meta": {
              "tenant-ip": "10.0.1.1"
          },
          "icon": "/static/images/vpn.png"
      },
      {
          "level": 1,
          "name": "testPERMANENT-10.0.1.2",
          "meta": {
              "tenant-ip": "10.0.1.2"
          },
          "icon": "/static/images/vpn.png"
      },
      {
          "level": 2,
          "name": "INTERNET",
          "icon": "/static/images/internet.png"
      }
  ],
  "links": [
      {
          "source": "10.0.0.0/24",
          "target": "AZ1-testpermanent-muc-rs1"
      },
      {
          "source": "10.0.0.0/24",
          "target": "AZ1-testpermanent-muc-rs2"
      },
      {
          "source": "10.0.0.0/24",
          "target": "ACI-FCIPOCLEAF102"
      },
      {
          "source": "10.0.0.0/24",
          "target": "ACI-FCIPOCLEAF101"
      },
      {
          "source": "10.0.0.0/24",
          "target": "ACI-FCIPOCLEAF201"
      },
      {
          "source": "10.0.0.0/24",
          "target": "ACI-FCIPOCLEAF202"
      },
      {
          "source": "10.0.1.0/24",
          "target": "AZ2-testpermanent-muc-rs3"
      },
      {
          "source": "10.0.1.0/24",
          "target": "AZ2-testpermanent-muc-rs4"
      },
      {
          "source": "10.0.1.0/24",
          "target": "ACI-FCIPOCLEAF102"
      },
      {
          "source": "10.0.1.0/24",
          "target": "ACI-FCIPOCLEAF101"
      },
      {
          "source": "10.0.1.0/24",
          "target": "ACI-FCIPOCLEAF201"
      },
      {
          "source": "10.0.1.0/24",
          "target": "ACI-FCIPOCLEAF202"
      },
      {
          "source": "INTERNET",
          "target": "AZ1-testpermanent-123-muc-vpn1"
      },
      {
          "source": "INTERNET",
          "target": "AZ2-testpermanent-123-muc-vpn2"
      },
      {
          "source": "INTERNET",
          "target": "testPERMANENT-10.0.1.1"
      },
      {
          "source": "INTERNET",
          "target": "testPERMANENT-10.0.1.2"
      },
      {
          "source": "10.0.0.0/24",
          "target": "AZ1-testpermanent-123-muc-vpn1"
      },
      {
          "source": "10.0.1.0/24",
          "target": "AZ2-testpermanent-123-muc-vpn2"
      }
  ],"constraints":[
    {
      "name": "leveled",
      "sets": {"partition": "level"},
      "forEach": [
        { "constraint": "order", "axis": "x", "by": "name", "reverse": true, "gap": 150 },
        { "constraint": "align", "axis": "x", "reverse":true}
     ]
    },
    {
      "sets": ["leveled"],
      "forEach": [{ 
        "constraint": "order",
        "axis": "y", 
        "by": "level",
        "gap": 100,
        "reverse": true
      }]
    }
  ]

}

Really nice looking topology - here

[codeout]

Thanks! This is awesome 🎉

So I think you can include option into Diagram class

Yes, that's the last thing we have to do. I'd like to introduce a switch to plug setCola stuff into the original inet-henge, so that users can choose expected behavior - note that injecting constraints will cause extra CPU load in each calculation tick.

Besides that, we need to examine the magic number 50, 50, 50 is reasonable for other layouts. Or we should make them configurable maybe.