Skip to content

Commit c70d29b

Browse files
jpinsonneaujpinsonneau
authored
NETOBSERV-1214 & NETOBSERV-1399 Manage cluster name and zones (#462)
* Manage cluster name and zones * feature filtering from config
1 parent a34b76c commit c70d29b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+804
-140
lines changed

‎config/sample-config.yaml‎

Lines changed: 59 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,16 +14,20 @@ loki:
1414
tenantID: netobserv
1515
authCheck: none
1616
useMocks: false
17-
frontend:
17+
frontend:
1818
recordTypes:
1919
- flowLog
2020
# - newConnection
2121
# - heartbeat
2222
# - endConnection
2323
features:
24+
# eBPF agent features
2425
# - pktDrop
2526
# - dnsTracking
2627
# - flowRTT
28+
# processor features
29+
# - multiCluster
30+
# - zones
2731
portNaming:
2832
enable: true
2933
portNames:
@@ -55,6 +59,14 @@ frontend:
5559
filter: id
5660
default: true
5761
width: 15
62+
- id: ClusterName
63+
name: Cluster
64+
tooltip: The cluster ID or Name.
65+
field: K8S_ClusterName
66+
filter: cluster_name
67+
default: false
68+
width: 15
69+
feature: multiCluster
5870
- id: SrcK8S_Name
5971
group: Source
6072
name: Name
@@ -167,6 +179,14 @@ frontend:
167179
calculated: getConcatenatedValue(SrcAddr,SrcPort)
168180
default: false
169181
width: 15
182+
- id: SrcZone
183+
group: Source
184+
name: Zone
185+
field: SrcK8S_Zone
186+
filter: src_zone
187+
default: false
188+
width: 15
189+
feature: zones
170190
- id: DstK8S_Name
171191
group: Destination
172192
name: Name
@@ -279,6 +299,14 @@ frontend:
279299
calculated: getConcatenatedValue(DstAddr,DstPort)
280300
default: false
281301
width: 15
302+
- id: DstZone
303+
group: Destination
304+
name: Zone
305+
field: DstK8S_Zone
306+
filter: dst_zone
307+
default: false
308+
width: 15
309+
feature: zones
282310
- id: K8S_Name
283311
name: Names
284312
calculated: getSrcOrDstValue(SrcK8S_Name,DstK8S_Name)
@@ -339,6 +367,12 @@ frontend:
339367
calculated: '[column.SrcK8S_OwnerObject,column.DstK8S_OwnerObject]'
340368
default: false
341369
width: 15
370+
- id: K8S_FlowLayer
371+
name: Flow layer
372+
field: K8S_FlowLayer
373+
filter: flow_layer
374+
default: false
375+
width: 15
342376
- id: AddrPort
343377
name: IPs & Ports
344378
calculated: '[column.SrcAddrPort,column.DstAddrPort]'
@@ -440,6 +474,7 @@ frontend:
440474
filter: dns_id
441475
default: false
442476
width: 5
477+
feature: dnsTracking
443478
- id: DNSLatency
444479
group: DNS
445480
name: DNS Latency
@@ -448,6 +483,7 @@ frontend:
448483
filter: dns_latency
449484
default: true
450485
width: 5
486+
feature: dnsTracking
451487
- id: DNSResponseCode
452488
group: DNS
453489
name: DNS Response Code
@@ -456,6 +492,7 @@ frontend:
456492
filter: dns_flag_response_code
457493
default: true
458494
width: 5
495+
feature: dnsTracking
459496
- id: DNSErrNo
460497
group: DNS
461498
name: DNS Error
@@ -464,14 +501,20 @@ frontend:
464501
filter: dns_errno
465502
default: false
466503
width: 5
504+
feature: dnsTracking
467505
- id: TimeFlowRttMs
468506
name: Flow RTT
469507
tooltip: TCP handshake Round Trip Time
470508
field: TimeFlowRttNs
471509
filter: time_flow_rtt
472510
default: true
473511
width: 5
512+
feature: flowRTT
474513
filters:
514+
- id: cluster_name
515+
name: Cluster
516+
component: autocomplete
517+
hint: Specify a cluster ID or name.
475518
- id: src_namespace
476519
name: Namespace
477520
component: autocomplete
@@ -576,6 +619,16 @@ frontend:
576619
- Starting text like cluster, "cluster-*"
577620
- Ending text like "*-registry"
578621
- Pattern like "cluster-*-registry", "c*-*-r*y", -i*e-
622+
- id: src_zone
623+
name: Zone
624+
component: autocomplete
625+
category: source
626+
hint: Specify a single zone.
627+
- id: dst_zone
628+
name: Zone
629+
component: autocomplete
630+
category: destination
631+
hint: Specify a single zone.
579632
- id: src_resource
580633
name: Resource
581634
component: autocomplete
@@ -739,6 +792,11 @@ frontend:
739792
component: autocomplete
740793
placeholder: 'E.g: Ingress, Egress, Inner'
741794
hint: Specify the direction of the Flow observed at the Node observation point.
795+
- id: flow_layer
796+
name: Flow layer
797+
component: text
798+
placeholder: 'Either infra or app'
799+
hint: Specify the layer of Flow.
742800
- id: interface
743801
name: Network interface
744802
component: text

‎mocks/updateMocks.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@ curl 'http://localhost:3100/loki/api/v1/query_range?query=\{app=%22netobserv-flo
1717
echo 'Getting metrics'
1818
curl 'http://localhost:3100/loki/api/v1/query_range?query=topk(5,sum%20by(app)%20(rate(\{app=%22netobserv-flowcollector%22,FlowDirection=%221%22\}|~`Duplicate%22:false`|json|unwrap%20Packets|__error__=%22%22\[720s\])))&limit=5&step=360s'\
1919
| jq > ./loki/flow_metrics_app.json
20+
curl 'http://localhost:3100/loki/api/v1/query_range?query=topk(50,sum%20by(K8S_ClusterName)%20(rate(\{app=%22netobserv-flowcollector%22,FlowDirection=%221%22\}|~`Duplicate%22:false`|json|unwrap%20Packets|__error__=%22%22\[720s\])))&limit=50&step=360s'\
21+
| jq > ./loki/flow_metrics_cluster.json
22+
curl 'http://localhost:3100/loki/api/v1/query_range?query=topk(50,sum%20by(SrcK8S_Zone,DstK8S_Zone)%20(rate(\{app=%22netobserv-flowcollector%22,FlowDirection=%221%22\}|~`Duplicate%22:false`|json|unwrap%20Packets|__error__=%22%22\[720s\])))&limit=50&step=360s'\
23+
| jq > ./loki/flow_metrics_zone.json
2024
curl 'http://localhost:3100/loki/api/v1/query_range?query=topk(50,sum%20by(SrcK8S_HostName,DstK8S_HostName)%20(rate(\{app=%22netobserv-flowcollector%22,FlowDirection=%221%22\}|~`Duplicate%22:false`|json|unwrap%20Packets|__error__=%22%22\[720s\])))&limit=50&step=360s'\
2125
| jq > ./loki/flow_metrics_host.json
2226
curl 'http://localhost:3100/loki/api/v1/query_range?query=topk(50,sum%20by(SrcK8S_Namespace,DstK8S_Namespace)%20(rate(\{app=%22netobserv-flowcollector%22,FlowDirection=%221%22\}|~`Duplicate%22:false`|json|unwrap%20Packets|__error__=%22%22\[720s\])))&limit=50&step=360s'\

‎pkg/handler/config.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ type Column struct {
5555
Filter string `yaml:"filter,omitempty" json:"filter,omitempty"`
5656
Default bool `yaml:"default,omitempty" json:"default,omitempty"`
5757
Width int `yaml:"width,omitempty" json:"width,omitempty"`
58+
Feature string `yaml:"feature" json:"feature"`
5859
}
5960

6061
type Filter struct {

‎pkg/handler/lokiclientmock/loki_client_mock.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,10 @@ func (o *LokiClientMock) Get(url string) ([]byte, int, error) {
3333
path += "_state.json"
3434
} else if strings.Contains(url, "by(PktDropLatestDropCause)") {
3535
path += "_cause.json"
36+
} else if strings.Contains(url, "by(K8S_ClusterName)") {
37+
path += "_cluster.json"
38+
} else if strings.Contains(url, "by(SrcK8S_Zone,DstK8S_Zone)") {
39+
path += "zone.json"
3640
} else if strings.Contains(url, "by(SrcK8S_HostName,DstK8S_HostName)") {
3741
path += "_host.json"
3842
} else if strings.Contains(url, "by(SrcK8S_Namespace,DstK8S_Namespace)") {

‎pkg/handler/resources.go‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,59 @@ import (
1919
"github.com/netobserv/network-observability-console-plugin/pkg/utils"
2020
)
2121

22+
func GetClusters(cfg *loki.Config) func(w http.ResponseWriter, r *http.Request) {
23+
return func(w http.ResponseWriter, r *http.Request) {
24+
lokiClient := newLokiClient(cfg, r.Header, false)
25+
var code int
26+
startTime := time.Now()
27+
defer func() {
28+
metrics.ObserveHTTPCall("GetClusters", code, startTime)
29+
}()
30+
31+
// Fetch and merge values for K8S_ClusterName
32+
values, code, err := getLabelValues(cfg, lokiClient, fields.Cluster)
33+
if err != nil {
34+
writeError(w, code, "Error while fetching label cluster values from Loki: "+err.Error())
35+
return
36+
}
37+
38+
code = http.StatusOK
39+
writeJSON(w, code, utils.NonEmpty(utils.Dedup(values)))
40+
}
41+
}
42+
43+
func GetZones(cfg *loki.Config) func(w http.ResponseWriter, r *http.Request) {
44+
return func(w http.ResponseWriter, r *http.Request) {
45+
lokiClient := newLokiClient(cfg, r.Header, false)
46+
var code int
47+
startTime := time.Now()
48+
defer func() {
49+
metrics.ObserveHTTPCall("GetZones", code, startTime)
50+
}()
51+
52+
// Initialize values explicitly to avoid null json when empty
53+
values := []string{}
54+
55+
// Fetch and merge values for SrcK8S_Zone and DstK8S_Zone
56+
values1, code, err := getLabelValues(cfg, lokiClient, fields.SrcZone)
57+
if err != nil {
58+
writeError(w, code, "Error while fetching label source zone values from Loki: "+err.Error())
59+
return
60+
}
61+
values = append(values, values1...)
62+
63+
values2, code, err := getLabelValues(cfg, lokiClient, fields.DstZone)
64+
if err != nil {
65+
writeError(w, code, "Error while fetching label destination zone values from Loki: "+err.Error())
66+
return
67+
}
68+
values = append(values, values2...)
69+
70+
code = http.StatusOK
71+
writeJSON(w, code, utils.NonEmpty(utils.Dedup(values)))
72+
}
73+
}
74+
2275
func GetNamespaces(cfg *loki.Config) func(w http.ResponseWriter, r *http.Request) {
2376
return func(w http.ResponseWriter, r *http.Request) {
2477
lokiClient := newLokiClient(cfg, r.Header, false)

‎pkg/loki/topology_query.go‎

Lines changed: 40 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -77,28 +77,20 @@ func NewTopologyQuery(cfg *Config, start, end, limit, rateInterval, step string,
7777
}, nil
7878
}
7979

80-
func getLabels(aggregate, groups string) string {
81-
var fields []string
82-
switch aggregate {
83-
case "app":
84-
fields = []string{"app"}
85-
case "droppedState":
86-
fields = []string{"PktDropLatestState"}
87-
case "droppedCause":
88-
fields = []string{"PktDropLatestDropCause"}
89-
case "dnsRCode":
90-
fields = []string{"DnsFlagsResponseCode"}
91-
case "host":
92-
fields = []string{"SrcK8S_HostName", "DstK8S_HostName"}
93-
case "namespace":
94-
fields = []string{"SrcK8S_Namespace", "DstK8S_Namespace"}
95-
case "owner":
96-
fields = []string{"SrcK8S_OwnerName", "SrcK8S_OwnerType", "DstK8S_OwnerName", "DstK8S_OwnerType", "SrcK8S_Namespace", "DstK8S_Namespace"}
97-
default:
98-
fields = []string{"SrcK8S_Name", "SrcK8S_Type", "SrcK8S_OwnerName", "SrcK8S_OwnerType", "SrcK8S_Namespace", "SrcAddr", "SrcK8S_HostName", "DstK8S_Name", "DstK8S_Type", "DstK8S_OwnerName", "DstK8S_OwnerType", "DstK8S_Namespace", "DstAddr", "DstK8S_HostName"}
99-
}
100-
80+
func manageGroupLabels(fields []string, groups string) []string {
10181
if len(groups) > 0 {
82+
if strings.Contains(groups, "clusters") {
83+
if !utils.Contains(fields, "K8S_ClusterName") {
84+
fields = append(fields, "K8S_ClusterName")
85+
}
86+
}
87+
88+
if strings.Contains(groups, "zones") {
89+
if !utils.Contains(fields, "SrcK8S_Zone") {
90+
fields = append(fields, "SrcK8S_Zone", "DstK8S_Zone")
91+
}
92+
}
93+
10294
if strings.Contains(groups, "hosts") {
10395
if !utils.Contains(fields, "SrcK8S_HostName") {
10496
fields = append(fields, "SrcK8S_HostName", "DstK8S_HostName")
@@ -117,7 +109,34 @@ func getLabels(aggregate, groups string) string {
117109
}
118110
}
119111
}
112+
return fields
113+
}
120114

115+
func getLabels(aggregate, groups string) string {
116+
var fields []string
117+
switch aggregate {
118+
case "app":
119+
fields = []string{"app"}
120+
case "droppedState":
121+
fields = []string{"PktDropLatestState"}
122+
case "droppedCause":
123+
fields = []string{"PktDropLatestDropCause"}
124+
case "dnsRCode":
125+
fields = []string{"DnsFlagsResponseCode"}
126+
case "cluster":
127+
fields = []string{"K8S_ClusterName"}
128+
case "zone":
129+
fields = []string{"SrcK8S_Zone", "DstK8S_Zone"}
130+
case "host":
131+
fields = []string{"SrcK8S_HostName", "DstK8S_HostName"}
132+
case "namespace":
133+
fields = []string{"SrcK8S_Namespace", "DstK8S_Namespace"}
134+
case "owner":
135+
fields = []string{"SrcK8S_OwnerName", "SrcK8S_OwnerType", "DstK8S_OwnerName", "DstK8S_OwnerType", "SrcK8S_Namespace", "DstK8S_Namespace"}
136+
default:
137+
fields = []string{"SrcK8S_Name", "SrcK8S_Type", "SrcK8S_OwnerName", "SrcK8S_OwnerType", "SrcK8S_Namespace", "SrcAddr", "SrcK8S_HostName", "DstK8S_Name", "DstK8S_Type", "DstK8S_OwnerName", "DstK8S_OwnerType", "DstK8S_Namespace", "DstAddr", "DstK8S_HostName"}
138+
}
139+
fields = manageGroupLabels(fields, groups)
121140
return strings.Join(fields[:], ",")
122141
}
123142

‎pkg/model/fields/fields.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,10 @@ const (
3030
HostName = "K8S_HostName"
3131
SrcHostName = Src + HostName
3232
DstHostName = Dst + HostName
33+
Zone = "K8S_Zone"
34+
SrcZone = Src + Zone
35+
DstZone = Dst + Zone
36+
Cluster = "K8S_ClusterName"
3337
Packets = "Packets"
3438
PktDropPackets = "PktDropPackets"
3539
Proto = "Proto"

‎pkg/server/routes.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,8 @@ func setupRoutes(cfg *Config, authChecker auth.Checker) *mux.Router {
3737
api.HandleFunc("/loki/flow/records", handler.GetFlows(&cfg.Loki))
3838
api.HandleFunc("/loki/flow/metrics", handler.GetTopology(&cfg.Loki))
3939
api.HandleFunc("/loki/export", handler.ExportFlows(&cfg.Loki))
40+
api.HandleFunc("/resources/clusters", handler.GetClusters(&cfg.Loki))
41+
api.HandleFunc("/resources/zones", handler.GetZones(&cfg.Loki))
4042
api.HandleFunc("/resources/namespaces", handler.GetNamespaces(&cfg.Loki))
4143
api.HandleFunc("/resources/namespace/{namespace}/kind/{kind}/names", handler.GetNames(&cfg.Loki))
4244
api.HandleFunc("/resources/kind/{kind}/names", handler.GetNames(&cfg.Loki))

‎web/locales/en/plugin__netobserv-plugin.json‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,20 @@
55
"n/a": "n/a",
66
"View alert details": "View alert details",
77
"View health dashboard": "View health dashboard",
8-
"Namespaces + Owners": "Namespaces + Owners",
8+
"Clusters": "Clusters",
9+
"Clusters + Nodes": "Clusters + Nodes",
10+
"Clusters + Zones": "Clusters + Zones",
11+
"Clusters + Namespaces": "Clusters + Namespaces",
12+
"Clusters + Owners": "Clusters + Owners",
13+
"Zones": "Zones",
14+
"Zones + Nodes": "Zones + Nodes",
15+
"Zones + Namespaces": "Zones + Namespaces",
16+
"Zones + Owners": "Zones + Owners",
17+
"Nodes": "Nodes",
918
"Nodes + Namespaces": "Nodes + Namespaces",
1019
"Nodes + Owners": "Nodes + Owners",
11-
"Nodes": "Nodes",
1220
"Namespaces": "Namespaces",
21+
"Namespaces + Owners": "Namespaces + Owners",
1322
"Owners": "Owners",
1423
"None": "None",
1524
"3D": "3D",
@@ -82,6 +91,8 @@
8291
"1 hour": "1 hour",
8392
"2 hours": "2 hours",
8493
"1 day": "1 day",
94+
"Cluster": "Cluster",
95+
"Zone": "Zone",
8596
"Node": "Node",
8697
"Namespace": "Namespace",
8798
"Owner": "Owner",
@@ -269,6 +280,7 @@
269280
"Unable to get flows": "Unable to get flows",
270281
"Step into this {{name}}": "Step into this {{name}}",
271282
"Filter by source or destination {{name}}": "Filter by source or destination {{name}}",
283+
"Filter by {{name}}": "Filter by {{name}}",
272284
"Unpin this element": "Unpin this element",
273285
"Pin this element": "Pin this element",
274286
"Name": "Name",
@@ -287,6 +299,7 @@
287299
"Average rate": "Average rate",
288300
"Latest time": "Latest time",
289301
"Latest rate": "Latest rate",
302+
"Cluster name": "Cluster name",
290303
"Edge": "Edge",
291304
"Drops": "Drops",
292305
"Unable to get topology": "Unable to get topology",

0 commit comments

Comments
 (0)