diff --git a/docs/en/api/promql-service.md b/docs/en/api/promql-service.md index 4088ff52d7d0..69cc3966ae46 100644 --- a/docs/en/api/promql-service.md +++ b/docs/en/api/promql-service.md @@ -268,10 +268,31 @@ GET|POST /api/v1/series | match[] | series selector | yes | no | | start | start, format: RFC3399 or unix_timestamp in seconds | yes | no | | end | end, format: RFC3399 or unix_timestamp in seconds | yes | no | +| limit | integer, maximum number of returned series | yes | yes | + +**For metadata metrics**: +**Note: SkyWalking's metadata exists in the following metrics(traffics):** + +| Name | Require Labels | Optional Labels | Support Label Match | +|------------------|----------------|--------------------------|-----------------------------------------------------| +| service_traffic | layer | service, limit | =, (only service label support !=, =~, !~) | +| instance_traffic | layer, service | service_instance, limit | =, (only service_instance label support !=, =~, !~) | +| endpoint_traffic | layer, service | endpoint, keyword, limit | =, (only endpoint label support !=, =~, !~) | + +- **=**: Label value equals the provided string. +- **!=**: Label value does not equal the provided string. +- **=~**: Label value regex-match the provided string. +- **!~**: Label value does not regex-match the provided string + +**If the `limit` is not set by parameter or label, the default value is 100. If the `limit ' is also set in the query parameter, it returns the minimum of the two.** For example: ```text -/api/v1/series?match[]=service_traffic{layer='GENERAL'}&start=1677479336&end=1677479636 +/api/v1/series?match[]=service_traffic{layer='GENERAL'}&start=1677479336&end=1677479636&limit=5 +``` +or +```text +/api/v1/series?match[]=service_traffic{layer='GENERAL', limit='5'}&start=1677479336&end=1677479636 ``` Result: @@ -313,10 +334,23 @@ Result: } ``` -**Note: SkyWalking's metadata exists in the following metrics(traffics):** -- service_traffic -- instance_traffic -- endpoint_traffic +- You can use the `service` label to filter the service_traffic result. +```text +/api/v1/series?match[]=service_traffic{layer='GENERAL', service='agent::songs'}&start=1677479336&end=1677479636 +``` +use regex: +```text +/api/v1/series?match[]=service_traffic{layer='GENERAL', service=~'agent::songs|agent::recommendation'}&start=1677479336&end=1677479636 +``` +- You can use the `service_instance` label to filter the instance_traffic result. +```text +/api/v1/series?match[]=service_traffic{layer='GENERAL', service='agent::songs', service_instance=~'instance1|instance2'}&start=1677479336&end=1677479636 +``` +- You can use the `endpoint` label to filter the endpoint_traffic result. +```text +/api/v1/series?match[]=service_traffic{layer='GENERAL', service='agent::songs', endpoint=~'endpoint1|endpoint2'}&start=1677479336&end=1677479636 +``` + #### Getting label names [Prometheus Docs Reference](https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names) @@ -330,6 +364,7 @@ GET|POST /api/v1/labels | match[] | series selector | yes | yes | | start | start, format: RFC3399 or unix_timestamp in seconds | **no** | yes | | end | end timestamp, if end time is not present, use current time as default end time | yes | yes | +| limit | integer, maximum number of returned labels, default 100 | yes | yes | For example: ```text @@ -363,11 +398,12 @@ GET /api/v1/label//values | match[] | series selector | yes | yes | | start | start, format: RFC3399 or unix_timestamp in seconds | **no** | yes | | end | end, format: RFC3399 or unix_timestamp in seconds, if end time is not present, use current time as default end time | yes | yes | +| limit | integer, maximum number of returned label values, default 100 | yes | yes | For example: ```text /api/v1/label/__name__/values -``` +``` Result: ```json @@ -391,6 +427,39 @@ Result: } ``` +**For metadata metrics**: + +| Name | Require Labels | Optional Labels | Support Label Match | +|------------------|----------------|--------------------------|-----------------------------------------------------| +| service_traffic | layer | service, limit | =, (only service label support !=, =~, !~) | +| instance_traffic | layer, service | service_instance, limit | =, (only service_instance label support !=, =~, !~) | +| endpoint_traffic | layer, service | endpoint, keyword, limit | =, (only endpoint label support !=, =~, !~) | + +- **=**: Label value equal to the provided string. +- **!=**: Label value not equal to the provided string. +- **=~**: Label value regex-match the provided string. +- **!~**: Label value do not regex-match the provided string + +**If the `limit` is not set by parameter or label, the default value is 100. And If the `limit` also set in the query parameter, will return the min number of the two.** + +For example: +- If you want to query the label values of the `service` label in the `service_traffic` metric: +```text +/api/v1/label/service/values?match[]=service_traffic{layer='GENERAL', service='agent::songs|agent::recommendation'}&limit=1 +``` +or +```text +/api/v1/label/service/values?match[]=service_traffic{layer='GENERAL', service='agent::songs|agent::recommendation',limit='1'} +``` +- If you want to query the label values of the `service_instance` label in the `instance_traffic` metric: +```text +/api/v1/label/service_instance/values?match[]=instance_traffic{layer='GENERAL', service='agent::songs', service_instance='instance1|instance2'} +``` +- If you want to query the label values of the `endpoint` label in the `endpoint_traffic` metric: +```text +/api/v1/label/endpoint/values?match[]=endpoint_traffic{layer='GENERAL', service='agent::songs', endpoint='endpoint1|endpoint2'} +``` + #### Querying metric metadata [Prometheus Docs Reference](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata) diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index 0086af48dac2..35aa82b436db 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -7,6 +7,7 @@ * BanyanDB: Support `hot/warm/cold` stages configuration. * Fix query continues profiling policies error when the policy is already in the cache. * Support `hot/warm/cold` stages TTL query in the status API. +* PromQL Service: traffic query support `limit` and regex match. #### UI diff --git a/oap-server/server-query-plugin/promql-plugin/src/main/antlr4/org/apache/skywalking/promql/rt/grammar/PromQLLexer.g4 b/oap-server/server-query-plugin/promql-plugin/src/main/antlr4/org/apache/skywalking/promql/rt/grammar/PromQLLexer.g4 index 5fc2b7c5da37..1fc2f9e2e57c 100644 --- a/oap-server/server-query-plugin/promql-plugin/src/main/antlr4/org/apache/skywalking/promql/rt/grammar/PromQLLexer.g4 +++ b/oap-server/server-query-plugin/promql-plugin/src/main/antlr4/org/apache/skywalking/promql/rt/grammar/PromQLLexer.g4 @@ -32,6 +32,11 @@ L_BRACE: '{'; R_BRACE: '}'; EQ: '='; +//regex-match +RM: '=~'; +//regex-not-match +NRM: '!~'; + // Scalar Binary operators SUB: '-'; ADD: '+'; diff --git a/oap-server/server-query-plugin/promql-plugin/src/main/antlr4/org/apache/skywalking/promql/rt/grammar/PromQLParser.g4 b/oap-server/server-query-plugin/promql-plugin/src/main/antlr4/org/apache/skywalking/promql/rt/grammar/PromQLParser.g4 index 84b743bde053..f81c2431dfb4 100644 --- a/oap-server/server-query-plugin/promql-plugin/src/main/antlr4/org/apache/skywalking/promql/rt/grammar/PromQLParser.g4 +++ b/oap-server/server-query-plugin/promql-plugin/src/main/antlr4/org/apache/skywalking/promql/rt/grammar/PromQLParser.g4 @@ -50,10 +50,12 @@ metricRange: metricInstant L_BRACKET DURATION R_BRACKET; labelName: NAME_STRING; labelValue: VALUE_STRING; -label: labelName EQ labelValue; +label: labelName matchOp labelValue; labelList: label (COMMA label)*; labelNameList: labelName (COMMA labelName)*; +matchOp: EQ | NEQ | RM | NRM; + numberLiteral: NUMBER; badRange: NUMBER L_BRACKET DURATION R_BRACKET; diff --git a/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/entity/response/LabelsQueryRsp.java b/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/entity/response/LabelsQueryRsp.java index edbb05ef41ce..8cb5bf7e533c 100644 --- a/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/entity/response/LabelsQueryRsp.java +++ b/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/entity/response/LabelsQueryRsp.java @@ -26,5 +26,5 @@ @EqualsAndHashCode(callSuper = true) @Data public class LabelsQueryRsp extends QueryResponse { - private final List data = new ArrayList<>(); + private List data = new ArrayList<>(); } diff --git a/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/entity/response/QueryFormatRsp.java b/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/entity/response/QueryFormatRsp.java new file mode 100644 index 000000000000..8847b6fd6634 --- /dev/null +++ b/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/entity/response/QueryFormatRsp.java @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.query.promql.entity.response; + +import lombok.Data; +import lombok.EqualsAndHashCode; + +@EqualsAndHashCode(callSuper = true) +@Data +public class QueryFormatRsp extends QueryResponse { + private String data; +} diff --git a/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/handler/PromQLApiHandler.java b/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/handler/PromQLApiHandler.java index d6fccf0a7096..203ede81054b 100644 --- a/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/handler/PromQLApiHandler.java +++ b/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/handler/PromQLApiHandler.java @@ -60,12 +60,14 @@ import org.apache.skywalking.oap.query.promql.entity.response.MetricRspData; import org.apache.skywalking.oap.query.promql.entity.response.MetadataQueryRsp; import org.apache.skywalking.oap.query.promql.entity.response.MetricType; +import org.apache.skywalking.oap.query.promql.entity.response.QueryFormatRsp; import org.apache.skywalking.oap.query.promql.entity.response.QueryResponse; import org.apache.skywalking.oap.query.promql.entity.response.ResultType; import org.apache.skywalking.oap.query.promql.entity.response.ScalarRspData; import org.apache.skywalking.oap.query.promql.entity.response.SeriesQueryRsp; import org.apache.skywalking.oap.query.promql.rt.PromOpUtils; import org.apache.skywalking.oap.query.promql.rt.PromQLMatchVisitor; +import org.apache.skywalking.oap.query.promql.rt.exception.IllegalExpressionException; import org.apache.skywalking.oap.query.promql.rt.exception.ParseErrorListener; import org.apache.skywalking.oap.query.promql.rt.result.MatcherSetResult; import org.apache.skywalking.oap.query.promql.rt.result.MetricsRangeResult; @@ -188,12 +190,14 @@ public HttpResponse metadata( public HttpResponse labels( @Param("match[]") Optional match, @Param("start") Optional start, - @Param("end") Optional end) throws IOException { + @Param("end") Optional end, + @Param("limit") Optional limit) throws IOException { LabelsQueryRsp response = new LabelsQueryRsp(); long endTS = System.currentTimeMillis(); if (end.isPresent()) { endTS = formatTimestamp2Millis(end.get()); } + int limitNum = limit.orElse(100); Duration duration = getDayDurationFromTimestamp(endTS); if (match.isPresent()) { MatcherSetResult parseResult; @@ -219,12 +223,17 @@ public HttpResponse labels( if (metaData.isMultiIntValues()) { response.getData().remove(DataLabel.GENERAL_LABEL_NAME); } + } else if (ServiceTraffic.INDEX_NAME.equals(metricName) || InstanceTraffic.INDEX_NAME.equals(metricName) + || EndpointTraffic.INDEX_NAME.equals(metricName)) { + response.getData().addAll(buildLabelNamesForTraffic(metricName)); } } else { Arrays.stream(LabelName.values()).forEach(label -> { response.getData().add(label.getLabel()); }); } + List result = response.getData().stream().limit(limitNum).collect(Collectors.toList()); + response.setData(result); response.setStatus(ResultStatus.SUCCESS); return jsonResponse(response); } @@ -241,7 +250,8 @@ public HttpResponse labelValues( @Param("label_name") String labelName, @Param("match[]") Optional match, @Param("start") Optional start, - @Param("end") Optional end) throws IOException { + @Param("end") Optional end, + @Param("limit") Optional limit) throws IOException { LabelValuesQueryRsp response = new LabelValuesQueryRsp(); response.setStatus(ResultStatus.SUCCESS); long endTS = System.currentTimeMillis(); @@ -249,20 +259,21 @@ public HttpResponse labelValues( endTS = formatTimestamp2Millis(end.get()); } Duration duration = getDayDurationFromTimestamp(endTS); + int limitNum = limit.orElse(100); //general labels if (LabelName.NAME.getLabel().equals(labelName)) { - getMetricsMetadataQueryService().listMetrics("").forEach(definition -> { + getMetricsMetadataQueryService().listMetrics("").stream().limit(limitNum).forEach(definition -> { response.getData().add(definition.getName()); }); return jsonResponse(response); } else if (LabelName.LAYER.getLabel().equals(labelName)) { - for (Layer layer : Layer.values()) { + for (Layer layer : Arrays.stream(Layer.values()).limit(limitNum).collect(Collectors.toList())) { response.getData().add(layer.name()); } return jsonResponse(response); } else if (LabelName.SCOPE.getLabel().equals(labelName)) { - for (Scope scope : Scope.values()) { + for (Scope scope : Arrays.stream(Scope.values()).limit(limitNum).collect(Collectors.toList())) { response.getData().add(scope.name()); } return jsonResponse(response); @@ -283,43 +294,35 @@ public HttpResponse labelValues( metricName); if (valueColumn.isPresent() && Column.ValueDataType.LABELED_VALUE == valueColumn.get().getDataType()) { List matchedMetrics = getMatcherMetricsValues(parseResult, duration); - response.getData().addAll(buildLabelValuesFromQuery(matchedMetrics, labelName)); + response.getData().addAll(buildLabelValuesFromQuery(matchedMetrics, labelName).stream().limit(limitNum).collect(Collectors.toList())); } else { - // Make compatible with Grafana 11 when use old config variables - // e.g. query service list config: `label_values(service_traffic{layer='$layer'}, service)` - // Grafana 11 will query this API by default rather than `/api/v1/series`(< 11) - if (Objects.equals(metricName, ServiceTraffic.INDEX_NAME)) { - String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel()); - if (StringUtil.isNotBlank(serviceName)) { - Service service = metadataQuery.findService(serviceName).join(); - response.getData().add(service.getName()); - } else { - List services = metadataQuery.listServices( - parseResult.getLabelMap().get(LabelName.LAYER.getLabel())).join(); - services.forEach(service -> { + try { + // Make compatible with Grafana 11 when use old config variables + // e.g. query service list config: `label_values(service_traffic{layer='$layer'}, service)` + // Grafana 11 will query this API by default rather than `/api/v1/series`(< 11) + limitNum = getLimitNum(limit, parseResult); + String layer = parseResult.getLabelMap().get(LabelName.LAYER.getLabel()); + if (Objects.equals(metricName, ServiceTraffic.INDEX_NAME)) { + queryServiceTraffic(parseResult, layer, limitNum).forEach(service -> { response.getData().add(service.getName()); }); + } else if (Objects.equals(metricName, InstanceTraffic.INDEX_NAME)) { + String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel()); + queryInstanceTraffic(parseResult, duration, layer, serviceName, limitNum).forEach(instance -> { + response.getData().add(instance.getName()); + }); + } else if (Objects.equals(metricName, EndpointTraffic.INDEX_NAME)) { + String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel()); + String keyword = parseResult.getLabelMap().getOrDefault(LabelName.KEYWORD.getLabel(), ""); + queryEndpointTraffic(parseResult, duration, layer, serviceName, keyword, limitNum).forEach( + endpoint -> { + response.getData().add(endpoint.getName()); + }); } - } else if (Objects.equals(metricName, InstanceTraffic.INDEX_NAME)) { - String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel()); - String layer = parseResult.getLabelMap().get(LabelName.LAYER.getLabel()); - List instances = metadataQuery.listInstances( - duration, IDManager.ServiceID.buildId(serviceName, Layer.valueOf(layer).isNormal())).join(); - instances.forEach(instance -> { - response.getData().add(instance.getName()); - }); - } else if (Objects.equals(metricName, EndpointTraffic.INDEX_NAME)) { - String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel()); - String layer = parseResult.getLabelMap().get(LabelName.LAYER.getLabel()); - String keyword = parseResult.getLabelMap().getOrDefault(LabelName.KEYWORD.getLabel(), ""); - String limit = parseResult.getLabelMap().getOrDefault(LabelName.LIMIT.getLabel(), "100"); - List endpoints = metadataQuery.findEndpoint( - keyword, IDManager.ServiceID.buildId(serviceName, Layer.valueOf(layer).isNormal()), - Integer.parseInt(limit), duration - ).join(); - endpoints.forEach(endpoint -> { - response.getData().add(endpoint.getName()); - }); + } catch (IllegalExpressionException e) { + response.setStatus(ResultStatus.ERROR); + response.setErrorType(ErrorType.BAD_DATA); + response.setError(e.getMessage()); } } } @@ -333,7 +336,8 @@ public HttpResponse labelValues( public HttpResponse series( @Param("match[]") String match, @Param("start") String start, - @Param("end") String end) throws IOException { + @Param("end") String end, + @Param("limit") Optional limit) throws IOException { long startTS = formatTimestamp2Millis(start); long endTS = formatTimestamp2Millis(end); Duration duration = DurationUtils.timestamp2Duration(startTS, endTS); @@ -350,43 +354,37 @@ public HttpResponse series( String metricName = parseResult.getMetricName(); Optional valueColumn = ValueColumnMetadata.INSTANCE.readValueColumnDefinition( metricName); + int limitNum = getLimitNum(limit, parseResult); if (valueColumn.isPresent()) { ValueColumnMetadata.ValueColumn metaData = valueColumn.get(); Scope scope = Scope.Finder.valueOf(metaData.getScopeId()); Column.ValueDataType dataType = metaData.getDataType(); response.getData().add(buildMetaMetricInfo(metricName, scope, dataType)); - } else if (Objects.equals(metricName, ServiceTraffic.INDEX_NAME)) { - String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel()); - if (StringUtil.isNotBlank(serviceName)) { - Service service = metadataQuery.findService(serviceName).join(); - response.getData().add(buildMetricInfoFromTraffic(metricName, service)); - } else { - List services = metadataQuery.listServices( - parseResult.getLabelMap().get(LabelName.LAYER.getLabel())).join(); - services.forEach(service -> { - response.getData().add(buildMetricInfoFromTraffic(metricName, service)); - }); + } else { + try { + String layer = parseResult.getLabelMap().get(LabelName.LAYER.getLabel()); + if (Objects.equals(metricName, ServiceTraffic.INDEX_NAME)) { + queryServiceTraffic(parseResult, layer, limitNum).forEach(service -> { + response.getData().add(buildMetricInfoFromTraffic(metricName, service)); + }); + } else if (Objects.equals(metricName, InstanceTraffic.INDEX_NAME)) { + String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel()); + queryInstanceTraffic(parseResult, duration, layer, serviceName, limitNum).forEach(instance -> { + response.getData().add(buildMetricInfoFromTraffic(metricName, instance)); + }); + } else if (Objects.equals(metricName, EndpointTraffic.INDEX_NAME)) { + String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel()); + String keyword = parseResult.getLabelMap().getOrDefault(LabelName.KEYWORD.getLabel(), ""); + queryEndpointTraffic(parseResult, duration, layer, serviceName, keyword, limitNum).forEach( + endpoint -> { + response.getData().add(buildMetricInfoFromTraffic(metricName, endpoint)); + }); + } + } catch (IllegalExpressionException e) { + response.setStatus(ResultStatus.ERROR); + response.setErrorType(ErrorType.BAD_DATA); + response.setError(e.getMessage()); } - } else if (Objects.equals(metricName, InstanceTraffic.INDEX_NAME)) { - String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel()); - String layer = parseResult.getLabelMap().get(LabelName.LAYER.getLabel()); - List instances = metadataQuery.listInstances( - duration, IDManager.ServiceID.buildId(serviceName, Layer.valueOf(layer).isNormal())).join(); - instances.forEach(instance -> { - response.getData().add(buildMetricInfoFromTraffic(metricName, instance)); - }); - } else if (Objects.equals(metricName, EndpointTraffic.INDEX_NAME)) { - String serviceName = parseResult.getLabelMap().get(LabelName.SERVICE.getLabel()); - String layer = parseResult.getLabelMap().get(LabelName.LAYER.getLabel()); - String keyword = parseResult.getLabelMap().getOrDefault(LabelName.KEYWORD.getLabel(), ""); - String limit = parseResult.getLabelMap().getOrDefault(LabelName.LIMIT.getLabel(), "100"); - List endpoints = metadataQuery.findEndpoint( - keyword, IDManager.ServiceID.buildId(serviceName, Layer.valueOf(layer).isNormal()), - Integer.parseInt(limit), duration - ).join(); - endpoints.forEach(endpoint -> { - response.getData().add(buildMetricInfoFromTraffic(metricName, endpoint)); - }); } response.setStatus(ResultStatus.SUCCESS); @@ -467,6 +465,7 @@ public HttpResponse query_range( @Param("timeout") Optional timeout) throws IOException { long startTS = formatTimestamp2Millis(start); long endTS = formatTimestamp2Millis(end); + //OAP downsample data by min/hour/day step, and generate step query condition automatically by the time range. Duration duration = DurationUtils.timestamp2Duration(startTS, endTS); ExprQueryRsp response = new ExprQueryRsp(); PromQLLexer lexer = new PromQLLexer( @@ -509,6 +508,15 @@ public HttpResponse query_range( return jsonResponse(response); } + @Get + @Post + @Path("/api/v1/format_query") + public HttpResponse query_range(@Param("query") String query) throws IOException { + QueryFormatRsp rsp = new QueryFormatRsp(); + rsp.setData(query.replaceAll("\\s", "")); + return jsonResponse(rsp); + } + @Get @Post @Path("/api/v1/status/buildinfo") @@ -617,6 +625,17 @@ private List buildLabelNames(Scope scope, ValueColumnMetadata.ValueColum return labelNames; } + private List buildLabelNamesForTraffic(String metricName) { + List labelNames = new ArrayList<>(); + labelNames.add(LabelName.LAYER.getLabel()); + labelNames.add(LabelName.LIMIT.getLabel()); + labelNames.add(LabelName.SERVICE.getLabel()); + if (Objects.equals(metricName, EndpointTraffic.INDEX_NAME)) { + labelNames.add(LabelName.KEYWORD.getLabel()); + } + return labelNames; + } + private List buildLabelNamesFromQuery(List metricsValues) { Set labelNames = new LinkedHashSet<>(); metricsValues.forEach(metricsValue -> { @@ -727,6 +746,112 @@ private List getMatcherMetricsValues(MatcherSetResult parseResult return getMetricsQueryService().readLabeledMetricsValuesWithoutEntity(metricName, matchLabels, duration); } + private int getLimitNum(Optional limitParam, MatcherSetResult parseResult) { + String limitLabel = parseResult.getLabelMap().getOrDefault(LabelName.LIMIT.getLabel(), "100"); + int limitNum = Integer.parseInt(limitLabel); + if (limitParam.isPresent()) { + limitNum = Integer.min(limitParam.get(), Integer.parseInt(limitLabel)); + } + return limitNum; + } + + private List queryServiceTraffic(MatcherSetResult parseResult, String layer, int limitNum) throws IllegalExpressionException { + if (StringUtil.isBlank(layer)) { + throw new IllegalExpressionException("label {layer} should not be empty."); + } + List result = new ArrayList<>(); + MatcherSetResult.NameMatcher matcher = parseResult.getNameMatcher(); + if (matcher != null) { + String serviceName = matcher.getMatchString(); + if (matcher.getMatchOp() == PromQLParser.EQ) { + Service service = metadataQuery.findService(serviceName).join(); + if (service != null) { + result.add(service); + } + } else if (matcher.getMatchOp() == PromQLParser.NEQ) { + List services = metadataQuery.listServices(layer).join(); + services.stream().filter(s -> !s.getName().equals(serviceName)).limit(limitNum).forEach(result::add); + } else if (matcher.getMatchOp() == PromQLParser.RM) { + List services = metadataQuery.listServices(layer).join(); + services.stream().filter(s -> s.getName().matches(serviceName)).limit(limitNum).forEach(result::add); + } else if (matcher.getMatchOp() == PromQLParser.NRM) { + List services = metadataQuery.listServices(layer).join(); + services.stream().filter(s -> !s.getName().matches(serviceName)).limit(limitNum).forEach(result::add); + } + } else { + List services = metadataQuery.listServices( + parseResult.getLabelMap().get(LabelName.LAYER.getLabel())).join(); + services.stream().limit(limitNum).forEach(result::add); + } + return result; + } + + private List queryInstanceTraffic(MatcherSetResult parseResult, + Duration duration, + String layer, + String serviceName, + int limitNum) throws IllegalExpressionException { + if (StringUtil.isBlank(layer)) { + throw new IllegalExpressionException("label {layer} should not be empty."); + } + if (StringUtil.isBlank(serviceName)) { + throw new IllegalExpressionException("label {service} should not be empty."); + } + List result = new ArrayList<>(); + MatcherSetResult.NameMatcher matcher = parseResult.getNameMatcher(); + List instances = metadataQuery.listInstances( + duration, IDManager.ServiceID.buildId(serviceName, Layer.valueOf(layer).isNormal())).join(); + if (matcher != null) { + String instanceName = matcher.getMatchString(); + if (matcher.getMatchOp() == PromQLParser.EQ) { + instances.stream().filter(n -> n.getName().equals(instanceName)).findFirst().ifPresent(result::add); + } else if (matcher.getMatchOp() == PromQLParser.NEQ) { + instances.stream().filter(n -> !n.getName().equals(instanceName)).limit(limitNum).forEach(result::add); + } else if (matcher.getMatchOp() == PromQLParser.RM) { + instances.stream().filter(n -> n.getName().matches(instanceName)).limit(limitNum).forEach(result::add); + } else if (matcher.getMatchOp() == PromQLParser.NRM) { + instances.stream().filter(n -> !n.getName().matches(instanceName)).limit(limitNum).forEach(result::add); + } + } else { + instances.stream().limit(limitNum).forEach(result::add); + } + return result; + } + + private List queryEndpointTraffic(MatcherSetResult parseResult, + Duration duration, + String layer, + String serviceName, + String keyword, + int limitNum) throws IllegalExpressionException { + if (StringUtil.isBlank(layer)) { + throw new IllegalExpressionException("label {layer} should not be empty."); + } + if (StringUtil.isBlank(serviceName)) { + throw new IllegalExpressionException("label {service} should not be empty."); + } + List result = new ArrayList<>(); + List endpoints = metadataQuery.findEndpoint( + keyword, IDManager.ServiceID.buildId(serviceName, Layer.valueOf(layer).isNormal()), limitNum, duration + ).join(); + MatcherSetResult.NameMatcher matcher = parseResult.getNameMatcher(); + if (matcher != null) { + String endpointName = matcher.getMatchString(); + if (matcher.getMatchOp() == PromQLParser.EQ) { + endpoints.stream().filter(e -> e.getName().equals(endpointName)).findFirst().ifPresent(result::add); + } else if (matcher.getMatchOp() == PromQLParser.NEQ) { + endpoints.stream().filter(e -> !e.getName().equals(endpointName)).limit(limitNum).forEach(result::add); + } else if (matcher.getMatchOp() == PromQLParser.RM) { + endpoints.stream().filter(e -> e.getName().matches(endpointName)).limit(limitNum).forEach(result::add); + } else if (matcher.getMatchOp() == PromQLParser.NRM) { + endpoints.stream().filter(e -> !e.getName().matches(endpointName)).limit(limitNum).forEach(result::add); + } + } else { + endpoints.stream().limit(limitNum).forEach(result::add); + } + return result; + } + public enum QueryType { INSTANT, RANGE, diff --git a/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/rt/PromQLMatchVisitor.java b/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/rt/PromQLMatchVisitor.java index 7f1cee026fc2..447038d4a5e3 100644 --- a/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/rt/PromQLMatchVisitor.java +++ b/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/rt/PromQLMatchVisitor.java @@ -19,8 +19,12 @@ package org.apache.skywalking.oap.query.promql.rt; import java.util.Map; +import org.apache.skywalking.oap.query.promql.entity.LabelName; import org.apache.skywalking.oap.query.promql.rt.result.MatcherSetResult; import org.apache.skywalking.oap.query.promql.rt.result.ParseResultType; +import org.apache.skywalking.oap.server.core.analysis.manual.endpoint.EndpointTraffic; +import org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceTraffic; +import org.apache.skywalking.oap.server.core.analysis.manual.service.ServiceTraffic; import org.apache.skywalking.promql.rt.grammar.PromQLParser; import org.apache.skywalking.promql.rt.grammar.PromQLParserBaseVisitor; @@ -38,7 +42,17 @@ public MatcherSetResult visitMetricInstant(PromQLParser.MetricInstantContext ctx String labelName = labelCtx.labelName().getText(); String labelValue = labelCtx.labelValue().getText(); String labelValueTrim = labelValue.substring(1, labelValue.length() - 1); - labelMap.put(labelName, labelValueTrim); + if ((metricName.equals(ServiceTraffic.INDEX_NAME) && LabelName.SERVICE.getLabel().equals(labelName)) + || ((metricName.equals(InstanceTraffic.INDEX_NAME)) && LabelName.SERVICE_INSTANCE.getLabel() + .equals(labelName)) + || ((metricName.equals(EndpointTraffic.INDEX_NAME)) && LabelName.ENDPOINT.getLabel() + .equals(labelName))) { + result.setNameMatcher( + new MatcherSetResult.NameMatcher( + metricName, labelValueTrim, labelCtx.matchOp().getStart().getType())); + } else { + labelMap.put(labelName, labelValueTrim); + } } } return result; diff --git a/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/rt/result/MatcherSetResult.java b/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/rt/result/MatcherSetResult.java index c877edea9e85..4fb630675fc2 100644 --- a/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/rt/result/MatcherSetResult.java +++ b/oap-server/server-query-plugin/promql-plugin/src/main/java/org/apache/skywalking/oap/query/promql/rt/result/MatcherSetResult.java @@ -27,5 +27,13 @@ @Data public class MatcherSetResult extends ParseResult { private String metricName; + private NameMatcher nameMatcher; private Map labelMap = new HashMap<>(); + + @Data + public static class NameMatcher { + public final String trafficName; + public final String matchString; + public final int matchOp; + } } diff --git a/test/e2e-v2/cases/promql/docker-compose.yml b/test/e2e-v2/cases/promql/docker-compose.yml index 314631750737..437c8473bf9d 100644 --- a/test/e2e-v2/cases/promql/docker-compose.yml +++ b/test/e2e-v2/cases/promql/docker-compose.yml @@ -20,15 +20,12 @@ services: extends: file: ../../script/docker-compose/base-compose.yml service: oap - environment: - SW_STORAGE: elasticsearch - SW_STORAGE_ES_CLUSTER_NODES: es:9200 volumes: - ./oal/core.oal:/skywalking/config/oal/core.oal ports: - 9090 depends_on: - es: + banyandb: condition: service_healthy networks: - e2e @@ -57,21 +54,12 @@ services: provider: condition: service_healthy - es: - image: elastic/elasticsearch:7.10.2 + banyandb: + extends: + file: ../../script/docker-compose/base-compose.yml + service: banyandb ports: - - 9200:9200 - networks: - - e2e - environment: - - discovery.type=single-node - - cluster.routing.allocation.disk.threshold_enabled=false - - xpack.security.enabled=false - healthcheck: - test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/9200"] - interval: 5s - timeout: 60s - retries: 120 + - 17912 networks: diff --git a/test/e2e-v2/cases/promql/expected/endpoint-names.yml b/test/e2e-v2/cases/promql/expected/endpoint-names.yml new file mode 100644 index 000000000000..d5f1f13bf87b --- /dev/null +++ b/test/e2e-v2/cases/promql/expected/endpoint-names.yml @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{ + "status": "success", + "data": [ + "POST:/users" + ] +} diff --git a/test/e2e-v2/cases/promql/expected/instance-names.yml b/test/e2e-v2/cases/promql/expected/instance-names.yml new file mode 100644 index 000000000000..43b5c0cabcfa --- /dev/null +++ b/test/e2e-v2/cases/promql/expected/instance-names.yml @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{ + "status": "success", + "data": [ + "provider1" + ] +} diff --git a/test/e2e-v2/cases/promql/expected/service-names-limit.yml b/test/e2e-v2/cases/promql/expected/service-names-limit.yml new file mode 100644 index 000000000000..f0a37c5b481e --- /dev/null +++ b/test/e2e-v2/cases/promql/expected/service-names-limit.yml @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{ + "status": "success", + "data": [ + "e2e-service-provider" + ] +} diff --git a/test/e2e-v2/cases/promql/expected/service-names.yml b/test/e2e-v2/cases/promql/expected/service-names.yml new file mode 100644 index 000000000000..eec14eb3153a --- /dev/null +++ b/test/e2e-v2/cases/promql/expected/service-names.yml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{ + "status": "success", + "data": [ + "e2e-service-provider", + "e2e-service-consumer" + ] +} diff --git a/test/e2e-v2/cases/promql/expected/service-traffic-limit.yml b/test/e2e-v2/cases/promql/expected/service-traffic-limit.yml new file mode 100644 index 000000000000..d6566bb5c9f4 --- /dev/null +++ b/test/e2e-v2/cases/promql/expected/service-traffic-limit.yml @@ -0,0 +1,26 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{ + "status": "success", + "data": [ + { + "__name__": "service_traffic", + "service": "e2e-service-provider", + "scope": "Service", + "layer": "GENERAL" + } + ] +} diff --git a/test/e2e-v2/cases/promql/promql-cases.yaml b/test/e2e-v2/cases/promql/promql-cases.yaml index e53eb38a360f..480feaac78e6 100644 --- a/test/e2e-v2/cases/promql/promql-cases.yaml +++ b/test/e2e-v2/cases/promql/promql-cases.yaml @@ -17,12 +17,22 @@ cases: # traffics query - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=service_traffic{layer="GENERAL"}&start='$(($(date +%s)-1800))'&end='$(date +%s) expected: expected/service-traffic.yml + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=service_traffic{layer="GENERAL"}&limit=1&start='$(($(date +%s)-1800))'&end='$(date +%s) + expected: expected/service-traffic-limit.yml + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=service_traffic{layer="GENERAL",limit="1"}&start='$(($(date +%s)-1800))'&end='$(date +%s) + expected: expected/service-traffic-limit.yml + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=service_traffic{layer="GENERAL",service=~".*-provider"}&start='$(($(date +%s)-1800))'&end='$(date +%s) + expected: expected/service-traffic-limit.yml #RFC3399 - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=service_traffic{layer="GENERAL"}&start='$(($(date +%s)-1800))'&end='$(date -u +%Y-%m-%dT%H:%M:%S.111Z) expected: expected/service-traffic.yml - - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=instance_traffic{layer="GENERAL", service="e2e-service-provider"}&start='$(($(date +%s)-1800))'&end='$(date +%s) + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=instance_traffic{layer="GENERAL", service="e2e-service-provider"}&limit=1&start='$(($(date +%s)-1800))'&end='$(date +%s) expected: expected/instance-traffic.yml - - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=endpoint_traffic{layer="GENERAL", service="e2e-service-provider", keyword="POST:/users"}&start='$(($(date +%s)-1800))'&end='$(date +%s) + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=instance_traffic{layer="GENERAL", service="e2e-service-provider", service_instance=~"provider1|provider2"}&start='$(($(date +%s)-1800))'&end='$(date +%s) + expected: expected/instance-traffic.yml + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=endpoint_traffic{layer="GENERAL", service="e2e-service-provider", keyword="POST:/users"}&limit=1&start='$(($(date +%s)-1800))'&end='$(date +%s) + expected: expected/endpoint-traffic.yml + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=endpoint_traffic{layer="GENERAL", service="e2e-service-provider", endpoint=~"^POST:.*"}&start='$(($(date +%s)-1800))'&end='$(date +%s) expected: expected/endpoint-traffic.yml # metrics series - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=service_cpm&start='$(($(date +%s)-1800))'&end='$(date +%s) @@ -31,6 +41,23 @@ cases: expected: expected/instance-metric-series.yml - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/series -d 'match[]=endpoint_cpm&start='$(($(date +%s)-1800))'&end='$(date +%s) expected: expected/endpoint-metric-series.yml + # traffic names query + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/service/values -d 'match[]=service_traffic{layer="GENERAL"}&start='$(($(date +%s)-1800))'&end='$(date -u +%Y-%m-%dT%H:%M:%S.111Z) + expected: expected/service-names.yml + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/service/values -d 'match[]=service_traffic{layer="GENERAL",limit="1"}&start='$(($(date +%s)-1800))'&end='$(date -u +%Y-%m-%dT%H:%M:%S.111Z) + expected: expected/service-names-limit.yml + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/service/values -d 'match[]=service_traffic{layer="GENERAL"}&limit=1' + expected: expected/service-names-limit.yml + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/service/values -d 'match[]=service_traffic{layer="GENERAL",service=~".*-provider"}' + expected: expected/service-names-limit.yml + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/service_instance/values -d 'match[]=instance_traffic{layer="GENERAL", service="e2e-service-provider"}&limit=1' + expected: expected/instance-names.yml + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/service_instance/values -d 'match[]=instance_traffic{layer="GENERAL", service="e2e-service-provider", service_instance=~"provider1|provider2"}&limit=1' + expected: expected/instance-names.yml + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/endpoint/values -d 'match[]=endpoint_traffic{layer="GENERAL", service="e2e-service-provider", keyword="POST:/users"}&limit=1' + expected: expected/endpoint-names.yml + - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/endpoint/values -d 'match[]=endpoint_traffic{layer="GENERAL", service="e2e-service-provider", endpoint=~"^POST:.*"}' + expected: expected/endpoint-names.yml # metrics names query - query: curl -X GET http://${oap_host}:${oap_9090}/api/v1/label/__name__/values expected: expected/metrics-names.yml