Skip to content

Commit 841a92b

Browse files
wu-shengwu-sheng
authored
Support Get Effective TTL Configurations API. (#12847)
* [Breaking Change] Rename `debugging-query` module to `status-query` module. Relative exposed APIs are **UNCHANGED**. * Support `Get Effective TTL Configurations` API.
1 parent 1c9c685 commit 841a92b

File tree

45 files changed

+527
-58
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+527
-58
lines changed

‎docs/en/changes/changes.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
* This change would not impact the existing deployment and its feature for our official release users.
77
* **Warning** If there are custom query plugins for our Elasticsearch indices, this change could break them as
88
sort queries and aggregation queries which used the unexpected fields are being blocked.
9+
* [Breaking Change] Rename `debugging-query` module to `status-query` module. Relative exposed APIs are **UNCHANGED**.
910

1011
#### OAP Server
1112

@@ -34,6 +35,7 @@
3435
* BanyanDB: Support update the Schema when OAP starting.
3536
* BanyanDB: Speed up OAP booting while initializing BanyanDB.
3637
* BanyanDB: Support `@EnableSort` on the column to enable sorting for `IndexRule` and set the default to false.
38+
* Support `Get Effective TTL Configurations` API.
3739

3840
#### UI
3941

@@ -61,5 +63,6 @@
6163
* Improve BanyanDB documentation.
6264
* Improve component-libraries documentation.
6365
* Improve configuration-vocabulary documentation.
66+
* Add `Get Effective TTL Configurations` API documentation.
6467

6568
All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/224?closed=1)

‎docs/en/debugging/config_dump.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,15 @@
1-
# Scratch The OAP Config Dump
1+
# Dump Effective Initial Configurations
22

33
SkyWalking OAP behaviors could be controlled through hundreds of configurations. It is hard to know what is the final
4-
configuration as all the configurations could be overrided by system environments.
4+
configuration as all the configurations could be override by system environments.
55

6-
The core config file [application.yml](../../../oap-server/server-starter/src/main/resources/application.yml) lists all
7-
the configurations
6+
The core config file [application.yml](../../../oap-server/server-starter/src/main/resources/application.yml) lists all the configurations
87
and their default values. However, it is still hard to know the runtime value.
98

10-
Scratch is a tool to dump the final configuration. It is provided within OAP rest server, which could be accessed
11-
through HTTP GET `http://{core restHost}:{core restPort}/debugging/config/dump`.
9+
Dump Effective Initial Configurations API is designed to help users to understand the effective configurations, no matter
10+
they are initialized in the `application.yml`, or override through system environments.
11+
- URL, `http://{core restHost}:{core restPort}/debugging/config/dump`
12+
- HTTP GET method.
1213

1314
```shell
1415
> curl http://127.0.0.1:12800/debugging/config/dump

‎docs/en/setup/backend/configuration-vocabulary.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -344,7 +344,7 @@ The Configuration Vocabulary lists all available configurations provided by `app
344344
| - | - | kafkaTopicLog | Kafka topic name for log. | SW_EXPORTER_KAFKA_TOPIC_LOG | skywalking-export-log |
345345
| - | - | exportErrorStatusTraceOnly | Export error status trace segments through the Kafka channel. | SW_EXPORTER_KAFKA_TRACE_FILTER_ERROR | false |
346346
| health-checker | default | checkIntervalSeconds | The period of checking OAP internal health status (in seconds). | SW_HEALTH_CHECKER_INTERVAL_SECONDS | 5 |
347-
| debugging-query | default | | | | |
347+
| status-query | default | | | | |
348348
| - | - | keywords4MaskingSecretsOfConfig | Include the list of keywords to filter configurations including secrets. Separate keywords by a comma. | SW_DEBUGGING_QUERY_KEYWORDS_FOR_MASKING_SECRETS | user,password,token,accessKey,secretKey,authentication |
349349
| configuration-discovery | default | disableMessageDigest | If true, agent receives the latest configuration every time, even without making any changes. By default, OAP uses the SHA512 message digest mechanism to detect changes in configuration. | SW_DISABLE_MESSAGE_DIGEST | false |
350350
| receiver-event | default | gRPC services that handle events data. | - | - | |

‎docs/en/setup/backend/ttl.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
1-
# TTL
1+
# Data Lifecycle. Time To Live (TTL)
2+
Time To Live (TTL) is a mechanism to automatically delete data that is older than the specified time.
3+
24
In SkyWalking, there are two types of observability data:
35
1. Records include traces, logs, topN sampled statements and alarm. `recordDataTTL` applies to **record** data.
46
1. Metrics include all metrics for service, instance, endpoint, and topology map. Metadata(lists of services, instances, or endpoints) also belongs to metrics. `metricsDataTTL` applies to **Metrics** data.
@@ -12,6 +14,9 @@ These are the settings for the different types:
1214
1315
## BanyanDB TTL
1416
15-
BanyanDB has a TTL mechanism to automatically delete data that is older than the specified time. When you use BanyanDB as the storage backend, `recordDataTTL` and `metricsDataTTL` are not used. Instead, you should configure the TTL settings in `storage.banyandb`.
17+
BanyanDB has a TTL mechanism to automatically delete data that is older than the specified time. When you use BanyanDB
18+
as the storage backend, `recordDataTTL` and `metricsDataTTL` are not used. Instead, you should configure the TTL
19+
settings in `storage.banyandb`.
1620

17-
Please refer to the [Storage BanyanDB](storages/banyandb.md) document for more information.
21+
Please refer to the [Storage BanyanDB](storages/banyandb.md) and [BanyanDB's Progressive TTL](../../banyandb/ttl.md)
22+
documents for more information.

‎docs/en/status/query_ttl_setup.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Get Effective TTL Configurations
2+
3+
Time To Live (TTL) mechanism has different behaviors according to different storage implementations. By default, the
4+
core module provides two TTL configurations: [`recordDataTTL` and `metricsDataTTL`](../setup/backend/ttl.md).
5+
But some storage implementations could override these settings and provide its own TTL configurations, for example,
6+
BanyanDB provides its native TTL mechanism to support [progressive TTL](../banyandb/ttl.md) feature.
7+
8+
This API is used to get the unified and effective TTL configurations.
9+
- URL, `http://{core restHost}:{core restPort}/status/config/ttl`
10+
- HTTP GET method.
11+
12+
```shell
13+
> curl -X GET "http://127.0.0.1:12800/status/config/ttl"
14+
# Metrics TTL includes the definition of the TTL of the metrics-ish data in the storage,
15+
# e.g.
16+
# 1. The metadata of the service, instance, endpoint, topology map, etc.
17+
# 2. Generated metrics data from OAL and MAL engines.
18+
#
19+
# TTLs for each granularity metrics are listed separately.
20+
metrics.minute=7
21+
metrics.hour=7
22+
metrics.day=7
23+
24+
# Records TTL includes the definition of the TTL of the records data in the storage,
25+
# Records include traces, logs, sampled slow SQL statements, HTTP requests(by Rover), alarms, etc.
26+
# Super dataset of records are traces and logs, which volume should be much larger.
27+
records.default=3
28+
records.superDataset=3
29+
```
30+
31+
This API also provides the response in JSON format, which is more friendly for programmatic usage.
32+
33+
```shell
34+
> curl -X GET "http://127.0.0.1:12800/status/config/ttl" \
35+
-H "Accept: application/json"
36+
37+
{
38+
"metrics": {
39+
"minute": 7,
40+
"hour": 7,
41+
"day": 7
42+
},
43+
"records": {
44+
"default": 3,
45+
"superDataset": 3
46+
}
47+
}
48+
```

‎docs/menu.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -352,12 +352,14 @@ catalog:
352352
path: "/en/concepts-and-designs/metrics-attributes"
353353
- name: "Security Notice"
354354
path: "/en/security/readme"
355-
- name: "Debugging"
355+
- name: "Status APIs"
356356
catalog:
357-
- name: "Scratch The OAP Config Dump"
357+
- name: "Dump Effective Initial Configurations"
358358
path: "/en/debugging/config_dump"
359359
- name: "Tracing Query Execution"
360360
path: "/en/debugging/query-tracing"
361+
- name: "Get Effective TTL Configurations"
362+
path: "/en/status/query_ttl_setup"
361363
- name: "Academy"
362364
catalog:
363365
- name: "Scaling SkyWalking server automatically in kubernetes"

‎oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModule.java‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@
5252
import org.apache.skywalking.oap.server.core.query.MetricsMetadataQueryService;
5353
import org.apache.skywalking.oap.server.core.query.MetricsQueryService;
5454
import org.apache.skywalking.oap.server.core.query.RecordQueryService;
55+
import org.apache.skywalking.oap.server.core.query.TTLStatusQuery;
5556
import org.apache.skywalking.oap.server.core.query.TagAutoCompleteQueryService;
5657
import org.apache.skywalking.oap.server.core.query.TopNRecordsQueryService;
5758
import org.apache.skywalking.oap.server.core.query.TopologyQueryService;
@@ -96,7 +97,7 @@ public Class[] services() {
9697

9798
addServerInterface(classes);
9899
addReceiverInterface(classes);
99-
addInsideService(classes);
100+
addInternalServices(classes);
100101
addCacheService(classes);
101102
addQueryService(classes);
102103
addProfileService(classes);
@@ -153,14 +154,15 @@ private void addQueryService(List<Class> classes) {
153154
classes.add(TagAutoCompleteQueryService.class);
154155
classes.add(RecordQueryService.class);
155156
classes.add(HierarchyQueryService.class);
157+
classes.add(TTLStatusQuery.class);
156158
}
157159

158160
private void addServerInterface(List<Class> classes) {
159161
classes.add(GRPCHandlerRegister.class);
160162
classes.add(HTTPHandlerRegister.class);
161163
}
162164

163-
private void addInsideService(List<Class> classes) {
165+
private void addInternalServices(List<Class> classes) {
164166
classes.add(ModelCreator.class);
165167
classes.add(IModelManager.class);
166168
classes.add(ModelManipulator.class);

‎oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleProvider.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@
7777
import org.apache.skywalking.oap.server.core.query.MetricsMetadataQueryService;
7878
import org.apache.skywalking.oap.server.core.query.MetricsQueryService;
7979
import org.apache.skywalking.oap.server.core.query.RecordQueryService;
80+
import org.apache.skywalking.oap.server.core.query.TTLStatusQuery;
8081
import org.apache.skywalking.oap.server.core.query.TagAutoCompleteQueryService;
8182
import org.apache.skywalking.oap.server.core.query.TopNRecordsQueryService;
8283
import org.apache.skywalking.oap.server.core.query.TopologyQueryService;
@@ -304,6 +305,13 @@ public void prepare() throws ServiceNotProvidedException, ModuleStartException {
304305
TagAutoCompleteQueryService.class, new TagAutoCompleteQueryService(getManager(), moduleConfig));
305306
this.registerServiceImplementation(RecordQueryService.class, new RecordQueryService(getManager()));
306307
this.registerServiceImplementation(HierarchyQueryService.class, new HierarchyQueryService(getManager(), moduleConfig));
308+
this.registerServiceImplementation(
309+
TTLStatusQuery.class, new TTLStatusQuery(
310+
getManager(),
311+
moduleConfig.getMetricsDataTTL(),
312+
moduleConfig.getRecordDataTTL()
313+
)
314+
);
307315

308316
// add profile service implementations
309317
this.registerServiceImplementation(

‎oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/TTLStatusQuery.java‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*
17+
*/
18+
19+
package org.apache.skywalking.oap.server.core.query;
20+
21+
import lombok.RequiredArgsConstructor;
22+
import org.apache.skywalking.oap.server.core.storage.StorageModule;
23+
import org.apache.skywalking.oap.server.core.storage.ttl.MetricsTTL;
24+
import org.apache.skywalking.oap.server.core.storage.ttl.RecordsTTL;
25+
import org.apache.skywalking.oap.server.core.storage.ttl.StorageTTLStatusQuery;
26+
import org.apache.skywalking.oap.server.core.storage.ttl.TTLDefinition;
27+
import org.apache.skywalking.oap.server.library.module.ModuleManager;
28+
import org.apache.skywalking.oap.server.library.module.Service;
29+
30+
@RequiredArgsConstructor
31+
public class TTLStatusQuery implements Service {
32+
private final ModuleManager moduleManager;
33+
private final int coreMetricsDataTTL;
34+
private final int coreRecordDataTTL;
35+
36+
private StorageTTLStatusQuery storageTTLStatusQuery;
37+
38+
private StorageTTLStatusQuery getStorageTTLStatusQuery() {
39+
if (storageTTLStatusQuery == null) {
40+
storageTTLStatusQuery = moduleManager.find(StorageModule.NAME)
41+
.provider()
42+
.getService(StorageTTLStatusQuery.class);
43+
}
44+
return storageTTLStatusQuery;
45+
}
46+
47+
/**
48+
* @return effective TTL configuration values.
49+
*/
50+
public TTLDefinition getTTL() {
51+
TTLDefinition ttlDefinition = getStorageTTLStatusQuery().getTTL();
52+
if (ttlDefinition == null) {
53+
ttlDefinition = new TTLDefinition(
54+
new MetricsTTL(coreMetricsDataTTL, coreMetricsDataTTL, coreMetricsDataTTL),
55+
new RecordsTTL(coreRecordDataTTL, coreRecordDataTTL)
56+
);
57+
}
58+
return ttlDefinition;
59+
}
60+
}

‎oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/StorageModule.java‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@
4646
import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
4747
import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
4848
import org.apache.skywalking.oap.server.core.storage.query.IZipkinQueryDAO;
49+
import org.apache.skywalking.oap.server.core.storage.ttl.StorageTTLStatusQuery;
4950
import org.apache.skywalking.oap.server.library.module.ModuleDefine;
5051

5152
/**
@@ -95,6 +96,7 @@ public Class[] services() {
9596
IAsyncProfilerTaskQueryDAO.class,
9697
IAsyncProfilerTaskLogQueryDAO.class,
9798
IJFRDataQueryDAO.class,
99+
StorageTTLStatusQuery.class
98100
};
99101
}
100102
}

0 commit comments

Comments
 (0)