Skip to content

Commit 8e468ef

Browse files
gunlee01gunlee01
authored
Merge pull request #339 from scouter-project/develop
Documentation of alert plugin.
2 parents 092a4f0 + 294fee0 commit 8e468ef

File tree

4 files changed

+290
-16
lines changed

4 files changed

+290
-16
lines changed

‎scouter.document/main/Alert-Plugin-Guide.md‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
# Alert Plugin Guide
2+
![Englsh](https://img.shields.io/badge/language-English-orange.svg) [![Korean](https://img.shields.io/badge/language-Korean-blue.svg)](Alert-Plugin-Guide_kr.md)
3+
4+
We can build our own alarm rules by handling alert scripting plugins which are able to compose various performance metrics.
5+
6+
## How to
7+
1. create 2 files in the server plugin directory (**[server_running_dir]/plugin** by default)
8+
* [PERFORMANCE_COUNTER_NAME].alert : script for alarm
9+
* [PERFORMANCE_COUNTER_NAME].conf : configuration for this alarm
10+
11+
2. [PERFORMANCE_COUNTER_NAME].conf
12+
* ```properties
13+
history_size=150
14+
silent_time=300
15+
check_time=30
16+
```
17+
* ```history_size``` : data count that will be kept
18+
* if it is set as 150ea, the oldest data may be 300 seconds before data because Scouter counter data is sent every 2 seconds.
19+
* ```silent_time``` : alarm sleep time - If the alarm is occurred in past [x] seconds, the same alarm is ignored.
20+
* ```check_time``` : script([PERFORMANCE_COUNTER_NAME].alert) invoking interval.
21+
22+
3. [PERFORMANCE_COUNTER_NAME].alert
23+
* script file for alert rule (java)
24+
25+
* sample1 (**GcTime.alert**)
26+
* alert when ```GcTime``` is over than 2 sec
27+
```java
28+
// void process(RealCounter $counter)
29+
int gcTime = $counter.intValue();
30+
if(gcTime > 2000) {
31+
$counter.fatal("gc time fatal", "gc time:" + respTime + "ms");
32+
}
33+
```
34+
35+
* sample2 (**Elasped90%.alert**)
36+
* alert when ```Elasped90%``` is over than 1.5 sec (ignore when TPS is lower than 3 sec.)
37+
```java
38+
// void process(RealCounter $counter)
39+
int warn = 1500;
40+
int fatal = 2000;
41+
42+
int tps = $counter.intValue("TPS");
43+
int respTime = $counter.intValue();
44+
45+
String objType = $counter.objType();
46+
String objName = $counter.objName();
47+
48+
java.text.NumberFormat f = java.text.NumberFormat.getInstance();
49+
f.setMaximumFractionDigits(2);
50+
51+
if(tps < 3) return;
52+
if(respTime > fatal) {
53+
$counter.fatal("resp time fatal high", "90% resp time:" + f.format((long)respTime) + "ms, tps:" + tps);
54+
} else if(respTime > warn) {
55+
$counter.warn("resp time warn high", "90% resp time:" + f.format((long)respTime) + "ms, tps:" + tps);
56+
}
57+
```
58+
59+
* sample3 (**TPS.alert**)
60+
* alert when ```TPS``` increase or decrease sharply.
61+
```java
62+
// void process(RealCounter $counter)
63+
64+
//increase
65+
float initTps = 20.0;
66+
float warnIncreaseRate = 1.2;
67+
float fatalIncreaseRate = 1.5;
68+
int compareBeforeSec = 180;
69+
70+
//decrease
71+
float fatal1DecreaseRate = 0.7; //30%
72+
float fatal2DecreaseRate = 0.5; //50%
73+
int compareBeforeDecreaseSec = 120; //last 2 minute
74+
75+
float tps = $counter.floatValue();
76+
String objType = $counter.objType();
77+
String objName = $counter.objName();
78+
float errorRate = $counter.floatValue();
79+
java.text.NumberFormat f = java.text.NumberFormat.getInstance();
80+
f.setMaximumFractionDigits(1);
81+
82+
if(tps > initTps) {
83+
float preValue = $counter.getAvg(compareBeforeSec, 4);
84+
if(preValue > 0.0f) {
85+
if(tps > preValue * fatalIncreaseRate) {
86+
$counter.fatal("TPS increase fatal"
87+
, "TPS is " + f.format((double)tps/preValue*100) + "% higher than " + compareBeforeSec + "sec ago"
88+
+ ", TPS: " + tps);
89+
90+
} else if(tps > preValue * warnIncreaseRate) {
91+
$counter.warn("TPS increase warning"
92+
, "TPS is " + f.format((double)tps/preValue*100) + "% higher than " + compareBeforeSec + "sec ago"
93+
+ ", TPS: " + tps);
94+
}
95+
}
96+
}
97+
98+
float preValue = $counter.getAvg(compareBeforeDecreaseSec, 4);
99+
if(preValue > 5.0f) {
100+
if(tps < preValue * fatal2DecreaseRate) {
101+
$counter.fatal("TPS decrease fatal"
102+
, "TPS is " + f.format(((double)1-tps/preValue)*100) + "% lower than " + compareBeforeDecreaseSec + "sec ago"
103+
+ ", TPS: " + tps);
104+
} else if(tps < preValue * fatal1DecreaseRate) {
105+
$counter.error("TPS decrease warn"
106+
, "TPS is " + f.format((double)(1-tps/preValue)*100) + "% lower than " + compareBeforeDecreaseSec + "sec ago"
107+
+ ", TPS: " + tps);
108+
}
109+
}
110+
```
111+
112+
### RealCounter API
113+
| method | desc |
114+
| ------------ | ---------- |
115+
| objName() | get object's name that produced the counter value |
116+
| objType() | get object's type that produced the counter value |
117+
| intValue() | get counter value as integer |
118+
| floatValue() | get counter value as float |
119+
| historySize() | get history size set by the conf file |
120+
| overCount(int value, int sec) | get how many times exceed the value in the seconds |
121+
| overCount(float value, int sec) | get how many times exceed the value in the seconds |
122+
| getAvg(int fromAgoSec, int durationSec) | calculate average of the counter values in fromAgoSec for durationSec |
123+
| getLatestAvg(int durationSec) | calculate average of the counter values from durationSec ago to now |
124+
| info(String title, String message) | invoke alarm as info level |
125+
| warn(String title, String message) | invoke alarm as warn level |
126+
| error(String title, String message) | invoke alarm as error level |
127+
| fatal(String title, String message) | invoke alarm as fatal level |
128+
| floatValue(String anotherCounterName) | get another counter's current value by the name |
129+
| intValue(String anotherCounterName) | get another counter's current value by the name |
130+
131+
### Counter names
132+
* [counters.xml](https://github.com/scouter-project/scouter/blob/fe74bdb73a34be2f390f8476991d59a5de6ea204/scouter.common/src/main/resources/scouter/lang/counters/counters.xml)
133+

‎scouter.document/main/Alert-Plugin-Guide_kr.md‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
# Alert Plugin Guide
2+
[![Englsh](https://img.shields.io/badge/language-English-orange.svg)](Alert-Plugin-Guide.md) ![Korean](https://img.shields.io/badge/language-Korean-blue.svg)
3+
4+
We can build our own alarm rules by handling alert scripting plugins which are able to compose various performance metrics.
5+
6+
## How to
7+
1. create 2 files in the server plugin directory (**[server_running_dir]/plugin** by default)
8+
* [PERFORMANCE_COUNTER_NAME].alert : script for alarm
9+
* [PERFORMANCE_COUNTER_NAME].conf : configuration for this alarm
10+
11+
2. [PERFORMANCE_COUNTER_NAME].conf
12+
* ```properties
13+
history_size=150
14+
silent_time=300
15+
check_time=30
16+
```
17+
* ```history_size``` : data count that will be kept
18+
* if it is set as 150ea, the oldest data may be 300 seconds before data because Scouter counter data is sent every 2 seconds.
19+
* ```silent_time``` : alarm sleep time - If the alarm is occurred in past [x] seconds, the same alarm is ignored.
20+
* ```check_time``` : script([PERFORMANCE_COUNTER_NAME].alert) invoking interval.
21+
22+
3. [PERFORMANCE_COUNTER_NAME].alert
23+
* script file for alert rule (java)
24+
25+
* sample1 (**GcTime.alert**)
26+
* alert when ```GcTime``` is over than 2 sec
27+
```java
28+
// void process(RealCounter $counter)
29+
int gcTime = $counter.intValue();
30+
if(gcTime > 2000) {
31+
$counter.fatal("gc time fatal", "gc time:" + respTime + "ms");
32+
}
33+
```
34+
35+
* sample2 (**Elasped90%.alert**)
36+
* alert when ```Elasped90%``` is over than 1.5 sec (ignore when TPS is lower than 3 sec.)
37+
```java
38+
// void process(RealCounter $counter)
39+
int warn = 1500;
40+
int fatal = 2000;
41+
42+
int tps = $counter.intValue("TPS");
43+
int respTime = $counter.intValue();
44+
45+
String objType = $counter.objType();
46+
String objName = $counter.objName();
47+
48+
java.text.NumberFormat f = java.text.NumberFormat.getInstance();
49+
f.setMaximumFractionDigits(2);
50+
51+
if(tps < 3) return;
52+
if(respTime > fatal) {
53+
$counter.fatal("resp time fatal high", "90% resp time:" + f.format((long)respTime) + "ms, tps:" + tps);
54+
} else if(respTime > warn) {
55+
$counter.warn("resp time warn high", "90% resp time:" + f.format((long)respTime) + "ms, tps:" + tps);
56+
}
57+
```
58+
59+
* sample3 (**TPS.alert**)
60+
* alert when ```TPS``` increase or decrease sharply.
61+
```java
62+
// void process(RealCounter $counter)
63+
64+
//increase
65+
float initTps = 20.0;
66+
float warnIncreaseRate = 1.2;
67+
float fatalIncreaseRate = 1.5;
68+
int compareBeforeSec = 180;
69+
70+
//decrease
71+
float fatal1DecreaseRate = 0.7; //30%
72+
float fatal2DecreaseRate = 0.5; //50%
73+
int compareBeforeDecreaseSec = 120; //last 2 minute
74+
75+
float tps = $counter.floatValue();
76+
String objType = $counter.objType();
77+
String objName = $counter.objName();
78+
float errorRate = $counter.floatValue();
79+
java.text.NumberFormat f = java.text.NumberFormat.getInstance();
80+
f.setMaximumFractionDigits(1);
81+
82+
if(tps > initTps) {
83+
float preValue = $counter.getAvg(compareBeforeSec, 4);
84+
if(preValue > 0.0f) {
85+
if(tps > preValue * fatalIncreaseRate) {
86+
$counter.fatal("TPS increase fatal"
87+
, "TPS is " + f.format((double)tps/preValue*100) + "% higher than " + compareBeforeSec + "sec ago"
88+
+ ", TPS: " + tps);
89+
90+
} else if(tps > preValue * warnIncreaseRate) {
91+
$counter.warn("TPS increase warning"
92+
, "TPS is " + f.format((double)tps/preValue*100) + "% higher than " + compareBeforeSec + "sec ago"
93+
+ ", TPS: " + tps);
94+
}
95+
}
96+
}
97+
98+
float preValue = $counter.getAvg(compareBeforeDecreaseSec, 4);
99+
if(preValue > 5.0f) {
100+
if(tps < preValue * fatal2DecreaseRate) {
101+
$counter.fatal("TPS decrease fatal"
102+
, "TPS is " + f.format(((double)1-tps/preValue)*100) + "% lower than " + compareBeforeDecreaseSec + "sec ago"
103+
+ ", TPS: " + tps);
104+
} else if(tps < preValue * fatal1DecreaseRate) {
105+
$counter.error("TPS decrease warn"
106+
, "TPS is " + f.format((double)(1-tps/preValue)*100) + "% lower than " + compareBeforeDecreaseSec + "sec ago"
107+
+ ", TPS: " + tps);
108+
}
109+
}
110+
```
111+
112+
### RealCounter API
113+
| method | desc |
114+
| ------------ | ---------- |
115+
| objName() | get object's name that produced the counter value |
116+
| objType() | get object's type that produced the counter value |
117+
| intValue() | get counter value as integer |
118+
| floatValue() | get counter value as float |
119+
| historySize() | get history size set by the conf file |
120+
| overCount(int value, int sec) | get how many times exceed the value in the seconds |
121+
| overCount(float value, int sec) | get how many times exceed the value in the seconds |
122+
| getAvg(int fromAgoSec, int durationSec) | calculate average of the counter values in fromAgoSec for durationSec |
123+
| getLatestAvg(int durationSec) | calculate average of the counter values from durationSec ago to now |
124+
| info(String title, String message) | invoke alarm as info level |
125+
| warn(String title, String message) | invoke alarm as warn level |
126+
| error(String title, String message) | invoke alarm as error level |
127+
| fatal(String title, String message) | invoke alarm as fatal level |
128+
| floatValue(String anotherCounterName) | get another counter's current value by the name |
129+
| intValue(String anotherCounterName) | get another counter's current value by the name |
130+
131+
### Counter names
132+
* [counters.xml](https://github.com/scouter-project/scouter/blob/fe74bdb73a34be2f390f8476991d59a5de6ea204/scouter.common/src/main/resources/scouter/lang/counters/counters.xml)
133+

‎scouter.document/main/Plugin-Guide.md‎

Lines changed: 18 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,17 @@
11
# Scouter Plugin Guide
22
![Englsh](https://img.shields.io/badge/language-English-orange.svg) [![Korean](https://img.shields.io/badge/language-Korean-blue.svg)](Plugin-Guide_kr.md)
33

4-
This article explains plugin funtion which enables scouter's extensibility.
4+
This article explains plugin functions which enable scouter's extensibility.
55

6-
With the plugin fuction of Scouter collector sever, data collected by scouter can be pre-handled and can be shared to other softwares.
6+
With the plugin function of Scouter collector sever, data collected by scouter can be pre-handled and can be shared to other softwares.
77
With agent plugin, certain types of data can be modified and other business-meaningful data can be added data to XLog or profile.
88

9-
> With Scouter plugin, configuration and extension can be done to enable collaboration with other open source.
9+
> Scouter plugins enable Scouter to collaborate with other open sources.
1010
1111
Scouter has 2 type of plugins - server plugin and an agent plugin.
1212
**server plug-in** is for to collector server
1313
and **agent Plugin** is for to Java agent.
14-
Server plugin has 2 type plugins including **scripting plugin**, **built-in-type plugin**.
14+
Server plugin has 2 type plugins including **script-type plugin**, **built-in-type plugin**.
1515

1616
## List of available plugins
1717
Below are the list of official plugins from scouter project and from contributors.
@@ -28,9 +28,9 @@ Below are the list of official plugins from scouter project and from contributor
2828

2929
## Server Plugin - How to
3030

31-
### 1. Scripting Plugin
31+
### 1. Script (type) Plugin
3232
A scripting plugin is called right before xlog data is stored in a repository.
33-
You can add simple script for manipulate some data on the plugin files which are located in the directory **[server_runnig_dir]/plugin** by default.
33+
You can add simple script for manipulate some data on the plugin files which are located in the directory **[server_running_dir]/plugin** by default.
3434

3535
Scouter distribution has the samples and the file name can not be modified.
3636
Currently 6 types of scripting plugins are supported.
@@ -44,12 +44,12 @@ Currently 6 types of scripting plugins are supported.
4444
refer to the link for details.
4545
**[Scripting plugin Server API](Server-Plugin-Scripting.md)**를 참고한다.
4646

47-
### 2. Built-in Plugin
47+
### 2. Built-in (type) Plugin
4848
Builing scripting plugin is very simple and can be dynamically loaded on runtime environment.
4949
On the other hand if you need the function permanently, it's too easy to fragile.
5050
So scouter provides another plugin type which allow you can attach pre-built compiled plugin and it's called as **Built-in Plugin**.
5151

52-
Scouter load the plugins on startup, if the plugins are located in scouter server's library directory.(default:**[server_runnig_dir]/lib**)
52+
Scouter load the plugins on startup, if the plugins are located in scouter server's library directory.(default:**[server_running_dir]/lib**)
5353

5454
#### 2.1 Server Built-in Plugin development guide
5555
##### 1. dependecny
@@ -87,8 +87,8 @@ public class NullPlugin {
8787
```
8888

8989
* Check it.
90-
> 1. The file is located sub package of **scouter.plugin.server** because annotation scan scope is from scouter.plugin.server package.
91-
> 2. The plugin can use scouter server's configuration and the option name must start with **ext_plugin_xxx**.
90+
> 1. The file is located sub package of **scouter.plugin.server** because annotation scan scope is from scouter.plugin.server package.
91+
> 2. The plugin can use scouter server's configuration and the option name must start with **ext_plugin_xxx**.
9292
9393
* ```ServerPlugin``` annotations
9494

@@ -104,14 +104,17 @@ Provided sample plugin that just prints the data collected.
104104
* Sample plugin : [https://github.com/scouter-project/scouter-plugin-server-null](https://github.com/scouter-project/scouter-plugin-server-null)
105105
* Download : [scouter-plugin-server-null.jar](https://github.com/scouter-project/scouter-plugin-server-null/releases/download/v1.0/scouter-plugin-server-null.jar)
106106

107+
### 3. Alert scripting (type) plugin
108+
We can build our own alarm rules by handling alert scripting plugins which are able to compose various performance metrics.
109+
* [Alert Plugin Guide](./Alert-Plugin-Guide.md)
107110

108111
## Agent Plugin - Scripting Plugin
109112

110113
#### Java agent plugin
111-
**Scripting plugin can be loaded dynamically on runtime so it is used for debugging also**
112-
It's very useful to print some method's parameters or stacktrace on specific point and also can add additional (user-defined) profile information to the xlog or xlog profile.
113-
The scripting plugin is invoked at some important points and the default location of the plugin file is **./plugin**.
114-
Scouter distribution includes sample plugin files and the file name can not be modified.
114+
**Scripting plugin can be loaded dynamically on runtime so it is used for debugging also**
115+
It's very useful to print some method's parameters or stacktrace on specific point and also can add additional (user-defined) profile information to the xlog or xlog profile.
116+
The scripting plugin is invoked at some important points and the default location of the plugin file is **./plugin**.
117+
Scouter distribution includes sample plugin files and the file name can not be modified.
115118

116119
|filename | desc |
117120
|-------------------|-------------------------|
@@ -121,4 +124,4 @@ Scouter distribution includes sample plugin files and the file name can not be m
121124
|**capture.plug** | Invoked at init, start, end of methods that are set by options ```hook_method_patterns``` or ```hook_constructor_patterns``` |
122125
|**jdbcpoolplug** | Invoked at calling DB connection URL |
123126

124-
Refer to the link for details **[Scripting plugin java agent API](JavaAgent-Plugin-Scripting.md)**를 참고한다.
127+
Refer to the link for details **[Scripting plugin java agent API](JavaAgent-Plugin-Scripting.md)**

0 commit comments

Comments
 (0)