Skip to content

Commit cf7bf99

Browse files
DebashisBorgohainO2DebashisBorgohainO2
committed
update doc: functions
1 parent 276b8e7 commit cf7bf99

File tree

2 files changed

+88
-77
lines changed

2 files changed

+88
-77
lines changed

‎docs/user-guide/functions/functions-in-openobserve.md

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# Functions
2+
3+
## What are functions?
4+
5+
Functions in OpenObserve are defined using Vector Remap Language ([VRL](https://vrl.dev)) and can be used during ingestion or query to aid advanced capabilities like enrichment, redaction, log reduction, compliance, etc.
6+
7+
There are also inbuilt query functions like `match_all` etc which can be used for full text search based on user's settings for stream or default settings. Please refer [SQL functions reference](../../sql_reference.md) for complete list of inbuilt functions.
8+
9+
To navigate to functions in OpenObserve, select preferred organization using organization selection control, then click on `Pipelines > Functions` menu, which will take you to functions list screen. This screen lists all the functions for selected organization.
10+
11+
<kbd>
12+
![Functions](./images/functions_list.webp)
13+
</kbd>
14+
15+
List screen details:
16+
17+
- Search in listed functions
18+
- Create new function
19+
- Name of existing function
20+
- Action — update or delete function
21+
22+
There are two ways to use function during query:
23+
24+
- Function with row as input
25+
- Function with specified input columns/fields
26+
27+
28+
## Function with row as input
29+
30+
On logs search page, you can select existing function or write new function using vrl function editor to apply function on row. The returned results will be based on function being applied.
31+
32+
Please note that functions on rows can be used to experiment with result of function application on a specific stream , however applying functions at query time is costly operation .Hence if applicable, after exploration and desired outcome of function during query time , we encourage users to apply such function at ingest time by associating function with stream.
33+
34+
<kbd>
35+
![Functions](./images/functions_logs.webp)
36+
</kbd>
37+
38+
## Function with specified input columns/fields
39+
These are like sql functions, which are defined by user and act on specified input columns/fields.
40+
41+
## Example
42+
Let's try a function on logs page to parse vpc flow logs ,mentioned below is sample vpc flow log record in OpenObserve.
43+
44+
```json
45+
{
46+
"_timestamp": 1683089619868496,
47+
"message": "2 058694856476 eni-03c0f5ba79a66ef17 10.3.166.71 10.3.35.163 443 53672 6 49 12973 1680838556 1680838578 ACCEPT OK"
48+
}
49+
```
50+
Create a vrl function which retains the `_timestamp` field from original record and parses `message` field to multiple fields like `account_id`, `action` etc:
51+
52+
```ruby
53+
ts = ._timestamp # store value of _timestamp in ts
54+
. = parse_aws_vpc_flow_log!(.message) # assign value of object resulting from parse_aws_vpc_flow_log to current record
55+
._timestamp = ts #set value of _timestamp from ts
56+
. # return record
57+
```
58+
59+
The function outputs the record below:
60+
```json
61+
{
62+
"_timestamp": 1683097426943815,
63+
"account_id": 58694856476,
64+
"action": "ACCEPT",
65+
"bytes": 12973,
66+
"dstaddr": "10.3.35.163",
67+
"dstport": 53672,
68+
"end": 1680838578,
69+
"interface_id": "eni-03c0f5ba79a66ef17",
70+
"log_status": "OK",
71+
"packets": 49,
72+
"protocol": 6,
73+
"srcaddr": "10.3.166.71",
74+
"srcport": 443,
75+
"start": 1680838556,
76+
"version": 2
77+
}
78+
```
79+
80+
The function can be saved using save button on top of vrl function editor, additional one can select existing function to try.
81+
82+
The same function can be associated with a stream to get applied at ingestion.

‎docs/user-guide/functions/index.md

Lines changed: 6 additions & 77 deletions
Original file line numberDiff line numberDiff line change
@@ -1,83 +1,12 @@
11
# Functions
22

3-
## What are functions?
3+
In OpenObserve, functions are VRL (Vector Remap Language) scripts that transform your stream data. You create and manage functions via the OpenObserve UI or API. In the UI, go to **Pipelines > Functions** to open a built-in editor where you can write, test, and save VRL code. Once saved, a function can be associated with a stream so it runs on incoming data at ingest time.
44

5-
Functions in OpenObserve are defined using Vector Remap Language ([VRL](https://vrl.dev)) and can be used during ingestion or query to aid advanced capabilities like enrichment, redaction, log reduction, compliance, etc.
5+
**Learn more**:
66

7-
There are also inbuilt query functions like `match_all` etc which can be used for full text search based on user's settings for stream or default settings. Please refer [SQL functions reference](../../sql_reference.md) for complete list of inbuilt functions.
7+
- [Functions in OpenObserve](functions-in-openobserve.md)
8+
- [Create and Manage Functions Using API](../../api/function/)
89

9-
To navigate to functions in OpenObserve, select preferred organization using organization selection control, then click on `Pipelines > Functions` menu, which will take you to functions list screen. This screen lists all the functions for selected organization.
10+
**Related link**:
1011

11-
<kbd>
12-
![Functions](./images/functions_list.webp)
13-
</kbd>
14-
15-
List screen details:
16-
17-
- Search in listed functions
18-
- Create new function
19-
- Name of existing function
20-
- Action — update or delete function
21-
22-
There are two ways to use function during query:
23-
24-
- Function with row as input
25-
- Function with specified input columns/fields
26-
27-
To use functions during data ingestion please refer section :[Stream Association](./stream-association.md)
28-
29-
## Function with row as input
30-
31-
On logs search page, you can select existing function or write new function using vrl function editor to apply function on row. The returned results will be based on function being applied.
32-
33-
Please note that functions on rows can be used to experiment with result of function application on a specific stream , however applying functions at query time is costly operation .Hence if applicable, after exploration and desired outcome of function during query time , we encourage users to apply such function at ingest time by [associating function with stream](./stream-association.md).
34-
35-
<kbd>
36-
![Functions](./images/functions_logs.webp)
37-
</kbd>
38-
39-
## Function with specified input columns/fields
40-
These are like sql functions, which are defined by user and act on specified input columns/fields.
41-
42-
## Example
43-
Let's try a function on logs page to parse vpc flow logs ,mentioned below is sample vpc flow log record in OpenObserve.
44-
45-
```json
46-
{
47-
"_timestamp": 1683089619868496,
48-
"message": "2 058694856476 eni-03c0f5ba79a66ef17 10.3.166.71 10.3.35.163 443 53672 6 49 12973 1680838556 1680838578 ACCEPT OK"
49-
}
50-
```
51-
Create a vrl function which retains the `_timestamp` field from original record and parses `message` field to multiple fields like `account_id`, `action` etc:
52-
53-
```ruby
54-
ts = ._timestamp # store value of _timestamp in ts
55-
. = parse_aws_vpc_flow_log!(.message) # assign value of object resulting from parse_aws_vpc_flow_log to current record
56-
._timestamp = ts #set value of _timestamp from ts
57-
. # return record
58-
```
59-
60-
The function outputs the record below:
61-
```json
62-
{
63-
"_timestamp": 1683097426943815,
64-
"account_id": 58694856476,
65-
"action": "ACCEPT",
66-
"bytes": 12973,
67-
"dstaddr": "10.3.35.163",
68-
"dstport": 53672,
69-
"end": 1680838578,
70-
"interface_id": "eni-03c0f5ba79a66ef17",
71-
"log_status": "OK",
72-
"packets": 49,
73-
"protocol": 6,
74-
"srcaddr": "10.3.166.71",
75-
"srcport": 443,
76-
"start": 1680838556,
77-
"version": 2
78-
}
79-
```
80-
81-
The function can be saved using save button on top of vrl function editor, additional one can select existing function to try.
82-
83-
The same function can be associated with a stream to get applied at ingestion.
12+
- [SQL References](../../sql_reference.md)

0 commit comments

Comments
 (0)