Skip to content

Commit 27acefe

Browse files
Lukas955Lukas955
committed
doc: add JSON readme
1 parent 45a3af0 commit 27acefe

File tree

1 file changed

+194
-0
lines changed

1 file changed

+194
-0
lines changed

‎src/plugins/output/json/README.rst‎

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
JSON (output plugin)
2+
====================
3+
4+
The plugin converts IPFIX flow records into JSON format and sends them to remote destinations over
5+
network, stores them into files or prints them on standard output. Some elements (timestamps,
6+
IP/MAC addresses, TCP flags, etc.) can be converted into human readable format.
7+
Others are represented as plain numbers.
8+
9+
Each flow record is converted individually and send to outputs where at least one output
10+
method must be configured. All fields in an IPFIX record are formatted as: <field name>:<value>,
11+
where the *field name* represents a name of a corresponding Information Element as
12+
"<Enterprise Name>:<Information Element Name>" and *value* is a number, string,
13+
boolean or null (if conversion from IPFIX to JSON fails). The enterprise name is a name of an
14+
organization that manages the field, such as IANA, CISCO, etc.
15+
16+
If the field name and type is unknown and "ignoreUnknown" option is disabled, the field name
17+
use format 'enXX:idYY' where XX is an Enterprise Number and YY is an Information Element ID.
18+
See notes for instructions on how to add missing definitions of Information Elements.
19+
20+
Example output
21+
--------------
22+
23+
The following output is for illustration purpose only. Available IPFIX fields depend on
24+
a structure of records that your exporter sends. See documentation of the exporter,
25+
if you want to change available IPFIX fields.
26+
27+
.. code-block:: json
28+
29+
{
30+
"@type": "ipfix.entry",
31+
"iana:octetDeltaCount": 3568720,
32+
"iana:packetDeltaCount": 2383,
33+
"iana:flowStartMilliseconds": "2018-05-11T19:44:29.006Z",
34+
"iana:flowEndMilliseconds": "2018-05-11T19:44:30.364Z",
35+
"iana:ingressInterface": 1,
36+
"iana:ipVersion": 4,
37+
"iana:sourceIPv4Address": "192.168.0.100",
38+
"iana:destinationIPv4Address": "192.168.0.101",
39+
"iana:ipClassOfService": 0,
40+
"iana:ipTTL": 62,
41+
"iana:protocolIdentifier": "TCP",
42+
"iana:tcpControlBits": ".AP.S.",
43+
"iana:sourceTransportPort": 47200,
44+
"iana:destinationTransportPort": 20810,
45+
"iana:egressInterface": 0,
46+
"iana:samplingInterval": 0,
47+
"iana:samplingAlgorithm": 0
48+
}
49+
50+
Example configuration
51+
---------------------
52+
53+
Below you see a complex configuration with all available output options enabled.
54+
Don't forget to remove (or comment) outputs that you don't want to use!
55+
56+
.. code-block:: xml
57+
58+
<output>
59+
<name>JSON output</name>
60+
<plugin>json</plugin>
61+
<params>
62+
<tcpFlags>formatted</tcpFlags>
63+
<timestamp>formatted</timestamp>
64+
<protocol>formatted</protocol>
65+
<ignoreUnknown>true</ignoreUnknown>
66+
<ignoreOptions>true</ignoreOptions>
67+
<nonPrintableChar>true</nonPrintableChar>
68+
69+
<outputs>
70+
<server>
71+
<name>Local server</name>
72+
<port>8000</port>
73+
<blocking>no</blocking>
74+
</server>
75+
76+
<send>
77+
<name>Send to my server</name>
78+
<ip>127.0.0.1</ip>
79+
<port>8000</port>
80+
<protocol>tcp</protocol>
81+
<blocking>no</blocking>
82+
</send>
83+
84+
<print>
85+
<name>Printer to standard output</name>
86+
</print>
87+
88+
<file>
89+
<name>Store to files</name>
90+
<path>/tmp/ipfixcol/flow/%Y/%m/%d/</path>
91+
<prefix>json.</prefix>
92+
<timeWindow>300</timeWindow>
93+
<timeAlignment>yes</timeAlignment>
94+
</file>
95+
</outputs>
96+
</params>
97+
</output>
98+
99+
Parameters
100+
----------
101+
102+
Formatting parameters:
103+
104+
:``tcpFlags``:
105+
Convert TCP flags to common textual representation (formatted, e.g. ".A..S.")
106+
or to a number (raw). [values: formatted/raw, default: formatted]
107+
108+
:``timestamp``:
109+
Convert timestamp to ISO 8601 textual representation (all timestamps in UTC and milliseconds,
110+
e.g. "2018-01-22T09:29:57.828Z") or to a unix timestamp (all timestamps in milliseconds).
111+
[values: formatted/unix, default: formatted]
112+
113+
:``protocol``:
114+
Convert protocol identification to formatted style (e.g. instead 6 writes "TCP") or to a number.
115+
[values: formatted/raw, default: formatted]
116+
117+
:``ignoreUnknown``:
118+
Skip unknown Information Elements (i.e. record fields with unknown name and data type).
119+
If disabled, data of unknown elements are formatted as unsigned integer (the size of the
120+
field ≤ 8 bytes) or binary values. [values: true/false, default: true]
121+
122+
:``ignoreOptions``:
123+
Skip non-flow records used for reporting metadata about IPFIX Exporting and Metering Processes
124+
(i.e. records described by Options Templates). [values: true/false, default: true]
125+
126+
:``nonPrintableChar``:
127+
Ignore non-printable characters (newline, tab, control characters, etc.) in IPFIX strings.
128+
If disabled, these characters are escaped on output. [values: true/false, default: true]
129+
130+
----
131+
132+
Output types: At least one of the following output must be configured. Multiple server/send/file
133+
outputs can be used at the same time if the outputs are not in collision with each other.
134+
135+
:``server``:
136+
TCP (push) server provides data on a local port. Converted records are automatically send to
137+
all clients that are connected to the port. To test the server you can use, for example,
138+
``ncat(1)`` utility: "``ncat <server ip> <port>``".
139+
140+
:``name``: Identification name of the output. Used only for readability.
141+
:``port``: Local port number of the server.
142+
:``blocking``:
143+
Enable blocking on TCP sockets (true/false). If blocking mode is disabled and
144+
a client is not able to retrieve records fast enough, some flow records may be dropped
145+
(only individual clients are affected). On the other hand, if the blocking mode is enabled,
146+
no records are dropped. However, if at least one client is slow, the plugin waits (i.e.
147+
blocks) until data are send. This can significantly slow down the whole collector and other
148+
output plugins because processing records is suspended. In the worst-case scenario,
149+
if the client is not responding at all, the whole collector is blocked! Therefore,
150+
it is usually preferred (and much safer) to disable blocking.
151+
152+
:``send``:
153+
Send records over network to a client. If the destination is not reachable or the client
154+
is disconnected, the plugin drops all records and tries to reconnect every 5 seconds.
155+
As with the server, you can verify functionality using ``ncat(1)`` utility:
156+
"``ncat -lk <local ip> <local port>``"
157+
158+
:``name``: Identification name of the output. Used only for readability.
159+
:``ip``: IPv4/IPv6 address of the client
160+
:``port``: Remote port number
161+
:``protocol``: Transport protocol: TCP or UDP (this field is case insensitive)
162+
:``blocking``:
163+
Enable blocking on a socket (true/false). See the description of this property at the
164+
server output.
165+
166+
:``file``:
167+
Store data to files.
168+
169+
:``name``: Identification name of the output. Used only for readability.
170+
:``path``:
171+
The path specifies storage directory for data collected by the plugin. Format specifiers
172+
for day, month, etc. are supported so you can create suitable directory hierarchy.
173+
See "strftime" for conversion specification. (Note: UTC time)
174+
:``prefix``: Specifies name prefix for output files.
175+
:``timeWindow``:
176+
Specifies the time interval in seconds to rotate files [minimum 60, default 300]
177+
:``timeAlignment``:
178+
Align file rotation with next N minute interval (yes/no).
179+
180+
:``print``:
181+
Write data on standard output.
182+
183+
:``name``: Identification name of the output. Used only for readability.
184+
185+
Notes
186+
-----
187+
188+
If one or more Information Element definitions are missing, you can easily add them.
189+
All definitions are provided by libfds library. See the project website for help.
190+
191+
For higher performance, it is advisable to use non-formatted conversion of IPFIX data types.
192+
If performance matters, you should prefer, for example, timestamps as numbers over ISO 8601 strings.
193+
194+
Structured data types (basicList, subTemplateList, etc.) are not supported yet.

0 commit comments

Comments
 (0)