Forward Protocol Specification v0

Fluentd Forward Protocol Specification (v0)

This is a protocol specification for Fluentd forward input/output plugins. This protocol is also used by fluent-logger software, and many other software in ecosystem (e.g., Docker logging driver for Fluentd).

This protocol version is v0. This spec is supported by Fluentd v0.10 and v0.12, and will be changed at v0.14 (it will be versioned as protocol version v1).

Changes

• March 16, 2016: initial release of this specification
• August 30, 2016: add an optional field: size

Abstract

This specification describes the fluentd forward protocol, which is used to transport events from hosts to hosts over network.

Terminology

The keywords "MUST", "MUST NOT", "SHOULD", "SHOULD NOT" and "MAY" in this document are to be interpreted as described in RFC 2119. The following terms are also used:

• event: a virtual set of tag, time and record. tag MAY be shared by 2 or more events in actual representation if these events have same tag.
• client: an endpoint that sends events. out_forward plugin is the reference client implementation shipped with the fluentd distribution.
• server: an endpoint that receives events. in_forward plugin is the reference server implementation shipped with the fluentd distribution.
• connection: a TCP connection between two endpoints.
• msgpack: a light weight binary representation of serialized objects.

UDP Heartbeat Message

• Client MAY send UDP packets to the same port of connection, to check existence of Servers.
• Server SHOULD listen the same port of connection, and SHOULD respond to source address when heartbeat message arrived.

UDP heartbeat message SHOULD be a byte of 0x00 in both of Client and Server.

Fluentd Forward Client

• Client sends a msgpack array which contains one or more events to the server through TCP connection.
• Client MUST choose a carrier mode from three: Message, Forward and PackedForward described below.
• Client MAY send a msgpack nil value for heartbeat health check usage without any event record payload.
• Client MAY send multiple msgpack arrays on a single connection, continuously.

Fluentd Forward Server

• Server MUST receive one or more msgpack arrays from the client through TCP connection.
• Server MUST detect the carrier mode by inspecting the second element of the array.
• Server SHOULD ignore any request value other than array format.
• Note: In addition to the three carrier modes, in_forward plugin also accepts JSON representation of a single event for convenience. It detect it by the first byte of the request.

Message Mode

It carries just a event.

• tag is a string separated with '.' (e.g. myapp.access) to categorize events.
• time is a number of seconds since Unix epoch.
• record is key-value pairs of the event record.
• option is optional key-value pairs, to bring data to control servers' behavior.

name	Ruby type	msgpack fromat	content
tag	String	str	tag name
time	Integer | EventTime	int | ext	Unix time (second)
record	Hash	map	pairs of keys(String) and values(Object)
option	Hash	map	option (optional)

[
  "tag.name",
  1441588984,
  {"message": "bar"},
  {"option": "optional"}
]

NOTE for v1 protocol: time MAY be a representation of EventTime, which has nanosecond precision of time with msgpack ext format of type 0 is used when time_as_integer is false. Server SHOULD accept both formats.

Forward Mode

It carries a series of events as a msgpack array on a single request.

name	Ruby type	msgpack format	content
tag	String	str	tag name
entries	MultiEventStream	array	list of Entry
option	Object	map	option (optional)

[
  "tag.name",
  [
    [1441588984, {"message": "foo"}],
    [1441588985, {"message": "bar"}],
    [1441588986, {"message": "baz"}]
  ],
  {"option": "optional"}
]

PackedForward Mode

It carries a series of events as a msgpack binary on a single request.

• entries is a binary chunk of MessagePackEventStream which contains multiple raw msgpack representations of Entry.
• Client SHOULD send a MessagePackEventStream as msgpack bin format as it's a binary representation.
• Client MAY send a MessagePackEventStream as msgpack str format for compatibility reasons.
• Server MUST accept both formats of bin and str.
• Server MAY decode individual events on demand but MAY NOT do right after request arrival. It means it MAY costs less, compared to Forward mode, when decoding is not needed by any plugins.
• Note: out_forward plugin sends events by the PackedForward mode. It encloses event records with msgpack str format instead of bin format for a backward compatibility reason.

name	Ruby type	msgpack format	content
tag	String	str	tag name
entries	MessagePackEventStream	str | bin	msgpack stream of Entry
option	Object	map	option (optional)

[
  "tag.name",
  "<<MessagePackEventStream>>",
  {"option": "optional"}
]

Entry

Entry is an array representation of pairs of time and record, used in Forward and PackedForward mode.

name	Ruby type	msgpack fromat	content
time	Integer | EventTime	int | ext	Unix time (second)
record	Hash	map	pairs of keys(String) and values(Object)

Option

It carries an optional meta data for the request.

• Client MAY send key-value pairs of options.
• Server MAY just ignore any options given.
• size: Clients MAY send the size option to show the number of event records in an entries by an integer as a value. Server can know the number of events without unpacking entries (especially for PackedForward mode).
• chunk: Clients MAY send the chunk option to confirm the server receives event records. The value is a string of Base64 representation of 128 bits unique_id which is an ID of a set of events.

{"chunk": "p8n9gmxTQVC8/nh2wlKKeQ==", "size": 1023}

Response

• Server SHOULD close the connection silently with no response when the chunk option is not sent.
• ack: Server MUST respond ack when the chunk option is sent by client. The ack response value MUST be the same value given by chunk option from client. Client SHOULD retry to send events later when the request has a chunk but the response has no ack.

{"ack": "p8n9gmxTQVC8/nh2wlKKeQ=="}

EventTime Ext Format

EventTime uses msgpack extension format of type 0 to carry nanosecond precision of time.

• Client MAY send EventTime instead of plain integer representation of second since unix epoch.
• Server SHOULD accept both formats of integer and EventTime.

+----+----+----+----+----+----+----+----+----+----+
|  1 |  2 |  3 |  4 |  5 |  6 |  7 |  8 |  9 | 10 |
+----+----+----+----+----+----+----+----+----+----+
| C7 | 00 | second from epoch |     nanosecond    |
+----+----+----+----+----+----+----+----+----+----+
|ext |type| 32bits integer BE | 32bits integer BE |
+----+----+----+----+----+----+----+----+----+----+

Grammar

• Name? means that Name is optional.
• Name* means that Name can occur zero or more times.
• <<Name>> means binary msgpack representation of Name.
• [ A, B, C ] means an array.
• nil, string, integer and object means as it is.

Connection ::= <<Request>>*

Request ::= Message | Forward | PackedForward | nil

Message ::= [ Tag, Time, Record, Option? ]

Forward ::= [ Tag, MultiEventStream, Option? ]

MultiEventStream ::= [ Event* ]

PackedForward ::= [ Tag, MessagePackEventStream, Option? ]

MessagePackEventStream ::= <<Event>>*

Event ::= [ Time, Record ]

Tag ::= string

Time ::= integer | EventTime

Record ::= object

Option ::= object