Databus 2.0 event buffer design

Databus 2.0 Event Buffer Design

(Please note that the Event Buffer design and the APIs used to access it are being reworked as this is written. The description below reflects the “old” design and APIs, with a handful of annotations noting where changes are likely.)

DbusEventBuffer

As mentioned in the design document, the Event Buffer is a circular buffer implemented by the DbusEventBuffer class. It is currently implemented as an array of [[http://docs.oracle.com/javase/1.4.2/docs/api/java/nio/ByteBuffer.html][ByteBuffer]] objects. Each ByteBuffer is sized to a maximum of 2G and may be mmapped (depending on the policy). Events are stored in the ByteBuffer array in the transport format (see the DbusEvent section immediately below). The DbusEventBuffer class takes care of keeping track of the right offset for an event – the offset includes the individual ByteBuffer and the offset within the ByteBuffer where the event starts.

DbusEvent

This class represents a serialized Databus event stored in DbusEventBuffer. See the generated Javadoc for DbusEvent for details regarding representation, or see DbusEvent.pdf.

Binary serialization

See the generated Javadoc for the DbusEvent class.

JSON serialization

See the generated Javadoc for the DbusEvent class.

The JSON event format is as follows:

{
"key":90454270,
"scn":10,
"windowScn":10,
"partitionId":17350,
"timestamp":1289410594622,
"srcId":1,
"schemaId":"YWJjZGVmZ2hpamtsbW5vcA==",
"valueEnc":"JSON_PLAIN_VALUE",
"value":"m7Wkm4l8dkZ6JT2UrNsiothOz9HsvsReb5SKbEPfhHkMneYTSAPbHeBV5aiB5l1TBG4bKizGnA6QY4OwlXl0gBa9Bgk0Nn0Y4nqbeI8STM1qSAuxbPY6mgAHHb4pGgREN0keWhTdW6X7Jy2jr1F4LiiET64A8ltDf8927oauFLHYHBbg3y425OuN1vvhVlMzvbXDTzkKYuffQv6bOproQamkFnO2KUFzPCCv48OXYhtWdXfPtDYSRO5j35z4PHV6W81lprNEggTyyd2Iucd3nYxrhGGtCP3EjoGuCAEXW6TfTr3EbfxMjGMUsrc9EYIGAZ650Ry85HAuFlhqrwgHoknjBHbRT1wjRyVdpyo6kfaV4nRpASii9Kz5wHEvvrkLLCmxLXiWofTCL9tb6dxk7JcYAAxXFFfkS6M9Rmp2TVpAGZJhRoDahK4JDVw1hE6To7PlxYM5akIInPw33P5kEVAqKlfRxkK1JwUuLwdmqNsMbPICafFiUxMqsRnDap28NAAZ3etxj1TsdfYTcsnIz9GDi8fvk3EBMUjydPcVknwz8ZZ6t335ivz47vtVJH1k21fReN2uaktBdb56ndLsZwxyZwPHpWqQlBTDLICQXJkk3Sh5u5JxJu7tY2xJfSnUEXvqAQhvqZq3XisHAiAWcRvQxkrfRe2GJP8CrkpfEr3xz0lzcZWyYPk1TX6yXMQ7eJYWcUdsRodi1nzI0DzBQ6t6rp2okfTreC1rf5cxwhGyCNVZ4udnFq0SipvVwZCMqa1TxZZjYZIKfl3uLL0KmRSSc1bkCVQBsJNmGAl1PNhIEmwP60xcs3cHS6eYeD3PZMbBUWNdkrlMukQZNIKhioNXM1h10fik8KcWI1PdyqFXijBDPjBH30YQZLcbULFbyi5CqkcceqDOrDdm94sSd1YH0hF1ulsViDzH5dxgiCR9TMJ6u9rGMfs1vKMbjF4WEO4ttBJJcebKgc2RVlBPjUqj6xBXLzmvEwJn8EJp"
}

Adding (Writing) Events

The DbusEventBuffer class provides a couple of different ways of adding events to it:

1. You can use a combination of one call to startEvents() to begin a transaction, followed by one or more calls to appendEvent() to write individual events, followed by a single call to endEvents() to commit the transaction. (Or you may call rollbackEvents() instead of endEvents() to abort the transaction.) This is likely to be simplified in the redesign.
2. You can use the readEvents() batch interface. (“Read” in this context refers to the source database; from the buffer’s perspective, it’s a write call. This will be addressed in the redesign.)

See the generated Javadoc for DbusEventBuffer for details.

Reading Events

The DbusEventBuffer class provides two ways to read data from a buffer:

1. You can use streamEvents() API to source events directly into a [[http://docs.oracle.com/javase/1.4.2/docs/api/java/nio/channels/WritableByteChannel.html][WritableByteChannel]].
2. Alternatively, you can use the DbusEventIterator class to to iterate over the events in a DbusEventBuffer one at a time.