Databus 2.0 Client Design

Introduction

Databus Clients are responsible for

• Reading
• Listening

Architecture

Here is a high-level overview of the client.

Relay Connection

The connection used for obtaining real-time update stream from the relay using the [relay HTTP interface|https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Databus+v2.0++Protocol#Databusv2.0Protocol-Databus2Relay].

The consumption of events follows the [online stream consumption state machine|https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Databus+Client+State+Machine#DatabusClientStateMachine-Onlinestreamconsumption].

Bootstrap Connection

The connection used for obtaining long look-back updates from the bootstrap server using the [bootstrap HTTP interface|https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Databus+v2.0++Protocol#Databusv2.0Protocol-Databus2Bootstrap];

The consumption of events follows the [bootstrap consumption state machine|https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Databus+Client+State+Machine#DatabusClientStateMachine-Bootsrapconsumption].

Persistence is either local (in a file) or shared (in Zookeeper).

Dispatcher

The dispatcher reads incoming events from online streams or bootstrap and invokes the consumer code callbacks. Its main responsibilities are:

Determine the right callback

Monitor for errors and timeouts

Ensures that the client library persist the consumer progress of event consumption

The dispatcher follows the [state machine|https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Databus+Client+State+Machine#DatabusClientStateMachine-DISPATCHEROVERVIEW].

Consumer Code Callbacks

Consumer code that implements the [Consumer callback API|https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Chapter+II+-+Event+Model+and+Consumer+API#ChapterII-EventModelandConsumerAPI-DatabusConsumersAPI].

Callbacks are executed using the [following execution model|https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Chapter+II+-+Event+Model+and+Consumer+API#ChapterII-EventModelandConsumerAPI-ExecutionModel]. See also the section on callbacks in the [Databus migration wiki|https://iwww.corp.linkedin.com/wiki/cf/download/attachments/41117293/databus2_migration.pptx?version=1&modificationDate=1321480026000]

Checkpoint persistence

The checkpoint is an [internal representation|https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Databus+v2.0++Protocol#Databusv2.0Protocol-FormatofCheckpoint] of the point in the consumption of stream of updates by a consumer.