Skip to content

Databus 2.0 Client Design

Jump to bottom
phanindraganti edited this page Jan 16, 2013 · 11 revisions

Introduction

Databus Clients are responsible for

  • Reading
  • Listening

Architecture

Here is a high-level overview of the client.

Relay Connection

The connection used for obtaining real-time update stream from the relay using the https://github.com/linkedin/databus/wiki/Databus-2.0-protocol.

The consumption of events follows the https://github.com/linkedin/databus/wiki/Databus+Client+State+Machine#DatabusClientStateMachine.

Bootstrap Connection

The connection used for obtaining long look-back updates from the bootstrap server using the https://github.com/linkedin/databus/wiki/Databus+v2.0++Protocol#Databusv2.0Protocol-Databus2Bootstrap

The consumption of events follows the https://github.com/linkedin/databus/wiki/Databus+Client+State+Machine#DatabusClientStateMachine-Bootsrapconsumption.

Persistence is either local (in a file) or shared (in Zookeeper).

Dispatcher

The dispatcher reads incoming events from online streams or bootstrap and invokes the consumer code callbacks. Its main responsibilities are:

  1. Determine the right callback
  2. Monitor for errors and timeouts
  3. Ensures that the client library persist the consumer progress of event consumption

The dispatcher follows the https://github.com/linkedin/databus/wiki/Databus+Client+State+Machine#DatabusClientStateMachine-DISPATCHEROVERVIEW.

Consumer Code Callbacks

Consumer code that implements the https://github.com/linkedin/databus/wiki/Chapter+II+-+Event+Model+and+Consumer+API#ChapterII-EventModelandConsumerAPI-DatabusConsumersAPI.

Callbacks are executed using the [[following execution model][https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Chapter+II+-+Event+Model+and+Consumer+API#ChapterII-EventModelandConsumerAPI-ExecutionModel]. See also the section on callbacks in the [Databus migration wiki|https://iwww.corp.linkedin.com/wiki/cf/download/attachments/41117293/databus2_migration.pptx?version=1&modificationDate=1321480026000]]

Checkpoint persistence

The checkpoint is an https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Databus+v2.0++Protocol#Databusv2.0Protocol-FormatofCheckpoint of the point in the consumption of stream of updates by a consumer.

The default format is a JSON representation of the Checkpoint object.

{"windowOffset":-1,"snapshot_offset":-1,"prevScn":-1,"windowScn":5984508975840,"consumption_mode":"ONLINE_CONSUMPTION"}

Bootstrap checkpoint

{"snapshot_offset":0,"prevScn":5984488321377,"windowScn":5984488321377,"consumption_mode":"BOOTSTRAP_SNAPSHOT","windowOffset":-1,"bootstrap_since_scn":5984488321377,"bootstrap_start_scn":-1,"bootstrap_target_scn":-1,"bootstrap_snapshot_source_index":0,"bootstrap_catchup_source_index":0,"bootstrap_server_info":null,"snapshot_source":"com.linkedin.databus.example.Person"}

Clone this wiki locally