Skip to content

Connector Configuration

Jump to bottom
Chris Riccomini edited this page Oct 7, 2016 · 9 revisions

Configuring your Connector

topics

A list of Kafka topics to read from

  • Type: List of comma-delimited strings
  • Required (no default)

project

The BigQuery project to write to

  • Type: String
  • Required (no default)

datasets

Names for the datasets Kafka topics will write to (form of <topic regex>=<dataset>)

  • Type: List of comma-delimited strings
  • Required (no default)

keyfile

The file containing a JSON key with BigQuery service account credentials

  • Type: String
  • Default: ""

registry

The base URL of a schema registry

  • Type: String
  • Default: ""
  • Required if and only if autoCreateTables or autoUpdateSchemas is enabled

sanitizeTopics

Whether to automatically sanitize topic names before using them as table names; if not enabled topics names will be used directly as table names

  • Type: boolean
  • Default: false

includeKafkaData

Whether to include an extra block containing the Kafka source topic, offset, and partition information in the resulting BigQuery rows

  • Type: boolean
  • Default: false

avroDataCacheSize

The size of the cache to use when converting schemas from Avro to Kafka Connect

  • Type: int
  • Default: 100

autoCreateTables

Automatically create BigQuery tables if they don't already exist

  • Type: boolean
  • Default: false

autoUpdateSchemas

Whether or not to automatically update BigQuery schemas

  • Type: boolean
  • Default: false

bufferSize

The maximum number of records to buffer per table before temporarily halting the flow of new records, or -1 for unlimited buffering

  • Type: long
  • Default: 100000

bigQueryRetry

The number of retry attempts that will be made per BigQuery request that fails with a backend error

  • Type: int
  • Default: 0

bigQueryRetryWait

The amount of time, in milliseconds to wait between BigQuery backend error retries

  • Type: long
  • Default: 1,000

topicsToTables

A list of mappings from topic regexes to table names. Note the regex must include capture groups that are referenced in the format string using placeholders (i.e. $1) (form of =).

  • Type: List of comma-delimited strings
  • Default null
Clone this wiki locally