cleanup-bigquery

Author: mendonk

modules/pulsar-io/examples/connectors/sinks/bigquery/config.csv

@@ -1,15 +1,15 @@
 Name,Required,Default,Description
-kafkaConnectorSinkClass,true,,"A kafka-connector sink class to use. Unless you've developed your own, use the value ""com.wepay.kafka.connect.bigquery.BigQuerySinkConnector""."
+kafkaConnectorSinkClass,true,,"A Kafka-connector sink class to use. Unless you've developed your own, use the value ""com.wepay.kafka.connect.bigquery.BigQuerySinkConnector""."
 offsetStorageTopic,true,,Pulsar topic to store offsets at. This is an additional topic to your topic with the actual data going to BigQuery.
-sanitizeTopicName,true,,"Some connectors cannot handle pulsar topic names like persistent://a/b/topic and do not sanitize the topic name themselves. If enabled, all non alpha-digital characters in topic name will be replaced with underscores. In some cases it may result in topic name collisions (topic_a and topic.a will become the same)
+sanitizeTopicName,true,,"Some connectors cannot handle Pulsar topic names like persistent://a/b/topic and do not sanitize the topic name themselves. If enabled, all non alpha-digital characters in topic name will be replaced with underscores. In some cases this may result in topic name collisions (topic_a and topic.a will become the same)
 
 This value MUST be set to `true`. Any other value will result in an error."
-topic,true,,The Kafka topic name that passed to kafka sink.
 batchSize,false,16384,Size of messages in bytes the sink will attempt to batch messages together before flush.
-collapsePartitionedTopics,false,false,Supply kafka record with topic name without -partition- suffix for partitioned topics.
-kafkaConnectorConfigProperties,false,{},A key/value map of config properties to pass to the kafka connector. See the reference table below.
+collapsePartitionedTopics,false,false,Supply Kafka record with topic name without -partition- suffix for partitioned topics.
+kafkaConnectorConfigProperties,false,{},A key/value map of config properties to pass to the Kafka connector. See the reference table below.
 lingerTimeMs ,false,2147483647L,Time interval in milliseconds the sink will attempt to batch messages together before flush.
 maxBatchBitsForOffset,false,12,Number of bits (0 to 20) to use for index of message in the batch for translation into an offset. 0 to disable this behavior (Messages from the same batch will have the same offset which can affect some connectors.)
+topic,true,,The Kafka topic name that is passed to the Kafka sink.
 unwrapKeyValueIfAvailable ,false,true,In case of Record<KeyValue<>> data use key from KeyValue<> instead of one from Record.
 useIndexAsOffset,false,true,"Allows use of message index instead of message sequenceId as offset, if available. Requires AppendIndexMetadataInterceptor and exposingBrokerEntryMetadataToClientEnabled=true on brokers."
 useOptionalPrimitives,false,false,"Pulsar schema does not contain information whether the Schema is optional, Kafka's does. This provides a way to force all primitive schemas to be optional for Kafka."

modules/pulsar-io/examples/connectors/sinks/bigquery/kafkaConnectorConfigProperties.csv

@@ -1,17 +1,4 @@
 Name,Required,Default,Description
-defaultDataset,true,,The default dataset to be used
-keyfile,true,,"Can be either a string representation of the Google credentials file or the path to the Google credentials file itself.
-
-When using the Astra Streaming UI, the string representation must be used. But if using pulsar-admin with Astra Streaming, either the representation or file can be used."
-keySource,true,FILE,"Determines whether the keyfile configuration is the path to the credentials JSON file or to the JSON itself. Available values are `FILE` and `JSON`.
-
-When using the Astra Streaming UI, JSON will be the only option. But if using pulsar-admin with Astra Streaming, either the representation or file can be used."
-name,true,,The name of the connector. Use the same value as Pulsar sink name.
-project,true,,The BigQuery project to write to
-sanitizeTopics,true,false,"Designates whether to automatically sanitize topic names before using them as table names. If not enabled, topic names are used as table names.
-
-The only accepted value is `false`. Providing any other value will result in an error."
-topics,true,,"A list of Kafka topics to read from. Use the same name as the Pulsar topic (not the whole address, just the topic name)."
 allBQFieldsNullable,false,false,"If `true`, no fields in any produced BigQuery schema are REQUIRED. All non-nullable Avro fields are translated as NULLABLE (or REPEATED, if arrays)."
 allowBigQueryRequiredFieldRelaxation,false,false,"If true, fields in BigQuery Schema can be changed from REQUIRED to NULLABLE."
 allowNewBigQueryFields,false,false,"If true, new fields can be added to BigQuery tables during subsequent schema updates."
@@ -25,7 +12,7 @@ If allowSchemaUnionization, allowNewBigQueryFields, and allowBigQueryRequiredFie
 
 The key difference is that with unionization disabled, new record schemas have to be a superset of the table schema in BigQuery.
 
-In general when enabled, allowSchemaUnionization is useful to make things work. For instance, if you’d like to remove fields from data upstream, the updated schemas still work in the connector. Similarly it is useful when different tasks see records whose schemas contain different fields that are not in the table. However note with caution that if allowSchemaUnionization is set and some bad records are in the topic, the BigQuery schema may be permanently changed. This presents two issues: first, since BigQuery doesn’t allow columns to be dropped from tables, they’ll add unnecessary noise to the schema. Second, since BigQuery doesn’t allow column types to be modified, they could completely break pipelines down the road where well-behaved records have schemas whose field names overlap with the accidentally-added columns in the table, but use a different type."
+allowSchemaUnionization is a useful tool to make things work. For example, if you’d like to remove fields from data upstream, the updated schemas still work in the connector. It is similarly useful when different tasks see records whose schemas contain different fields that are not in the table. However, note with caution that if allowSchemaUnionization is set and some bad records are in the topic, the BigQuery schema may be permanently changed. This presents two issues: first, since BigQuery doesn’t allow columns to be dropped from tables, they’ll add unnecessary noise to the schema. Second, since BigQuery doesn’t allow column types to be modified, they could completely break pipelines down the road where well-behaved records have schemas whose field names overlap with the accidentally-added columns in the table, but use a different type."
 autoCreateBucket,false,true,"Whether to automatically create the given bucket, if it does not exist."
 autoCreateTables,false,false,Automatically create BigQuery tables if they don’t already exist
 avroDataCacheSize,false,100,The size of the cache to use when converting schemas from Avro to Kafka Connect.
@@ -36,16 +23,28 @@ bigQueryRetry,false,0,The number of retry attempts made for a BigQuery request t
 bigQueryRetryWait,false,1000,"The minimum amount of time, in milliseconds, to wait between retry attempts for a BigQuery backend or quota exceeded error."
 clusteringPartitionFieldNames,false,,Comma-separated list of fields where data is clustered in BigQuery.
 convertDoubleSpecialValues,false,false,Designates whether +Infinity is converted to Double.MAX_VALUE and whether -Infinity and NaN are converted to Double.MIN_VALUE to ensure successfull delivery to BigQuery.
+defaultDataset,true,,The default dataset to be used
 deleteEnabled,false,false,"Enable delete functionality on the connector through the use of record keys, intermediate tables, and periodic merge flushes. A delete will be performed when a record with a null value (that is–a tombstone record) is read. This feature will not work with SMTs that change the name of the topic."
 enableBatchLoad,false,“”,Beta Feature Use with caution. The sublist of topics to be batch loaded through GCS.
 gcsBucketName,false,"""”",The name of the bucket where Google Cloud Storage (GCS) blobs are located. These blobs are used to batch-load to BigQuery. This is applicable only if `enableBatchLoad` is configured.
 includeKafkaData,false,false,"Whether to include an extra block containing the Kafka source topic, offset, and partition information in the resulting BigQuery rows."
 intermediateTableSuffix,false,“.tmp”,"A suffix that will be appended to the names of destination tables to create the names for the corresponding intermediate tables. Multiple intermediate tables may be created for a single destination table, but their names will always start with the name of the destination table, followed by this suffix, and possibly followed by an additional suffix."
 kafkaDataFieldName,false,,"The Kafka data field name. The default value is null, which means the Kafka Data field will not be included."
 kafkaKeyFieldName,false,,"The Kafka key field name. The default value is null, which means the Kafka Key field will not be included."
+keyfile,true,,"Can be either a string representation of the Google credentials file or the path to the Google credentials file itself.
+
+When using the Astra Streaming UI, the string representation must be used. If using pulsar-admin with Astra Streaming, either the representation or file can be used."
+keySource,true,FILE,"Determines whether the keyfile configuration is the path to the credentials JSON file or to the JSON itself. Available values are `FILE` and `JSON`.
+
+When using the Astra Streaming UI, JSON will be the only option. If using pulsar-admin with Astra Streaming, either the representation or file can be used."
+name,true,,The name of the connector. Use the same value as Pulsar sink name.
 mergeIntervalMs,false,60_000L,"How often (in milliseconds) to perform a merge flush, if upsert/delete is enabled. Can be set to -1 to disable periodic flushing."
 mergeRecordsThreshold,false,-1,"How many records to write to an intermediate table before performing a merge flush, if upsert/delete is enabled. Can be set to -1 to disable record count-based flushing."
+project,true,,The BigQuery project to write to
 queueSize,false,-1,The maximum size (or -1 for no maximum size) of the worker queue for BigQuery write requests before all topics are paused. This is a soft limit; the size of the queue can go over this before topics are paused. All topics resume once a flush is triggered or the size of the queue drops under half of the maximum size.
+sanitizeTopics,true,false,"Designates whether to automatically sanitize topic names before using them as table names. If not enabled, topic names are used as table names.
+
+The only accepted value is `false`. Providing any other value will result in an error."
 schemaRetriever,false,com.wepay.kafka.connect.bigquery.retrieve.IdentitySchemaRetriever,A class that can be used for automatically creating tables and/or updating schemas.
 threadPoolSize,false,10,The size of the BigQuery write thread pool. This establishes the maximum number of concurrent writes to BigQuery.
 timePartitioningType,false,DAY,"The time partitioning type to use when creating tables. Existing tables will not be altered to use this partitioning type. Valid Values: (case insensitive) [MONTH, YEAR, HOUR, DAY]"
@@ -56,6 +55,7 @@ Format: comma-separated tuples, e.g. <topic-1>:<table-1>,<topic-2>:<table-2>,..
 
 Note, because `sanitizeTopicName` must be `true`, that in-turn means any alphanumeric character in the topic name will be replaced as underscore “_”. So when creating a mapping you need to take the underscores into account.
 
-In example, if the topic name is provided as “persistent://a/b/c-d” then the mapping topic name would be “persistent___a_b_c_d”.
-"
+For example, if the topic name is provided as “persistent://a/b/c-d” then the mapping topic name would be “persistent___a_b_c_d”.
+
+topics,true,,"A list of Kafka topics to read from. Use the same name as the Pulsar topic (not the whole address, just the topic name)."
 upsertEnabled,false,false,"Enable upsert functionality on the connector through the use of record keys, intermediate tables, and periodic merge flushes. Row-matching will be performed based on the contents of record keys. This feature won’t work with SMTs that change the name of the topic."

modules/pulsar-io/pages/connectors/sinks/google-bigquery.adoc

@@ -31,7 +31,7 @@ include::partial$connectors/sinks/monitoring.adoc[]
 
 == Connector Reference
 
-With the BigQuery Sink there a multiple sets of parameters. First the Astra Streaming parameters, then the Kafka Connect Adapter parameters, and finally the Google BigQuery parameters. Each provide a way to coordinate how data will be streamed from Pulsar to BigQuery.
+The BigQuery sink has multiple sets of parameters: the Astra Streaming parameters, the Kafka Connect Adapter parameters, and the Google BigQuery parameters. Each set of parameters provides a way to coordinate how data will be streamed from Pulsar to BigQuery.
 
 === Astra Streaming
 

modules/pulsar-io/partials/connectors/sinks/monitoring.adoc

@@ -62,4 +62,4 @@ include::partial$connectors/sinks/curl-status-response.adoc[]
 
 === Metrics
 
-Astra Streaming exposes Prometheus formatted metrics for every connector. Refer to xref:astra-streaming:operations:astream-scrape-metrics.adoc[scrape metrics with Prometheus] page for more detail.
+Astra Streaming exposes Prometheus formatted metrics for every connector. Refer to the xref:astra-streaming:operations:astream-scrape-metrics.adoc[scrape metrics with Prometheus] page for more detail.