Skip to content

Doc 3830 #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 44 additions & 44 deletions docs-src/pulsar-sink-core/modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
@@ -1,52 +1,52 @@
* Getting Started
** xref:index.adoc[]
*** xref:pulsarSizing.adoc[]
** link:https://github.com/datastax/release-notes/blob/master/DataStax_Apache_Pulsar_Connector_1.4_Release_Notes.md[Pulsar Connector release notes]
** xref:pulsarInstall.adoc[]
** xref:pulsarQuickStart.adoc[]
*** xref:pulsar-sizing.adoc[]
** link:https://github.com/datastax/release-notes/blob/master/DataStax_Apache_pulsar-_Connector_1.4_Release_Notes.md[pulsar- Connector release notes]
** xref:pulsar-install.adoc[]
** xref:pulsar-quickstart.adoc[]
* Guides and examples
** Configuration
*** xref:cfgPulsarConfigTasksMax.adoc[]
*** xref:cfgPulsarColumnWritetimeTimestamp.adoc[]
*** xref:cfgPulsarSettingRowLevelTtl.adoc[]
*** xref:cfgPulsarSettingJavaDriverConfig.adoc[]
*** xref:cfgPulsarMapTopicTable.adoc[]
**** xref:cfgPulsarDisplayingTopic.adoc[]
**** xref:cfgPulsarMapKeyPair.adoc[]
**** xref:cfgPulsarMapJson.adoc[]
***** xref:cfgPulsarStringJson.adoc[]
***** xref:cfgPulsarJsonMessageSchema.adoc[]
**** xref:cfgPulsarMapAvroMessages.adoc[]
**** xref:cfgPulsarRecordHeaderToTable.adoc[]
**** xref:cfgPulsarMapUdt.adoc[]
**** xref:cfgPulsarMapMultipleTables.adoc[]
**** xref:cfgPulsarMultTopicToTables.adoc[]
**** xref:cfgPulsarCqlQuery.adoc[]
**** xref:cfgPulsarNowFunction.adoc[]
** Configuration
*** xref:configure-tasks-max.adoc[]
*** xref:configure-column-writetime-timestamp.adoc[]
*** xref:configure-row-level-ttl.adoc[]
*** xref:configure-java-driver-config.adoc[]
*** xref:configure-map-topic-table.adoc[]
**** xref:configure-displaying-topic.adoc[]
**** xref:configure-map-key-pair.adoc[]
**** xref:configure-map-json.adoc[]
***** xref:configure-string-json.adoc[]
***** xref:configure-json-message-schema.adoc[]
**** xref:configure-map-avro-messages.adoc[]
**** xref:configure-record-header-to-table.adoc[]
**** xref:configure-map-udt.adoc[]
**** xref:configure-map-multiple-tables.adoc[]
**** xref:configure-multiple-topics-to-tables.adoc[]
**** xref:configure-cql-query.adoc[]
**** xref:configure-now-function.adoc[]
** Operations
*** xref:opsPulsarAboutOperations.adoc[]
*** xref:opsPulsarScaling.adoc[]
*** xref:opsPulsarChangeSchema.adoc[]
*** xref:opsPulsarRestart.adoc[]
*** xref:opsPulsarDisplayConfig.adoc[]
*** xref:opsPulsarUpdateConfig.adoc[]
*** xref:opsPulsarDeleteConnector.adoc[]
*** xref:opsPulsarConnectorStatus.adoc[]
*** xref:about-operations.adoc[]
*** xref:scaling.adoc[]
*** xref:change-schema.adoc[]
*** xref:restart.adoc[]
*** xref:display-config.adoc[]
*** xref:update-config.adoc[]
*** xref:delete-connector.adoc[]
*** xref:connector-status.adoc[]
** Security
*** xref:secPulsarInternalLdapAuth.adoc[]
** xref:monPulsarConnectMetrics.adoc[]
*** xref:internal-ldap-auth.adoc[]
** xref:connect-metrics.adoc[]
** Troubleshooting
*** xref:trbPulsarTsRecordFailsToWriteToTable.adoc[]
*** xref:trbPulsarTsLargeBatches.adoc[]
*** xref:trbPulsarTsRecordParsingFails.adoc[]
*** xref:trbPulsarTsLoadBalancing.adoc[]
*** xref:record-fails-to-write-to-table.adoc[]
*** xref:large-batches.adoc[]
*** xref:record-parsing-fails.adoc[]
*** xref:load-balancing.adoc[]
* Reference
**** xref:cfgRefPulsarConnector.adoc[]
**** xref:cfgRefPulsarDseConnection.adoc[]
**** xref:cfgRefPulsarDseTable.adoc[]
**** xref:cfgRefPulsarDates.adoc[]
**** xref:cfgRefPulsarAuth.adoc[]
***** xref:cfgRefPulsarAuthLdap.adoc[]
**** xref:cfgRefPulsarSsl.adoc[]
**** xref:cfgPulsarConfigureErrorHandling.adoc[]
**** xref:reference-pulsar-connector.adoc[]
**** xref:reference-pulsar-dse-connection.adoc[]
**** xref:reference-pulsar-dse-table.adoc[]
**** xref:reference-pulsar-dates.adoc[]
**** xref:reference-pulsar-auth.adoc[]
***** xref:reference-pulsar-auth-ldap.adoc[]
**** xref:reference-pulsar-ssl.adoc[]
**** xref:configure-error-handling.adoc[]

Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= About operating and maintaining the DataStax Connector

:page-aliases: opsPulsarAboutOperations.adoc
:page-tag: pulsar-connector,admin,manage,pulsar

To operate and maintain the DataStax Apache Pulsar™ Connector use the Apache Pulsar™ adminstration tool, `pulsar-admin` in concert with the Apache Pulsar™ REST Administration API.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
= Adding a field to a Pulsar Topic

:page-aliases: opsPulsarAddField.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

Use these steps to write a new field to an existing xref:index.adoc#supported-databases[supported database] table column.

. In the DataStax Apache Pulsar™ Connector configuration file, add the field-to-column definition.
See xref:cfgPulsarMapTopicTable.adoc[].
See xref:configure-map-topic-table.adoc[].
. Update the configuration with the new settings.
See xref:opsPulsarUpdateConfig.adoc[].
See xref:update-config.adoc[].
. Start the Pulsar producer that had the schema change.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Adding a column to the database table

:page-aliases: opsPulsarAddTableColumn.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

Add a column to the xref:index.adoc#supported-databases[supported database] table to synchronize an existing field from a Pulsar topic.
Expand All @@ -10,4 +10,4 @@ Add a column to the xref:index.adoc#supported-databases[supported database] tabl
IMPORTANT: Data synchronization may fail if the schema change is inconsistent across the entire cluster.

. In the DataStax Apache Pulsar™ Connector configuration file, add the field-to-column definition.
See xref:cfgPulsarMapTopicTable.adoc[].
See xref:configure-map-topic-table.adoc[].

This file was deleted.

10 changes: 10 additions & 0 deletions docs-src/pulsar-sink-core/modules/ROOT/pages/change-schema.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
= Changing the topic or table schema
:page-aliases: opsPulsarChangeSchema.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

How to update the DataStax Apache Pulsar™ Connector when schema changes are required.

* xref:add-field.adoc[] Add a field Pulsar Topic schema.
* xref:remove-field.adoc[] Remove a field from a Pulsar Topic.
* xref:add-table-column.adoc[] Add a column to the database table to synchronize an existing field from a Pulsar topic.
* xref:remove-column.adoc[] Remove a database column from the schema.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Specify writetime timestamp column

:page-aliases: cfgPulsarColumnWritetimeTimestamp.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

In the Apache Pulsar™ topic mapping, you can optionally specify which column should be used to note the timestamp when the Pulsar-produced record was inserted into any of the xref:index.adoc#supported-databases[supported databases].
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Provide CQL queries in mappings

:page-aliases: cfgPulsarCqlQuery.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

In the Pulsar topic mapping, you can optionally provide a CQL query that should run when the new record arrives on the Pulsar topic.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Determining topic data structure

:page-aliases: cfgPulsarDisplayingTopic.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

You can use the Apache Pulsar™ `pulsar-client` command line client to examine the structure of Pulsar messages for a topic.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Configure error handling

:page-aliases: cfgPulsarConfigureErrorHandling.adoc
:page-tag: pulsar-connector,admin,manage,pulsar

The DataStax Apache Pulsar™ Connector provides an optional parameter, `ignoreErrors`.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Pass Pulsar Connector settings directly to the DataStax Java driver

:page-aliases: cfgPulsarSettingJavaDriverConfig.adoc
:page-tag: pulsar-connector,dev,develop,pulsar,java

In your DataStax Apache Pulsar™ Connector configuration file, you can directly pass settings to the DataStax Java driver by using the `datastax-java-driver` prefix.
Expand All @@ -17,7 +17,7 @@ The following table identifies functionally equivalent DataStax Apache Pulsar Co
NOTE: If you define both in your configuration, the Pulsar Connector setting take precedence over the `datastax-java-driver.property-name`.
If you do not provide either in your configuration, Pulsar Connector defaults are in effect.

For information about the Java properties, refer to the link:https://docs.datastax.com/en/developer/java-driver-dse/2.3/manual/core/configuration/[DataStax Java driver documentation.] For information about the Pulsar Connector settings, refer to xref:cfgRefPulsarDseConnection.adoc[].
For information about the Java properties, refer to the link:https://docs.datastax.com/en/developer/java-driver-dse/2.3/manual/core/configuration/[DataStax Java driver documentation.] For information about the Pulsar Connector settings, refer to xref:reference-pulsar-dse-connection.adoc[].

|===
| DataStax Apache Pulsar Connector setting | Using datastax-java-driver prefix
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Mapping JSON messages

:page-aliases: cfgPulsarJsonMessageSchema.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

The DataStax Apache Pulsar™ Connector supports mapping JSON messages with or without a schema.
Expand Down Expand Up @@ -31,7 +31,7 @@ See the https://github.com/datastax/pulsar-examples/tree/master/producers/src/ma

. Set up the xref:index.adoc#supported-databases[supported database] table.
. Create the keyspace.
Ensure that keyspace is replicated to a datacenter that is set in the DataStax Apache Pulsar Connector xref:cfgRefPulsarDseConnection.adoc#contactPoints[contactPoints] parameter.
Ensure that keyspace is replicated to a datacenter that is set in the DataStax Apache Pulsar Connector xref:reference-pulsar-dse-connection.adoc#contactPoints[contactPoints] parameter.
For example, create the `stocks_keyspace`:
+
[source,language-bash]
Expand Down Expand Up @@ -67,9 +67,9 @@ nodetool describering -- keyspace\_name

. In the DataStax Apache Pulsar™ Connector configuration file:

.. Add the topic name to xref:cfgRefPulsarDseTable.adoc#topics[topics].
.. Define the topic-to-table map xref:cfgRefPulsarDseTable.adoc#prefix[prefix].
.. Define the xref:cfgRefPulsarDseTable.adoc#mapping[field-to-column map].
.. Add the topic name to xref:reference-pulsar-dse-table.adoc#topics[topics].
.. Define the topic-to-table map xref:reference-pulsar-dse-table.adoc#prefix[prefix].
.. Define the xref:reference-pulsar-dse-table.adoc#mapping[field-to-column map].
+
Example configurations for `stocks_topic` to `stocks_table` using the minimum required settings:
+
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Mapping Avro messages

:page-aliases: cfgPulsarMapAvroMessages.adoc
:page-tag: pulsar-connector,dev,develop,pulsar,avro

The DataStax Apache Pulsar™ Connector supports mapping individual fields from https://avro.apache.org/docs/1.8.1/gettingstartedjava.html[Avro] formatted key or values.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
= Mapping a message that contain JSON fields

:page-aliases: cfgPulsarMapJson.adoc
:page-tag: pulsar-connector,dev,develop,pulsar,json

When the data format for the key or value is JSON, the DataStax Apache Pulsar™ Connector mapping can include individual fields in the JSON structure.

NOTE: JSON records in Pulsar can also have a schema associated with them.

* xref:cfgPulsarStringJson.adoc[] When the data format for the Pulsar key or value is JSON, individual fields of that JSON structure can be specified in the connector mapping.
* xref:cfgPulsarJsonMessageSchema.adoc[] Supports mapping JSON messages with or without a schema.
* xref:configure-string-json.adoc[] When the data format for the Pulsar key or value is JSON, individual fields of that JSON structure can be specified in the connector mapping.
* xref:configure-json-message-schema.adoc[] Supports mapping JSON messages with or without a schema.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Mapping basic messages to table columns

:page-aliases: cfgPulsarMapKeyPair.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

When messages are created using a Basic or primitive serializer, the message contains a key-value pair. Map the key and value to table columns. Ensure that the data types of the message field are compatible with the data type of the target column.
Expand All @@ -17,7 +17,7 @@ Null values are not allowed in PK columns.

. Set up the xref:index.adoc#supported-databases[supported database] table.
. Create the keyspace.
Ensure that keyspace is replicated to a datacenter that is set in the DataStax Apache Pulsar Connector xref:cfgRefPulsarDseConnection.adoc#contactPoints[contactPoints] parameter.
Ensure that keyspace is replicated to a datacenter that is set in the DataStax Apache Pulsar Connector xref:reference-pulsar-dse-connection.adoc#contactPoints[contactPoints] parameter.
For example, create the world_keyspace:
+
[source,language-bash]
Expand All @@ -43,9 +43,9 @@ nodetool describering -- keyspace_name
----

. In the DataStax Apache Pulsar™ Connector configuration file:
.. Add the topic name to xref:cfgRefPulsarDseTable.adoc#topics[topics].
.. Define the topic-to-table map xref:cfgRefPulsarDseTable.adoc#prefix[prefix].
.. Define the xref:cfgRefPulsarDseTable.adoc#mapping[field-to-column map].
.. Add the topic name to xref:reference-pulsar-dse-table.adoc#topics[topics].
.. Define the topic-to-table map xref:reference-pulsar-dse-table.adoc#prefix[prefix].
.. Define the xref:reference-pulsar-dse-table.adoc#mapping[field-to-column map].
+
Example configurations for `world_topic` to `world_table` using the minimum required settings:
+
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Mapping a topic to multiple tables

:page-aliases: cfgPulsarMapMultipleTables.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

The DataStax Apache Pulsar™ Connector allows for mapping a single topic to multiple tables for a single connector instance.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
= Mapping pulsar topics to database tables
:page-aliases: cfgPulsarMapTopicTable.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

DataStax Apache Pulsar™ Connector has a simple yet powerful syntax for mapping fields from a Pulsar record to columns in a xref:index.adoc#supported-databases[supported database] table. An instance of the DataStax Apache Pulsar™ Connector can capture multiple topics and supports mapping a single topic to one or more tables.

== DataStax Apache Pulsar™ Connector topic mapping

DataStax Apache Pulsar™ Connector supports topic-to-table mappings with xref:index.adoc#supported-databases[supported databases]:

* xref:configure-displaying-topic.adoc[] Display messages to determine the data structure of the topic messages.
* xref:configure-map-key-pair.adoc[] Create a topic-table map for Pulsar messages that only contain a key and value in each record.
* xref:configure-map-json.adoc[] For JSON fields, map individual fields in the structure to columns.
* xref:configure-map-avro-messages.adoc[] Supports mapping individual fields from a Avro format field.
* xref:configure-record-header-to-table.adoc[] Extract values from Pulsar record header and write to the database table.
* xref:configure-map-udt.adoc[] Write complex types directly into User-defined Types (UDT).
* xref:configure-map-multiple-tables.adoc[] Ingest a single topic into multiple tables using a single connector instance.
* xref:configure-multiple-topics-to-tables.adoc[] Ingest multiple topics and write to different tables using a single connector instance.
* xref:configure-selective-update-maps-and-udts.adoc[] Selectively update maps and UDTs based on Pulsar fields.
* xref:configure-cql-query.adoc[] Provide CQL queries when new record arrives on the Pulsar topic.
* xref:configure-now-function.adoc[] You can use the now() function in mappings.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Mapping messages to table that has a User Defined Type

:page-aliases: cfgPulsarMapUdt.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

The DataStax Apache Pulsar™ Connector processes JSON, Avro, and complex types directly into a link:https://docs.datastax.com/en/dse/6.8/cql/cql/cql_using/useAlterType.html[user-defined type] column in the xref:index.adoc#supported-databases[supported database]. The Pulsar message embedded field names and the UDT field names must match.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Multiple topics to multiple tables

:page-aliases: cfgPulsarMultTopicToTables.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

The DataStax Apache Pulsar™ Connector allows for mapping multiple topics to multiple tables in a single connector instance.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= The now() function in mappings

:page-aliases: cfgPulsarNowFunction.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

In the Apache Pulsar™ topic mapping, you can use the ``now()``function.
Expand Down Expand Up @@ -38,7 +38,7 @@ In the example above, each occurrence of `now()` is evaluated and return a diffe

== Behavior of now() with DELETE records

When the value of a record arrives to DataStax Apache Pulsar Connector instance, it could be a `DELETE`, assuming `deletesEnabled` is `true`; see xref:cfgRefPulsarDseTable.adoc#deletesEnabled[deletesEnabled].
When the value of a record arrives to DataStax Apache Pulsar Connector instance, it could be a `DELETE`, assuming `deletesEnabled` is `true`; see xref:reference-pulsar-dse-table.adoc#deletesEnabled[deletesEnabled].

For example, consider a case where a user has mapping and used the `now()` function:

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Extract Pulsar record header values

:page-aliases: cfgPulsarRecordHeaderToTable.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

In the Apache Pulsar™ topic mapping, you can extract values from the message properties, and write the values to a xref:index.adoc#supported-databases[supported database table].
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Setting row-level TTL values from Pulsar fields

:page-aliases: cfgPulsarSettingRowLevelTtl.adoc
:page-tag: pulsar-connector,dev,develop,pulsar

In the Apache Pulsar™ topic mapping, you can optionally specify which column should be used as the ttl (time-to-live) of the record being inserted into the database table.
Expand Down
Loading