Skip to content

Commit 4f532bd

Browse files
authored
payg and new window (#49)
1 parent ca2a523 commit 4f532bd

35 files changed

+114
-111
lines changed

modules/ROOT/pages/index.adoc

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ A real-time data pipeline is a series of steps that takes data from its original
4141
--
4242
Starlight for JMS allows enterprises to take advantage of the scalability and resiliency of a modern streaming platform to run their existing JMS applications.
4343
44-
xref:streaming-learning:use-cases-architectures:starlight/jms/index.adoc[Get started now] | xref:starlight-for-jms:ROOT:index.adoc[Configuring] | https://github.com/datastax/pulsar-jms[Source Code^]
44+
xref:streaming-learning:use-cases-architectures:starlight/jms/index.adoc[Get started now] | xref:starlight-for-jms:ROOT:index.adoc[Configuring] | https://github.com/datastax/pulsar-jms[Source Code]
4545
--
4646
****
4747

@@ -51,7 +51,7 @@ xref:streaming-learning:use-cases-architectures:starlight/jms/index.adoc[Get sta
5151
--
5252
Starlight for Kafka brings native Apache Kafka® protocol support to Apache Pulsar by introducing a Kafka protocol handler on Pulsar brokers.
5353
54-
xref:streaming-learning:use-cases-architectures:starlight/kafka/index.adoc[Get started now] | xref:starlight-for-kafka:ROOT:index.adoc[Configuring] | https://github.com/datastax/starlight-for-kafka[Source Code^]
54+
xref:streaming-learning:use-cases-architectures:starlight/kafka/index.adoc[Get started now] | xref:starlight-for-kafka:ROOT:index.adoc[Configuring] | https://github.com/datastax/starlight-for-kafka[Source Code]
5555
--
5656
****
5757

@@ -61,7 +61,7 @@ xref:streaming-learning:use-cases-architectures:starlight/kafka/index.adoc[Get s
6161
--
6262
Starlight for RabbitMQ combines the AMQP 0.9.1 API with Pulsar, providing a powerful way to modernize your RabbitMQ infrastructure, improve performance, and reduce costs.
6363
64-
xref:streaming-learning:use-cases-architectures:starlight/rabbitmq/index.adoc[Get started now] | xref:starlight-for-rabbitmq:ROOT:index.adoc[Configuring] | https://github.com/datastax/starlight-for-rabbitmq[Source Code^]
64+
xref:streaming-learning:use-cases-architectures:starlight/rabbitmq/index.adoc[Get started now] | xref:starlight-for-rabbitmq:ROOT:index.adoc[Configuring] | https://github.com/datastax/starlight-for-rabbitmq[Source Code]
6565
--
6666
****
6767

modules/functions/pages/index.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ Custom functions are a powerful feature, but for common data transformations, we
77
DataStax has created the following transform functions for common data transforms, but we're always experimenting with new ones.
88
Check back as the list grows, or let us know some functions you'd find helpful in your deployment.
99

10-
Unqualified orgs can use transform functions without upgrading to a Pay as You Go plan.
10+
Unqualified orgs can use transform functions without upgrading to a Pay As You Go plan.
1111

1212
[#transform-list]
1313
== Transforms

modules/pulsar-io/pages/connectors/index.adoc

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,4 +126,5 @@ curl "$WEB_SERVICE_URL/admin/v3/sources/builtinsources" -H "Authorization: $ASTR
126126

127127
== What's next?
128128

129-
For more on Pulsar IO connectors, see the https://pulsar.apache.org/docs/en/io-overview/[Pulsar documentation^].
129+
For more on Pulsar IO connectors, see the https://pulsar.apache.org/docs/en/io-overview/[Pulsar documentation
130+
].

modules/pulsar-io/pages/connectors/sinks/astra-db.adoc

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,15 +7,15 @@
77

88
= Astra DB (Cassandra Enhanced)
99

10-
DataStax Astra DB Sink Connector is based on the open-source https://docs.datastax.com/en/pulsar-connector/docs/index.html[Cassandra sink connector for Apache Pulsar^]. Depending on how you deploy the connector, it can be used to sink topic messages with a table in Astra DB or a table in a Cassandra cluster outside of DB.
10+
DataStax Astra DB Sink Connector is based on the open-source https://docs.datastax.com/en/pulsar-connector/docs/index.html[Cassandra sink connector for Apache Pulsar]. Depending on how you deploy the connector, it can be used to sink topic messages with a table in Astra DB or a table in a Cassandra cluster outside of DB.
1111

1212
The Astra Streaming portal provides simple way to connect this sink and a table in Astra DB with simply a token. Using pulsar-admin or the REST API, you can configure the sink to connect with a Cassandra connection manually.
1313

1414
This reference assumes you are manually connecting to a Cassandra table.
1515

1616
[TIP]
1717
====
18-
If you would like to see the code, refer to the https://github.com/datastax/pulsar-sink[open source here^].
18+
If you would like to see the code, refer to the https://github.com/datastax/pulsar-sink[open source here].
1919
====
2020

2121
== Get Started
@@ -66,18 +66,18 @@ include::example$connectors/sinks/{connectorType}/auth.csv[]
6666

6767
These values are provided in the *topic* area in the above cassandra connection parameters.
6868

69-
Refer to the official documentation for a https://docs.datastax.com/en/pulsar-connector/docs/cfgRefPulsarDseConnection.html[connection properties reference^].
69+
Refer to the official documentation for a https://docs.datastax.com/en/pulsar-connector/docs/cfgRefPulsarDseConnection.html[connection properties reference].
7070

7171
=== Mapping topic data to table columns
7272

7373
[TIP]
7474
====
75-
There are quite a few examples in the "https://docs.datastax.com/en/pulsar-connector/docs/cfgPulsarMapTopicTable.html[Mapping pulsar topics to database tables^]" area of the official documentation
75+
There are quite a few examples in the "https://docs.datastax.com/en/pulsar-connector/docs/cfgPulsarMapTopicTable.html[Mapping pulsar topics to database tables]" area of the official documentation
7676
====
7777

7878
An essential part of using this sink connector is mapping message values to table columns. There are many factors that influence how this done and what is possible. The 'mapping' string is a simple comma-separated list of column names and message value fields.
7979

80-
While the getting started examples above show how to configure the connector in one large command, it is easier to manage this as a separate file. The following example show how to configure the connector using a configuration in json format. The "https://docs.datastax.com/en/pulsar-connector/docs/pulsarQuickStart.html[Pulsar Connector single instance quick start^]" guide provides a good example of this. Below are the minimum requirements.
80+
While the getting started examples above show how to configure the connector in one large command, it is easier to manage this as a separate file. The following example show how to configure the connector using a configuration in json format. The "https://docs.datastax.com/en/pulsar-connector/docs/pulsarQuickStart.html[Pulsar Connector single instance quick start]" guide provides a good example of this. Below are the minimum requirements.
8181

8282
Create a file named configs.json using the following structure:
8383

modules/pulsar-io/pages/connectors/sinks/cloud-storage.adoc

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ By default, when the connector receives a message with a non-supported schema ty
8585

8686
== Dead-letter topics
8787

88-
To use a dead-letter topic, set `skipFailedMessages` to `false` in the cloud provider config. Then using either pulsar-admin or curl, set `--max-redeliver-count` and `--dead-letter-topic`. For more info about dead-letter topics, see the https://pulsar.apache.org/docs/en/concepts-messaging/#dead-letter-topic[Pulsar documentation^]. If a message fails to be sent to the Cloud Storage sink and there is a dead-letter topic, the connector will send the message to the assigned topic.
88+
To use a dead-letter topic, set `skipFailedMessages` to `false` in the cloud provider config. Then using either pulsar-admin or curl, set `--max-redeliver-count` and `--dead-letter-topic`. For more info about dead-letter topics, see the https://pulsar.apache.org/docs/en/concepts-messaging/#dead-letter-topic[Pulsar documentation]. If a message fails to be sent to the Cloud Storage sink and there is a dead-letter topic, the connector will send the message to the assigned topic.
8989

9090
== Managing the Connector
9191

@@ -152,8 +152,8 @@ include::example$connectors/sinks/cloud-storage/azure-blob.csv[]
152152

153153
== What's next?
154154

155-
Learn more about https://cloud.google.com/storage[Google’s Cloud Storage^].
155+
Learn more about https://cloud.google.com/storage[Google’s Cloud Storage].
156156

157-
Learn more about https://azure.microsoft.com/en-us/products/storage/blobs[Azure Blob Store^].
157+
Learn more about https://azure.microsoft.com/en-us/products/storage/blobs[Azure Blob Store].
158158

159-
Learn more about https://aws.amazon.com/s3/[AWS S3^].
159+
Learn more about https://aws.amazon.com/s3/[AWS S3].

modules/pulsar-io/pages/connectors/sinks/elastic-search.adoc

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,13 @@ Use Elasticsearch to store, search, and manage data for logs, metrics, search ba
1313

1414
[NOTE]
1515
====
16-
Astra Streaming currently supports Apache Pulsar {pulsar_version}, which uses the https://opensearch.org/docs/1.2/clients/java-rest-high-level/[OpenSearch 1.2.4 library^] to interact with
16+
Astra Streaming currently supports Apache Pulsar {pulsar_version}, which uses the https://opensearch.org/docs/1.2/clients/java-rest-high-level/[OpenSearch 1.2.4 library] to interact with
1717
Elasticsearch.
1818
====
1919

2020
[NOTE]
2121
====
22-
To understand compatibility between OpenSearch and Elasticsearch, refer to https://opensearch.org/faq/#q1.8[question 1.8^] in the FAQs.
22+
To understand compatibility between OpenSearch and Elasticsearch, refer to https://opensearch.org/faq/#q1.8[question 1.8] in the FAQs.
2323
====
2424

2525
== Get Started
@@ -49,10 +49,10 @@ include::example$connectors/sinks/astra.csv[]
4949

5050
These values are provided in the "configs" area.
5151

52-
The Astra Streaming Elasticsearch sink connector supports all configuration properties provided by Apache Pulsar. Please refer to the https://pulsar.apache.org/docs/io-elasticsearch-sink/#property[connector properties^] for a complete list.
52+
The Astra Streaming Elasticsearch sink connector supports all configuration properties provided by Apache Pulsar. Please refer to the https://pulsar.apache.org/docs/io-elasticsearch-sink/#property[connector properties] for a complete list.
5353

5454
== What's next?
5555

56-
Learn more about Elasticsearch’s features and capabilities on https://www.elastic.co/elasticsearch/[Elasticsearch^].
56+
Learn more about Elasticsearch’s features and capabilities on https://www.elastic.co/elasticsearch/[Elasticsearch].
5757

58-
Learn more about Apache Pulsar's Elasticsearch sink in the https://pulsar.apache.org/docs/io-elasticsearch-sink/[Pulsar documentation^].
58+
Learn more about Apache Pulsar's Elasticsearch sink in the https://pulsar.apache.org/docs/io-elasticsearch-sink/[Pulsar documentation].

modules/pulsar-io/pages/connectors/sinks/google-bigquery.adoc

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,13 @@
88

99
BigQuery is a fully managed enterprise data warehouse that helps you manage and analyze your data with built-in features like machine learning, geospatial analysis, and business intelligence. BigQuery's serverless architecture lets you use SQL queries to answer your organization's biggest questions with zero infrastructure management. BigQuery's scalable, distributed analysis engine lets you query terabytes in seconds and petabytes in minutes.
1010

11-
BigQuery Pulsar Sink is not integrated with BigQuery directly. It uses Pulsar’s built-in https://pulsar.apache.org/docs/adaptors-kafka/[Kafka Connect adapter^] library to transform message data into a Kafka compatible format. Then the https://docs.confluent.io/kafka-connectors/bigquery/current/kafka_connect_bigquery_config.html[Kafka Connect BigQuery Sink^] is used as the actual BigQuery integration. The adaptor provides a flexible and extensible framework for data transformation and processing. It supports various data formats, including JSON, Avro, and Protobuf, and enables users to apply transformations on the data as it is being streamed from Pulsar.
11+
BigQuery Pulsar Sink is not integrated with BigQuery directly. It uses Pulsar’s built-in https://pulsar.apache.org/docs/adaptors-kafka/[Kafka Connect adapter] library to transform message data into a Kafka compatible format. Then the https://docs.confluent.io/kafka-connectors/bigquery/current/kafka_connect_bigquery_config.html[Kafka Connect BigQuery Sink] is used as the actual BigQuery integration. The adaptor provides a flexible and extensible framework for data transformation and processing. It supports various data formats, including JSON, Avro, and Protobuf, and enables users to apply transformations on the data as it is being streamed from Pulsar.
1212

1313
You will notice references to Kafka throughout the configuration. *You don’t need a running instance of Kafka to use this connector.* The Kafka references are used as "translation points” by this connector.
1414

1515
[NOTE]
1616
====
17-
If you would like some background on the Kafka Connect adapter and its use cases, read "https://www.datastax.com/blog/simplify-migrating-kafka-to-pulsar-kafka-connect-support[Simplify migrating from Kafka to Pulsar with Kafka Connect Support^]" and also "https://medium.com/building-the-open-data-stack/datastax-presents-snowflake-sink-connector-for-apache-pulsar-53629b196064[DataStax’s Snowflake Sink Connector for Apache Pulsar^]".
17+
If you would like some background on the Kafka Connect adapter and its use cases, read "https://www.datastax.com/blog/simplify-migrating-kafka-to-pulsar-kafka-connect-support[Simplify migrating from Kafka to Pulsar with Kafka Connect Support]" and also "https://medium.com/building-the-open-data-stack/datastax-presents-snowflake-sink-connector-for-apache-pulsar-53629b196064[DataStax’s Snowflake Sink Connector for Apache Pulsar]".
1818
====
1919

2020
== Get Started
@@ -42,7 +42,7 @@ include::example$connectors/sinks/astra.csv[]
4242

4343
=== Kafka Connect Adapter Configuration (configs)
4444

45-
These values are provided in the “configs” area. View the code for these configurations https://github.com/apache/pulsar/blob/master/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java[here^].
45+
These values are provided in the “configs” area. View the code for these configurations https://github.com/apache/pulsar/blob/master/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java[here].
4646

4747
[%header,format=csv,cols="2,1,1,3"]
4848
|===
@@ -51,7 +51,7 @@ include::example$connectors/sinks/bigquery/config.csv[]
5151

5252
=== Google BigQuery Configuration (kafkaConnectorConfigProperties)
5353

54-
These values are provided in the "kafkaConnectorConfigProperties" area. View the code for these configurations https://github.com/confluentinc/kafka-connect-bigquery/blob/master/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/config/BigQuerySinkConfig.java[here^].
54+
These values are provided in the "kafkaConnectorConfigProperties" area. View the code for these configurations https://github.com/confluentinc/kafka-connect-bigquery/blob/master/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/config/BigQuerySinkConfig.java[here].
5555

5656
[%header,format=csv,cols="2,1,1,3"]
5757
|===
@@ -60,4 +60,4 @@ include::example$connectors/sinks/bigquery/kafkaConnectorConfigProperties.csv[]
6060

6161
== What's next?
6262

63-
Learn more about Google’s BigQuery features and capabilities on https://cloud.google.com/bigquery[their site^].
63+
Learn more about Google’s BigQuery features and capabilities on https://cloud.google.com/bigquery[their site].

modules/pulsar-io/pages/connectors/sinks/jdbc-clickhouse.adoc

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,11 +43,11 @@ include::example$connectors/sinks/astra.csv[]
4343
These values are provided in the "configs" area.
4444

4545
The Astra Streaming JDBC Clickhouse sink connector supports all configuration properties provided by Apache Pulsar. Please refer to the https://pulsar.apache.org/docs/io-jdbc-sink#property[connector
46-
properties^]
46+
properties]
4747
for a complete list.
4848

4949
== What's next?
5050

51-
Learn more about Clickhouse's features and capabilities on https://clickhouse.com/[their site^].
51+
Learn more about Clickhouse's features and capabilities on https://clickhouse.com/[their site].
5252

53-
Learn more about Apache Pulsar's JDBC sink connector https://pulsar.apache.org/docs/io-jdbc-sink[here^].
53+
Learn more about Apache Pulsar's JDBC sink connector https://pulsar.apache.org/docs/io-jdbc-sink[here].

modules/pulsar-io/pages/connectors/sinks/jdbc-mariadb.adoc

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,9 @@
77

88
= JDBC MariaDB
99

10-
MariaDB is the open source relational database loved by developers all over the world. Created by MySQL’s original developers, MariaDB is compatible with MySQL and guaranteed to stay open source forever. MariaDB powers some of the world’s most popular websites such as Wikipedia and WordPress.com. It is also the core engine behind banking, social media, mobile and e-commerce sites worldwide. MariaDB Connector/J is a Type 4 JDBC driver. It was developed specifically as a lightweight JDBC connector for use with MariaDB and MySQL database servers. It was originally based on the Drizzle JDBC code with numerous additions and bug fixes. Learn more about MariaDB on https://mariadb.org/[their site^].
10+
MariaDB is the open source relational database loved by developers all over the world. Created by MySQL’s original developers, MariaDB is compatible with MySQL and guaranteed to stay open source forever. MariaDB powers some of the world’s most popular websites such as Wikipedia and WordPress.com. It is also the core engine behind banking, social media, mobile and e-commerce sites worldwide. MariaDB Connector/J is a Type 4 JDBC driver. It was developed specifically as a lightweight JDBC connector for use with MariaDB and MySQL database servers. It was originally based on the Drizzle JDBC code with numerous additions and bug fixes. Learn more about MariaDB on https://mariadb.org/[their site].
1111

12-
Read more about Apache Pulsar's JDBC sink connector https://pulsar.apache.org/docs/io-jdbc-sink[here^].
12+
Read more about Apache Pulsar's JDBC sink connector https://pulsar.apache.org/docs/io-jdbc-sink[here].
1313

1414
[NOTE]
1515
====
@@ -44,5 +44,5 @@ include::example$connectors/sinks/astra.csv[]
4444
These values are provided in the "configs" area.
4545

4646
The Astra Streaming JDBC MariaDB sink connector supports all configuration properties provided by Apache Pulsar. Please refer to the https://pulsar.apache.org/docs/io-jdbc-sink#property[connector
47-
properties^]
47+
properties]
4848
for a complete list.

modules/pulsar-io/pages/connectors/sinks/jdbc-postgres.adoc

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,10 +45,10 @@ include::example$connectors/sinks/astra.csv[]
4545
These values are provided in the "configs" area.
4646

4747
The {product_name} JDBC PostgreSQL sink connector supports all configuration properties provided by Apache Pulsar. Refer to the https://pulsar.apache.org/docs/io-jdbc-sink#property[connector
48-
properties^] for a complete list.
48+
properties] for a complete list.
4949

5050
== What's next?
5151

52-
Learn more about PostgreSQL on https://www.postgresql.org/[their site^].
52+
Learn more about PostgreSQL on https://www.postgresql.org/[their site].
5353

54-
Read more about Apache Pulsar's PostgreSQL sink connector https://pulsar.apache.org/docs/io-jdbc-sink[here^].
54+
Read more about Apache Pulsar's PostgreSQL sink connector https://pulsar.apache.org/docs/io-jdbc-sink[here].

0 commit comments

Comments
 (0)