Skip to content

Commit f896c9a

Browse files
committed
Update README.md with pre-built image instructions
1 parent f668e92 commit f896c9a

File tree

1 file changed

+15
-41
lines changed

1 file changed

+15
-41
lines changed

README.md

Lines changed: 15 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -2,54 +2,28 @@
22

33
A Dataflow Flex template that reads messages from Pub/Sub and sends them to Better Stack Telemetry.
44

5-
## Building and Deploying the Template
5+
## Running the template using Web UI
66

7-
1. Clone this repository:
8-
```bash
9-
git clone https://github.com/BetterStackHQ/gcp-dataflow-pubsub-to-betterstack.git
10-
cd gcp-dataflow-pubsub-to-betterstack
11-
```
12-
13-
2. Choose Google Cloud Platform project to use
14-
```bash
15-
# See currently selected project
16-
gcloud config get-value project
17-
# You can switch to a different project using
18-
gcloud projects list
19-
gcloud config set project PROJECT_ID
20-
```
7+
1. Go to Google Cloud Console -> Dataflow -> [Create job from template](https://console.cloud.google.com/dataflow/createjob)
8+
2. Choose the name and region for the Dataflow job
9+
3. Select **Custom Template**
10+
4. As Template path, use `better-stack-gcs-dataflow/pubsub-to-betterstack.json`
11+
5. Set parameters based on your Google Cloud Pub/Sub Subscription and [Better Stack Telemetry source](https://telemetry.betterstack.com/team/260195/sources)
12+
6. Click **Run job**
2113

22-
3. Choose Google Cloud Platform region to use
23-
```bash
24-
# See currently selected region
25-
gcloud config get-value compute/region
26-
# You can switch to a different region using
27-
gcloud app regions list
28-
gcloud config set compute/region PROJECT_ID
29-
```
30-
31-
4. Create a Cloud Storage bucket for the template (if you don't have one):
32-
```bash
33-
BUCKET_NAME="dataflow-templates-$(gcloud config get-value project)"
34-
gsutil mb -l $(gcloud config get-value compute/region) gs://${BUCKET_NAME}
35-
```
14+
## Running the template using CLI
3615

37-
5. Set parameters based on your Google Cloud Pub/Sub Subscription and Better Stack Telemetry source
16+
1. Set parameters based on your Google Cloud Pub/Sub Subscription and [Better Stack Telemetry source](https://telemetry.betterstack.com/team/260195/sources)
3817
```bash
3918
INPUT_SUBSCRIPTION=projects/$(gcloud config get-value project)/subscriptions/<your-pubsub-subscription-name>
4019
SOURCE_TOKEN=<your-better-stack-source-token>
4120
INGESTING_HOST=<your-better-stack-ingesting-host>
4221
```
4322

44-
6. Build, deploy and run the template
23+
2. Create a Dataflow job using the template
4524
```bash
46-
gcloud builds submit --tag "gcr.io/$(gcloud config get-value project)/pubsub-to-betterstack" .
47-
gcloud dataflow flex-template build gs://$BUCKET_NAME/pubsub-to-betterstack.json \
48-
--image "gcr.io/$(gcloud config get-value project)/pubsub-to-betterstack" \
49-
--sdk-language "PYTHON" \
50-
--metadata-file "metadata.json"
5125
gcloud dataflow flex-template run "pubsub-to-betterstack-$(date +%Y%m%d-%H%M%S)" \
52-
--template-file-gcs-location=gs://$BUCKET_NAME/pubsub-to-betterstack.json \
26+
--template-file-gcs-location=gs://better-stack-gcs-dataflow/pubsub-to-betterstack.json \
5327
--parameters input_subscription=$INPUT_SUBSCRIPTION \
5428
--parameters better_stack_source_token=$SOURCE_TOKEN \
5529
--parameters better_stack_ingesting_host=$INGESTING_HOST \
@@ -60,10 +34,10 @@ gcloud dataflow flex-template run "pubsub-to-betterstack-$(date +%Y%m%d-%H%M%S)"
6034

6135
The template supports the following optional parameters:
6236

63-
- `batch_size`: Number of messages to batch before sending to Better Stack. Default: 100
64-
- `window_size`: Window size in seconds for batching messages. Default: 10
65-
- `max_retries`: Maximum number of retry attempts for failed requests. Default: 3
66-
- `initial_retry_delay`: Initial delay between retries in seconds. Default: 1
37+
- `batch_size` - Number of messages to batch before sending to Better Stack. Default: 100
38+
- `window_size` - Window size in seconds for batching messages. Default: 10
39+
- `max_retries` - Maximum number of retry attempts for failed requests. Default: 3
40+
- `initial_retry_delay` - Initial delay between retries in seconds. Default: 1
6741

6842
You can include these parameters in your Dataflow job by adding them to the run command, e.g. `gcloud dataflow flex-template run ... --parameters window_size=30`.
6943

0 commit comments

Comments
 (0)