Skip to content

Commit a016bc3

Browse files
NeurostepNeurostep
authored
Merge pull request #127 from scribd/maksimt/DEVPLAT-1850/pubsub-sqs
[DEVPLAT-1850] Add SQS support to PubSub
2 parents 4e5ec90 + 720e682 commit a016bc3

19 files changed

+1976
-214
lines changed

‎README.md

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ SDK, the Go version.
3030
- [Usage of ORM](#usage-of-orm)
3131
- [PubSub](#pubsub)
3232
- [Kafka specific configuration](#kafka-specific-configuration)
33+
- [SQS specific configuration](#sqs-specific-configuration)
3334
- [Cache](#cache)
3435
- [Redis](#redis-specific-configuration)
3536
- [Experimentation](#experimentation)
@@ -849,6 +850,23 @@ common: &common
849850
subscriber:
850851
topic: "test-topic"
851852
group_id: ""
853+
sqs:
854+
subscriber:
855+
# Set by APP_PUBSUB_SQS_SUBSCRIBER_ENABLED env variable
856+
enabled: true
857+
# Set by APP_PUBSUB_SQS_SUBSCRIBER_QUEUE_URL env variable
858+
queue_url: "https://sqs.us-east-2.amazonaws.com/123456789012/test-queue"
859+
# Set by APP_PUBSUB_SQS_SUBSCRIBER_MAX_MESSAGES env variable
860+
max_messages: 10
861+
# Set by APP_PUBSUB_SQS_SUBSCRIBER_WORKERS env variable
862+
workers: 1
863+
# Set by APP_PUBSUB_SQS_SUBSCRIBER_WAIT_TIME env variable
864+
wait_time: "10s"
865+
publisher:
866+
# Set by APP_PUBSUB_SQS_PUBLISHER_ENABLED env variable
867+
enabled: true
868+
# Set by APP_PUBSUB_SQS_PUBLISHER_QUEUE_URL env variable
869+
queue_url: "https://sqs.us-east-2.amazonaws.com/123456789012/test-queue"
852870
```
853871
854872
#### Kafka specific configuration
@@ -920,6 +938,25 @@ To authenticate the requests to Kafka, Go SDK provides a configuration set for T
920938
| Assumable role | This role will be used to establish connection to AWS MSK ignoring the static credentials | `role` | `APP_PUBSUB_KAFKA_SASL_AWS_MSK_IAM_ROLE` | string | AWS ARN string |
921939
| Session name | Will be passed to AWS STS when assuming the role | `session_name` | `APP_PUBSUB_KAFKA_SASL_AWS_MSK_IAM_SESSION_NAME` | string | application |
922940

941+
#### SQS specific configuration
942+
943+
Under the hood, the SQS integration relies on the provided SQS client. To learn more about the AWS services initialization, check the [AWS service configuration](#aws-service-configuration).
944+
945+
**Subscriber**:
946+
947+
| Setting | Description | YAML variable | Environment variable (ENV) | Type | Possible Values |
948+
|-------------------|-------------------------------------------------------------------------------------------------|-----------------------|---------------------------------------------------|---------|-----------------|
949+
| Queue URL | URL of the SQS queue | `queue_url` | `APP_PUBSUB_SQS_SUBSCRIBER_QUEUE_URL` | string | queue URL |
950+
| Max messages | Maximum number of messages to retrieve from the queue per polling iteration | `max_messages` | `APP_PUBSUB_SQS_SUBSCRIBER_MAX_MESSAGES` | number | 1,2... |
951+
| Number of workers | Number of workers processing incoming messages | `workers` | `APP_PUBSUB_SQS_SUBSCRIBER_WORKERS` | number | 1,2... |
952+
| Wait time | The maximum amount of time in time.Duration to wait for messages to be available for retrieval. | `wait_time` | `APP_PUBSUB_SQS_SUBSCRIBER_WAIT_TIME` | string | 10s |
953+
954+
**Publisher**:
955+
956+
| Setting | Description | YAML variable | Environment variable (ENV) | Type | Possible Values |
957+
|-----------|-----------------------|-----------------------|---------------------------------------------------|---------|-----------------|
958+
| Queue URL | URL of the SQS queue | `queue_url` | `APP_PUBSUB_SQS_PUBLISHER_QUEUE_URL` | string | queue URL |
959+
923960
### Cache
924961

925962
`go-sdk` provides a convenient way to create an application Cache configuration.

‎go.mod

Lines changed: 27 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -7,14 +7,15 @@ require (
77
github.com/DATA-DOG/go-txdb v0.1.9
88
github.com/DataDog/datadog-go v4.8.2+incompatible
99
github.com/aws/aws-sdk-go v1.44.327
10-
github.com/aws/aws-sdk-go-v2 v1.26.1
11-
github.com/aws/aws-sdk-go-v2/config v1.27.11
12-
github.com/aws/aws-sdk-go-v2/credentials v1.17.11
13-
github.com/aws/aws-sdk-go-v2/service/s3 v1.53.1
14-
github.com/aws/aws-sdk-go-v2/service/sagemakerruntime v1.27.4
15-
github.com/aws/aws-sdk-go-v2/service/sfn v1.19.4
16-
github.com/aws/aws-sdk-go-v2/service/sqs v1.24.4
17-
github.com/aws/aws-sdk-go-v2/service/sts v1.28.6
10+
github.com/aws/aws-sdk-go-v2 v1.32.3
11+
github.com/aws/aws-sdk-go-v2/config v1.28.1
12+
github.com/aws/aws-sdk-go-v2/credentials v1.17.42
13+
github.com/aws/aws-sdk-go-v2/service/s3 v1.66.2
14+
github.com/aws/aws-sdk-go-v2/service/sagemakerruntime v1.32.3
15+
github.com/aws/aws-sdk-go-v2/service/sfn v1.33.3
16+
github.com/aws/aws-sdk-go-v2/service/sqs v1.36.3
17+
github.com/aws/aws-sdk-go-v2/service/sts v1.32.3
18+
github.com/aws/smithy-go v1.22.0
1819
github.com/getsentry/sentry-go v0.12.0
1920
github.com/go-kit/kit v0.9.0
2021
github.com/go-sql-driver/mysql v1.7.1
@@ -48,24 +49,24 @@ require (
4849
github.com/DataDog/gostackparse v0.7.0 // indirect
4950
github.com/DataDog/sketches-go v1.4.2 // indirect
5051
github.com/Microsoft/go-winio v0.6.1 // indirect
51-
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 // indirect
52-
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.1 // indirect
53-
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.5 // indirect
54-
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.5 // indirect
55-
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
56-
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.5 // indirect
57-
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.21.4 // indirect
58-
github.com/aws/aws-sdk-go-v2/service/eventbridge v1.20.4 // indirect
59-
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.2 // indirect
60-
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.7 // indirect
61-
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.7.34 // indirect
62-
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.7 // indirect
63-
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.5 // indirect
64-
github.com/aws/aws-sdk-go-v2/service/kinesis v1.18.4 // indirect
65-
github.com/aws/aws-sdk-go-v2/service/sns v1.21.4 // indirect
66-
github.com/aws/aws-sdk-go-v2/service/sso v1.20.5 // indirect
67-
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.4 // indirect
68-
github.com/aws/smithy-go v1.20.2 // indirect
52+
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.6 // indirect
53+
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.18 // indirect
54+
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.22 // indirect
55+
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.22 // indirect
56+
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
57+
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.22 // indirect
58+
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.36.3 // indirect
59+
github.com/aws/aws-sdk-go-v2/service/ec2 v1.187.0 // indirect
60+
github.com/aws/aws-sdk-go-v2/service/eventbridge v1.35.3 // indirect
61+
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.0 // indirect
62+
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.4.3 // indirect
63+
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.10.3 // indirect
64+
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.3 // indirect
65+
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.3 // indirect
66+
github.com/aws/aws-sdk-go-v2/service/kinesis v1.32.3 // indirect
67+
github.com/aws/aws-sdk-go-v2/service/sns v1.33.3 // indirect
68+
github.com/aws/aws-sdk-go-v2/service/sso v1.24.3 // indirect
69+
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.3 // indirect
6970
github.com/cespare/xxhash/v2 v2.2.0 // indirect
7071
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
7172
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect

0 commit comments

Comments
 (0)