Skip to content

Commit d2acca9

Browse files
simkamsimkam
authored
allow testing against the real AWS services instead of LocalStack emulated services (#104)
* aws-s3-sink - allow testing against real AWS services * aws-eventbridge-sink - programatically get arn of new queue * aws-eventbridge-sink - decouple test code from infrastructure setup * aws-eventbridge-sink - add AWS variant of test * aws-sqs-sink - decouple test code from infrastructure setup * aws-sqs-sink - add AWS variant of test * aws-sns-sink - decouple test code from infrastructure setup * aws-sns-sink - add AWS variant of test * aws-sqs-source - decouple test code from infrastructure setup * aws-sqs-source - add AWS variant of test * aws-s3-source - decouple test code from infrastructure setup * aws-s3-source - add AWS variant of test * aws-ddb-streams-source - decouple test code from infrastructure setup * aws-ddb-streams-source - add AWS variant of test * add AWS cleanup script * switch to global maven profile instead of module's specific profiles * update README
1 parent 6386b62 commit d2acca9

File tree

50 files changed

+1734
-550
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

50 files changed

+1734
-550
lines changed

‎README.adoc‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -511,3 +511,81 @@ For more detailed description of all container image configuration options pleas
511511

512512
* https://quarkus.io/guides/deploying-to-kubernetes
513513
* https://quarkus.io/guides/container-image
514+
515+
== AWS Integration tests
516+
517+
Some modules have option to run integration tests against real AWS services instead of LocalStack emulated services.
518+
The test set executed during build is controlled by `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. When set, only AWS integration tests are executed,
519+
the rest is ignored.
520+
521+
[source,shell]
522+
----
523+
export AWS_ACCESS_KEY_ID=<access_key_id>
524+
export AWS_SECRET_ACCESS_KEY=<secret_access_key>
525+
526+
mvn test
527+
----
528+
529+
Tests require permissions to AWS services that they are testing. Necessary permission can be provided by
530+
531+
[source,shell]
532+
----
533+
aws iam create-user --user-name eventing-integrations-tests
534+
535+
# s3 sink + aws s3 source
536+
aws iam attach-user-policy --user-name eventing-integrations-tests --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
537+
538+
# sqs sink + eventbridge sink + sns sink + sqs source
539+
aws iam attach-user-policy --user-name eventing-integrations-tests --policy-arn arn:aws:iam::aws:policy/AmazonSQSFullAccess
540+
541+
# eventbridge sink
542+
aws iam attach-user-policy --user-name eventing-integrations-tests --policy-arn arn:aws:iam::aws:policy/AmazonEventBridgeFullAccess
543+
544+
# sns sink
545+
aws iam attach-user-policy --user-name eventing-integrations-tests --policy-arn arn:aws:iam::aws:policy/AmazonSNSFullAccess
546+
547+
# ddb streams source
548+
aws iam attach-user-policy --user-name eventing-integrations-tests --policy-arn arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess
549+
550+
# list all policies
551+
aws iam list-user-policies --user-name eventing-integrations-tests
552+
553+
# create access key
554+
aws iam create-access-key --user-name eventing-integrations-tests
555+
----
556+
557+
Tests do not clean AWS environment automatically. After tests execution, the AWS can be cleaned by
558+
559+
[source,shell]
560+
----
561+
# ddb streams source
562+
aws dynamodb delete-table --table-name ddb-streams-source
563+
564+
# eventbridge sink
565+
EVENT_BRIDGE_QUEUE_URL=$(aws sqs get-queue-url --queue-name eventbridge-sink-queue)
566+
aws sqs delete-queue --queue-url $EVENT_BRIDGE_QUEUE_URL
567+
aws events remove-targets --rule eventbridge-sink-cdc --ids eventbrindge-sqs-sub
568+
aws events delete-rule --name eventbridge-sink-cdc
569+
570+
# s3 sink
571+
aws s3api delete-bucket --bucket aws3-s3-sink-bucket
572+
573+
# s3 source
574+
aws s3api delete-bucket --bucket aws3-s3-source-bucket
575+
576+
# sns sink
577+
SNS_QUEUE_URL=$(aws sqs get-queue-url --queue-name sns-sink-queue)
578+
aws sqs delete-queue --queue-url $SNS_QUEUE_URL
579+
SNS_TOPIC_ARN=$(aws sns list-topics --query "Topics[?ends_with(TopicArn, 'sns-sink-topic')].TopicArn" --output text)
580+
SNS_SUBSCRIPTION_ARN=$(aws sns list-subscriptions-by-topic --topic-arn "$SNS_TOPIC_ARN" --query "Subscriptions[].SubscriptionArn" --output text)
581+
aws sns unsubscribe --subscription-arn $SNS_SUBSCRIPTION_ARN
582+
aws sns delete-topic --topic-arn $SNS_TOPIC_ARN
583+
584+
# sqs sink
585+
SQS_SINK_QUEUE_URL=$(aws sqs get-queue-url --queue-name sqs-sink-queue)
586+
aws sqs delete-queue --queue-url $SQS_SINK_QUEUE_URL
587+
588+
# sqs source
589+
SQS_SOURCE_QUEUE_URL=$(aws sqs get-queue-url --queue-name sqs-source-queue)
590+
aws sqs delete-queue --queue-url $SQS_SOURCE_QUEUE_URL
591+
----

‎aws-ddb-streams-source/pom.xml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,18 @@
110110
<artifactId>citrus-validation-json</artifactId>
111111
<scope>test</scope>
112112
</dependency>
113+
114+
<!-- Test dependencies - used with aws-integration-tests profile -->
115+
<dependency>
116+
<groupId>io.quarkiverse.amazonservices</groupId>
117+
<artifactId>quarkus-amazon-dynamodb</artifactId>
118+
<scope>test</scope>
119+
</dependency>
120+
<dependency>
121+
<groupId>software.amazon.awssdk</groupId>
122+
<artifactId>url-connection-client</artifactId>
123+
<scope>test</scope>
124+
</dependency>
113125
</dependencies>
114126

115127
<build>

aws-ddb-streams-source/src/test/java/dev/knative/eventing/connector/AwsDdbStreamsSourceTest.java renamed to aws-ddb-streams-source/src/test/java/dev/knative/eventing/connector/AwsDdbStreamsSourceTestBase.java

Lines changed: 36 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -20,62 +20,41 @@
2020
import java.util.Map;
2121

2222
import io.quarkus.test.junit.QuarkusTest;
23+
import jakarta.inject.Inject;
2324
import org.citrusframework.TestCaseRunner;
24-
import org.citrusframework.actions.testcontainers.aws2.AwsService;
2525
import org.citrusframework.annotations.CitrusResource;
2626
import org.citrusframework.context.TestContext;
27-
import org.citrusframework.http.endpoint.builder.HttpEndpoints;
2827
import org.citrusframework.http.server.HttpServer;
2928
import org.citrusframework.quarkus.CitrusSupport;
30-
import org.citrusframework.spi.BindToRegistry;
31-
import org.citrusframework.testcontainers.aws2.LocalStackContainer;
32-
import org.citrusframework.testcontainers.aws2.quarkus.LocalStackContainerSupport;
33-
import org.citrusframework.testcontainers.quarkus.ContainerLifecycleListener;
3429
import org.junit.jupiter.api.Test;
3530
import org.springframework.http.HttpStatus;
3631
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
37-
import software.amazon.awssdk.services.dynamodb.model.AttributeDefinition;
38-
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
39-
import software.amazon.awssdk.services.dynamodb.model.KeySchemaElement;
40-
import software.amazon.awssdk.services.dynamodb.model.KeyType;
41-
import software.amazon.awssdk.services.dynamodb.model.ProvisionedThroughput;
42-
import software.amazon.awssdk.services.dynamodb.model.ReturnValue;
43-
import software.amazon.awssdk.services.dynamodb.model.ScalarAttributeType;
44-
import software.amazon.awssdk.services.dynamodb.model.StreamSpecification;
45-
import software.amazon.awssdk.services.dynamodb.model.StreamViewType;
32+
import software.amazon.awssdk.services.dynamodb.model.*;
33+
import software.amazon.awssdk.services.dynamodb.waiters.DynamoDbWaiter;
4634

4735
import static org.citrusframework.actions.CreateVariablesAction.Builder.createVariable;
4836
import static org.citrusframework.http.actions.HttpActionBuilder.http;
4937

5038
@QuarkusTest
5139
@CitrusSupport
52-
@LocalStackContainerSupport(services = AwsService.DYNAMODB, containerLifecycleListener = AwsDdbStreamsSourceTest.class)
53-
public class AwsDdbStreamsSourceTest implements ContainerLifecycleListener<LocalStackContainer> {
40+
public abstract class AwsDdbStreamsSourceTestBase {
5441

5542
@CitrusResource
5643
private TestCaseRunner tc;
5744

58-
private final String ddbTableName = "movies";
45+
protected static final String ddbTableName = "ddb-streams-source";
5946

60-
@CitrusResource
61-
private LocalStackContainer localStackContainer;
62-
63-
@BindToRegistry
64-
public HttpServer knativeBroker = HttpEndpoints.http()
65-
.server()
66-
.port(8080)
67-
.timeout(5000L)
68-
.autoStart(true)
69-
.build();
47+
@Inject
48+
protected DynamoDbClient ddbClient;
7049

7150
@Test
7251
public void shouldProduceEvents() {
73-
tc.given(createVariable("aws.region", localStackContainer.getRegion()));
52+
tc.given(createVariable("aws.region", ddbClient.serviceClientConfiguration().region().id()));
7453

7554
tc.given(this::putItem);
7655

7756
tc.when(
78-
http().server(knativeBroker)
57+
http().server(getKnativeBroker())
7958
.receive()
8059
.post()
8160
.message()
@@ -110,7 +89,7 @@ public void shouldProduceEvents() {
11089
);
11190

11291
tc.then(
113-
http().server(knativeBroker)
92+
http().server(getKnativeBroker())
11493
.send()
11594
.response(HttpStatus.OK)
11695
);
@@ -122,41 +101,37 @@ private void putItem(TestContext context) {
122101
item.put("year", AttributeValue.builder().n("1977").build());
123102
item.put("title", AttributeValue.builder().s("Star Wars IV").build());
124103

125-
DynamoDbClient ddbClient = localStackContainer.getClient(AwsService.DYNAMODB);
126104
ddbClient.putItem(b -> b.tableName(ddbTableName)
127105
.item(item)
128106
.returnValues(ReturnValue.ALL_OLD));
129107
}
130108

131-
@Override
132-
public Map<String, String> started(LocalStackContainer container) {
133-
DynamoDbClient ddbClient = container.getClient(AwsService.DYNAMODB);
134-
135-
ddbClient.createTable(b -> {
136-
b.tableName(ddbTableName);
137-
b.keySchema(KeySchemaElement.builder().attributeName("id").keyType(KeyType.HASH).build());
138-
b.attributeDefinitions(AttributeDefinition.builder().attributeName("id").attributeType(ScalarAttributeType.N).build());
139-
140-
b.streamSpecification(StreamSpecification.builder()
141-
.streamEnabled(true)
142-
.streamViewType(StreamViewType.NEW_AND_OLD_IMAGES).build());
143-
144-
b.provisionedThroughput(
145-
ProvisionedThroughput.builder()
146-
.readCapacityUnits(1L)
147-
.writeCapacityUnits(1L).build());
148-
});
149-
150-
Map<String, String> conf = new HashMap<>();
151-
conf.put("camel.kamelet.aws-ddb-streams-source.accessKey", container.getAccessKey());
152-
conf.put("camel.kamelet.aws-ddb-streams-source.secretKey", container.getSecretKey());
153-
conf.put("camel.kamelet.aws-ddb-streams-source.region", container.getRegion());
154-
conf.put("camel.kamelet.aws-ddb-streams-source.table", ddbTableName);
155-
conf.put("camel.kamelet.aws-ddb-streams-source.uriEndpointOverride", container.getServiceEndpoint().toString());
156-
conf.put("camel.kamelet.aws-ddb-streams-source.overrideEndpoint", "true");
157-
conf.put("camel.kamelet.aws-ddb-streams-source.streamIteratorType", "FROM_START");
158-
159-
return conf;
109+
public static void setupDdb(DynamoDbClient ddbClient) {
110+
// create table if not exists
111+
try {
112+
ddbClient.describeTable(builder -> builder.tableName(ddbTableName));
113+
} catch (ResourceNotFoundException rnfe) {
114+
ddbClient.createTable(b -> {
115+
b.tableName(ddbTableName);
116+
b.keySchema(KeySchemaElement.builder().attributeName("id").keyType(KeyType.HASH).build());
117+
b.attributeDefinitions(AttributeDefinition.builder().attributeName("id").attributeType(ScalarAttributeType.N).build());
118+
119+
b.streamSpecification(StreamSpecification.builder()
120+
.streamEnabled(true)
121+
.streamViewType(StreamViewType.NEW_AND_OLD_IMAGES).build());
122+
123+
b.provisionedThroughput(
124+
ProvisionedThroughput.builder()
125+
.readCapacityUnits(1L)
126+
.writeCapacityUnits(1L).build());
127+
});
128+
129+
DynamoDbWaiter dynamoDbWaiter = ddbClient.waiter();
130+
DescribeTableRequest describeTableRequest = DescribeTableRequest.builder().tableName(ddbTableName).build();
131+
dynamoDbWaiter.waitUntilTableExists(describeTableRequest);
132+
}
160133
}
161134

135+
protected abstract HttpServer getKnativeBroker();
136+
162137
}

‎aws-ddb-streams-source/src/test/java/dev/knative/eventing/connector/AwsIntegrationTestProfile.java‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
/*
2+
* Copyright the original author or authors.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package dev.knative.eventing.connector;
18+
19+
import io.quarkus.test.junit.QuarkusTestProfile;
20+
21+
public class AwsIntegrationTestProfile implements QuarkusTestProfile {
22+
23+
@Override
24+
public String getConfigProfile() {
25+
return "aws-integration";
26+
}
27+
}

‎aws-ddb-streams-source/src/test/java/dev/knative/eventing/connector/aws/AwsDdbStreamsSourceIntegrationTest.java‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
/*
2+
* Copyright the original author or authors.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package dev.knative.eventing.connector.aws;
18+
19+
import dev.knative.eventing.connector.AwsDdbStreamsSourceTestBase;
20+
import dev.knative.eventing.connector.AwsIntegrationTestProfile;
21+
import io.quarkus.test.common.QuarkusTestResource;
22+
import io.quarkus.test.junit.QuarkusTest;
23+
import io.quarkus.test.junit.TestProfile;
24+
import org.citrusframework.http.endpoint.builder.HttpEndpoints;
25+
import org.citrusframework.http.server.HttpServer;
26+
import org.citrusframework.quarkus.CitrusSupport;
27+
import org.citrusframework.spi.BindToRegistry;
28+
29+
@QuarkusTest
30+
@CitrusSupport
31+
@TestProfile(AwsIntegrationTestProfile.class)
32+
@QuarkusTestResource(value = DynamoDBSetupResource.class, restrictToAnnotatedClass = true)
33+
public class AwsDdbStreamsSourceIntegrationTest extends AwsDdbStreamsSourceTestBase {
34+
35+
@BindToRegistry
36+
private HttpServer knativeBroker = HttpEndpoints.http()
37+
.server()
38+
.port(8080)
39+
.timeout(5000L)
40+
.autoStart(true)
41+
.build();
42+
43+
@Override
44+
protected HttpServer getKnativeBroker() {
45+
return knativeBroker;
46+
}
47+
}

‎aws-ddb-streams-source/src/test/java/dev/knative/eventing/connector/aws/DynamoDBSetupResource.java‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
/*
2+
* Copyright the original author or authors.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package dev.knative.eventing.connector.aws;
18+
19+
import dev.knative.eventing.connector.AwsDdbStreamsSourceTestBase;
20+
import io.quarkus.test.common.QuarkusTestResourceLifecycleManager;
21+
import org.eclipse.microprofile.config.Config;
22+
import org.eclipse.microprofile.config.ConfigProvider;
23+
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
24+
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
25+
import software.amazon.awssdk.regions.Region;
26+
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
27+
import software.amazon.awssdk.services.dynamodb.DynamoDbClientBuilder;
28+
29+
import java.util.Collections;
30+
import java.util.Map;
31+
32+
public class DynamoDBSetupResource implements QuarkusTestResourceLifecycleManager {
33+
34+
private final DynamoDbClient dynamoDbClient;
35+
36+
public DynamoDBSetupResource() {
37+
Config config = ConfigProvider.getConfig();
38+
String region = config.getValue("quarkus.dynamodb.aws.region", String.class);
39+
String accessKey = config.getValue("quarkus.dynamodb.aws.credentials.static-provider.access-key-id", String.class);
40+
String secretKey = config.getValue("quarkus.dynamodb.aws.credentials.static-provider.secret-access-key", String.class);
41+
42+
DynamoDbClientBuilder builder = DynamoDbClient.builder()
43+
.region(Region.of(region))
44+
.credentialsProvider(StaticCredentialsProvider.create(
45+
AwsBasicCredentials.create(accessKey, secretKey)));
46+
47+
dynamoDbClient = builder.build();
48+
}
49+
50+
@Override
51+
public Map<String, String> start() {
52+
AwsDdbStreamsSourceTestBase.setupDdb(dynamoDbClient);
53+
return Collections.emptyMap();
54+
}
55+
56+
@Override
57+
public void stop() {
58+
// noop
59+
}
60+
}

0 commit comments

Comments
 (0)