Skip to content

Commit f193e5f

Browse files
authored
[e2e] Setup qase integration (once again) (#3441)
* add checkNewBrokersTest * add manual cases and listeners * add manual cases and listeners * add manual cases and listeners * upd manual suite * upd listeners * add readme * upd template * upd naming * upd template * upd template * upd template * upd template * fix naming * fix MessagesTest * upd manual cases * upd comments
1 parent 18c046a commit f193e5f

File tree

32 files changed

+768
-225
lines changed

32 files changed

+768
-225
lines changed

.github/workflows/e2e-automation.yml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,6 @@ on:
88
required: true
99
type: choice
1010
options:
11-
- manual
12-
- qase
1311
- regression
1412
- sanity
1513
- smoke

.github/workflows/e2e-manual.yml

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
name: E2E Manual suite
2+
on:
3+
workflow_dispatch:
4+
inputs:
5+
test_suite:
6+
description: 'Select test suite to run'
7+
default: 'manual'
8+
required: true
9+
type: choice
10+
options:
11+
- manual
12+
- qase
13+
qase_token:
14+
description: 'Set Qase token to enable integration'
15+
required: true
16+
type: string
17+
18+
jobs:
19+
build-and-test:
20+
runs-on: ubuntu-latest
21+
steps:
22+
- uses: actions/checkout@v3
23+
with:
24+
ref: ${{ github.sha }}
25+
- name: Set up environment
26+
id: set_env_values
27+
run: |
28+
cat "./kafka-ui-e2e-checks/.env.ci" >> "./kafka-ui-e2e-checks/.env"
29+
- name: Set up JDK
30+
uses: actions/setup-java@v3
31+
with:
32+
java-version: '17'
33+
distribution: 'zulu'
34+
cache: 'maven'
35+
- name: Build with Maven
36+
id: build_app
37+
run: |
38+
./mvnw -B -ntp versions:set -DnewVersion=${{ github.sha }}
39+
./mvnw -B -V -ntp clean install -Pprod -Dmaven.test.skip=true ${{ github.event.inputs.extraMavenOptions }}
40+
- name: Run test suite
41+
run: |
42+
./mvnw -B -ntp versions:set -DnewVersion=${{ github.sha }}
43+
./mvnw -B -V -ntp -DQASEIO_API_TOKEN=${{ github.event.inputs.qase_token }} -Dsurefire.suiteXmlFiles='src/test/resources/${{ github.event.inputs.test_suite }}.xml' -Dsuite=${{ github.event.inputs.test_suite }} -f 'kafka-ui-e2e-checks' test -Pprod

kafka-ui-e2e-checks/QASE.md

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
### E2E integration with Qase.io TMS (for internal users)
2+
3+
### Table of Contents
4+
5+
- [Intro](#intro)
6+
- [Set up Qase.io integration](#set-up-qase-integration)
7+
- [Test case creation](#test-case-creation)
8+
- [Test run reporting](#test-run-reporting)
9+
10+
### Intro
11+
12+
We're using [Qase.io](https://help.qase.io/en/) as TMS to keep test cases and accumulate test runs.
13+
Integration is set up through API using [qase-api](https://mvnrepository.com/artifact/io.qase/qase-api)
14+
and [qase-testng](https://mvnrepository.com/artifact/io.qase/qase-testng) libraries.
15+
16+
### Set up Qase integration
17+
18+
To set up integration locally add next VM option `-DQASEIO_API_TOKEN='%s'`
19+
(add your [Qase token](https://app.qase.io/user/api/token) instead of '%s') into your run configuration
20+
21+
### Test case creation
22+
23+
All new test cases can be added into TMS by default if they have no QaseId and QaseTitle matching already existing
24+
cases.
25+
But to handle `@Suite` and `@Automation` we added custom QaseCreateListener. To create new test case for next sync with
26+
Qase (see example `kafka-ui-e2e-checks/src/test/java/com/provectus/kafka/ui/qaseSuite/Template.java`):
27+
28+
1. Create new class in `kafka-ui-e2e-checks/src/test/java/com/provectus/kafka/ui/qaseSuite/suit`
29+
2. Inherit it from `kafka-ui-e2e-checks/src/test/java/com/provectus/kafka/ui/qaseSuite/BaseQaseTest.java`
30+
3. Create new test method with some name inside the class and annotate it with:
31+
32+
- `@Automation` (optional - Not automated by default) - to set one of automation states: NOT_AUTOMATED, TO_BE_AUTOMATED,
33+
AUTOMATED
34+
- `@QaseTitle` (required) - to set title for new test case and to check is there no existing cases with same title in
35+
Qase.io
36+
- `@Status` (optional - Draft by default) - to set one of case statuses: ACTUAL, DRAFT, DEPRECATED
37+
- `@Suite` (optional) - to store new case in some existing package need to set its id, otherwise case will be stored in
38+
the root
39+
- `@Test` (required) - annotation from TestNG to specify this method as test
40+
41+
4. Create new private void step methods with some name inside the same class and annotate it with
42+
@io.qase.api.annotation.Step to specify this method as step.
43+
5. Use defined step methods inside created test method in concrete order
44+
6. If there are any additional cases to create you can repeat scenario in a new class
45+
7. There are two ways to sync newly created cases in the framework with Qase.io:
46+
47+
- sync can be performed locally - run new test classes with
48+
already [set up Qase.io integration](#Set up Qase.io integration)
49+
- also you can commit and push your changes, then
50+
run [E2E Manual suite](https://github.com/provectus/kafka-ui/actions/workflows/e2e-manual.yml) on your branch
51+
52+
8. No test run in Qase.io will be created, new test case will be stored defined directory
53+
in [project's repository](https://app.qase.io/project/KAFKAUI)
54+
9. To add expected results into created test case edit in Qase.io manually
55+
56+
### Test run reporting
57+
58+
To handle manual test cases with status `Skipped` we added custom QaseResultListener. To create new test run:
59+
60+
1. All test methods should be annotated with actual `@QaseId`
61+
2. There are two ways to sync newly created cases in the framework with Qase.io:
62+
63+
- run can be performed locally - run test classes (or suites) with
64+
already [set up Qase.io integration](#Set up Qase.io integration), they will be labeled as `Automation CUSTOM suite`
65+
- also you can commit and push your changes, then
66+
run [E2E Automation suite](https://github.com/provectus/kafka-ui/actions/workflows/e2e-automation.yml) on your branch
67+
68+
3. All new test runs will be added into [project's test runs](https://app.qase.io/run/KAFKAUI) with corresponding label
69+
using QaseId to identify existing cases
70+
4. All test cases from manual suite are set up to have `Skipped` status in test runs to perform them manually

kafka-ui-e2e-checks/README.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ This repository is for E2E UI automation.
77
- [Prerequisites](#prerequisites)
88
- [How to install](#how-to-install)
99
- [How to run checks](#how-to-run-checks)
10+
- [Qase.io integration (for internal users)](#qase-integration)
1011
- [Reporting](#reporting)
1112
- [Environments setup](#environments-setup)
1213
- [Test Data](#test-data)
@@ -50,6 +51,10 @@ docker-compose -f documentation/compose/e2e-tests.yaml up -d
5051
-Dbrowser=local
5152
```
5253

54+
### Qase integration
55+
56+
Found instruction for Qase.io integration (for internal use only) at `kafka-ui-e2e-checks/QASE.md`
57+
5358
### Reporting
5459

5560
Reports are in `allure-results` folder.

kafka-ui-e2e-checks/pom.xml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,10 @@
1717
<testcontainers.version>1.17.6</testcontainers.version>
1818
<selenide.version>6.11.2</selenide.version>
1919
<testng.version>7.7.0</testng.version>
20-
<allure.version>2.20.1</allure.version>
21-
<qase.io.version>3.0.2</qase.io.version>
20+
<allure.version>2.21.0</allure.version>
21+
<qase.io.version>3.0.3</qase.io.version>
2222
<aspectj.version>1.9.9.1</aspectj.version>
23-
<assertj.version>3.23.1</assertj.version>
23+
<assertj.version>3.24.2</assertj.version>
2424
<hamcrest.version>2.2</hamcrest.version>
2525
<slf4j.version>1.7.36</slf4j.version>
2626
<dotenv.version>2.3.1</dotenv.version>
Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
package com.provectus.kafka.ui.settings.listeners;
2+
3+
import com.provectus.kafka.ui.utilities.qaseUtils.annotations.Automation;
4+
import com.provectus.kafka.ui.utilities.qaseUtils.annotations.Status;
5+
import com.provectus.kafka.ui.utilities.qaseUtils.annotations.Suite;
6+
import io.qase.api.QaseClient;
7+
import io.qase.api.StepStorage;
8+
import io.qase.api.annotation.QaseId;
9+
import io.qase.client.ApiClient;
10+
import io.qase.client.api.CasesApi;
11+
import io.qase.client.model.*;
12+
import lombok.SneakyThrows;
13+
import lombok.extern.slf4j.Slf4j;
14+
import org.testng.Assert;
15+
import org.testng.ITestListener;
16+
import org.testng.ITestResult;
17+
import org.testng.TestListenerAdapter;
18+
19+
import java.lang.reflect.Method;
20+
import java.util.*;
21+
22+
import static io.qase.api.utils.IntegrationUtils.getCaseTitle;
23+
24+
@Slf4j
25+
public class QaseCreateListener extends TestListenerAdapter implements ITestListener {
26+
27+
private static final CasesApi QASE_API = getQaseApi();
28+
29+
private static CasesApi getQaseApi() {
30+
ApiClient apiClient = QaseClient.getApiClient();
31+
apiClient.setApiKey(System.getProperty("QASEIO_API_TOKEN"));
32+
return new CasesApi(apiClient);
33+
}
34+
35+
private static int getStatus(Method method) {
36+
if (method.isAnnotationPresent(Status.class))
37+
return method.getDeclaredAnnotation(Status.class).status().getValue();
38+
return 1;
39+
}
40+
41+
private static int getAutomation(Method method) {
42+
if (method.isAnnotationPresent(Automation.class))
43+
return method.getDeclaredAnnotation(Automation.class).state().getValue();
44+
return 0;
45+
}
46+
47+
@SneakyThrows
48+
private static HashMap<Long, String> getCaseTitlesAndIdsFromQase() {
49+
HashMap<Long, String> cases = new HashMap<>();
50+
boolean getCases = true;
51+
int offSet = 0;
52+
while (getCases) {
53+
getCases = false;
54+
TestCaseListResponse response = QASE_API.getCases(System.getProperty("QASE_PROJECT_CODE"),
55+
new GetCasesFiltersParameter().status(GetCasesFiltersParameter.SERIALIZED_NAME_STATUS), 100, offSet);
56+
TestCaseListResponseAllOfResult result = response.getResult();
57+
Assert.assertNotNull(result);
58+
List<TestCase> entities = result.getEntities();
59+
Assert.assertNotNull(entities);
60+
if (entities.size() > 0) {
61+
for (TestCase testCase : entities) {
62+
cases.put(testCase.getId(), testCase.getTitle());
63+
}
64+
offSet = offSet + 100;
65+
getCases = true;
66+
}
67+
}
68+
return cases;
69+
}
70+
71+
private static boolean isCaseWithTitleExistInQase(Method method) {
72+
HashMap<Long, String> cases = getCaseTitlesAndIdsFromQase();
73+
String title = getCaseTitle(method);
74+
if (cases.containsValue(title)) {
75+
for (Map.Entry<Long, String> map : cases.entrySet()) {
76+
if (map.getValue().matches(title)) {
77+
long id = map.getKey();
78+
log.warn(String.format("Test case with @QaseTitle='%s' already exists with @QaseId=%d. " +
79+
"Please verify @QaseTitle annotation", title, id));
80+
return true;
81+
}
82+
}
83+
}
84+
return false;
85+
}
86+
87+
@Override
88+
@SneakyThrows
89+
public void onTestSuccess(final ITestResult testResult) {
90+
Method method = testResult.getMethod()
91+
.getConstructorOrMethod()
92+
.getMethod();
93+
String title = getCaseTitle(method);
94+
if (!method.isAnnotationPresent(QaseId.class)) {
95+
if (title != null) {
96+
if (!isCaseWithTitleExistInQase(method)) {
97+
LinkedList<ResultCreateStepsInner> resultSteps = StepStorage.stopSteps();
98+
LinkedList<TestCaseCreateStepsInner> createSteps = new LinkedList<>();
99+
resultSteps.forEach(step -> {
100+
TestCaseCreateStepsInner caseStep = new TestCaseCreateStepsInner();
101+
caseStep.setAction(step.getAction());
102+
caseStep.setExpectedResult(step.getExpectedResult());
103+
createSteps.add(caseStep);
104+
});
105+
TestCaseCreate newCase = new TestCaseCreate();
106+
newCase.setTitle(title);
107+
newCase.setStatus(getStatus(method));
108+
newCase.setAutomation(getAutomation(method));
109+
newCase.setSteps(createSteps);
110+
if (method.isAnnotationPresent(Suite.class)) {
111+
long suiteId = method.getDeclaredAnnotation(Suite.class).id();
112+
newCase.suiteId(suiteId);
113+
}
114+
Long id = Objects.requireNonNull(QASE_API.createCase(System.getProperty("QASE_PROJECT_CODE"),
115+
newCase).getResult()).getId();
116+
log.info(String.format("New test case '%s' was created with @QaseId=%d", title, id));
117+
}
118+
} else
119+
log.warn("To create new test case in Qase.io please add @QaseTitle annotation");
120+
} else
121+
log.warn("To create new test case in Qase.io please remove @QaseId annotation");
122+
}
123+
}
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
package com.provectus.kafka.ui.settings.listeners;
2+
3+
import io.qase.api.StepStorage;
4+
import io.qase.api.config.QaseConfig;
5+
import io.qase.api.services.QaseTestCaseListener;
6+
import io.qase.client.model.ResultCreate;
7+
import io.qase.client.model.ResultCreateCase;
8+
import io.qase.client.model.ResultCreateStepsInner;
9+
import io.qase.testng.guice.module.TestNgModule;
10+
import lombok.AccessLevel;
11+
import lombok.Getter;
12+
import lombok.extern.slf4j.Slf4j;
13+
import org.testng.ITestContext;
14+
import org.testng.ITestListener;
15+
import org.testng.ITestResult;
16+
import org.testng.TestListenerAdapter;
17+
18+
import java.lang.reflect.Method;
19+
import java.util.LinkedList;
20+
import java.util.Optional;
21+
22+
import static io.qase.api.utils.IntegrationUtils.*;
23+
import static io.qase.client.model.ResultCreate.StatusEnum.*;
24+
25+
@Slf4j
26+
public class QaseResultListener extends TestListenerAdapter implements ITestListener {
27+
28+
private static final String REPORTER_NAME = "TestNG";
29+
30+
static {
31+
System.setProperty(QaseConfig.QASE_CLIENT_REPORTER_NAME_KEY, REPORTER_NAME);
32+
}
33+
34+
@Getter(lazy = true, value = AccessLevel.PRIVATE)
35+
private final QaseTestCaseListener qaseTestCaseListener = createQaseListener();
36+
37+
private static QaseTestCaseListener createQaseListener() {
38+
return TestNgModule.getInjector().getInstance(QaseTestCaseListener.class);
39+
}
40+
41+
@Override
42+
public void onTestStart(ITestResult result) {
43+
getQaseTestCaseListener().onTestCaseStarted();
44+
super.onTestStart(result);
45+
}
46+
47+
@Override
48+
public void onTestSuccess(ITestResult tr) {
49+
getQaseTestCaseListener()
50+
.onTestCaseFinished(resultCreate -> setupResultItem(resultCreate, tr, PASSED));
51+
super.onTestSuccess(tr);
52+
}
53+
54+
@Override
55+
public void onTestSkipped(ITestResult tr) {
56+
getQaseTestCaseListener()
57+
.onTestCaseFinished(resultCreate -> setupResultItem(resultCreate, tr, SKIPPED));
58+
super.onTestSuccess(tr);
59+
}
60+
61+
@Override
62+
public void onTestFailure(ITestResult tr) {
63+
getQaseTestCaseListener()
64+
.onTestCaseFinished(resultCreate -> setupResultItem(resultCreate, tr, FAILED));
65+
super.onTestFailure(tr);
66+
}
67+
68+
@Override
69+
public void onFinish(ITestContext testContext) {
70+
getQaseTestCaseListener().onTestCasesSetFinished();
71+
super.onFinish(testContext);
72+
}
73+
74+
private void setupResultItem(ResultCreate resultCreate, ITestResult result, ResultCreate.StatusEnum status) {
75+
Optional<Throwable> resultThrowable = Optional.ofNullable(result.getThrowable());
76+
String comment = resultThrowable
77+
.flatMap(throwable -> Optional.of(throwable.toString())).orElse(null);
78+
Boolean isDefect = resultThrowable
79+
.flatMap(throwable -> Optional.of(throwable instanceof AssertionError))
80+
.orElse(false);
81+
String stacktrace = resultThrowable
82+
.flatMap(throwable -> Optional.of(getStacktrace(throwable)))
83+
.orElse(null);
84+
Method method = result.getMethod()
85+
.getConstructorOrMethod()
86+
.getMethod();
87+
Long caseId = getCaseId(method);
88+
String caseTitle = null;
89+
if (caseId == null) {
90+
caseTitle = getCaseTitle(method);
91+
}
92+
LinkedList<ResultCreateStepsInner> steps = StepStorage.stopSteps();
93+
resultCreate
94+
._case(caseTitle == null ? null : new ResultCreateCase().title(caseTitle))
95+
.caseId(caseId)
96+
.status(status)
97+
.comment(comment)
98+
.stacktrace(stacktrace)
99+
.steps(steps.isEmpty() ? null : steps)
100+
.defect(isDefect);
101+
}
102+
}

kafka-ui-e2e-checks/src/main/java/com/provectus/kafka/ui/utilities/qaseUtils/QaseSetup.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
@Slf4j
1616
public class QaseSetup {
1717

18-
public static void testRunSetup() {
18+
public static void qaseIntegrationSetup() {
1919
String qaseApiToken = System.getProperty("QASEIO_API_TOKEN");
2020
if (isEmpty(qaseApiToken)) {
2121
log.warn("Integration with Qase is disabled due to run config or token wasn't defined.");

0 commit comments

Comments
 (0)