Skip to content

Commit 8178841

Browse files
authored
Merge branch 'develop' into MQE-2082
2 parents 93b45d6 + 846f9c6 commit 8178841

File tree

12 files changed

+460
-2487
lines changed

12 files changed

+460
-2487
lines changed

CHANGELOG.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,43 @@
11
Magento Functional Testing Framework Changelog
22
================================================
3+
3.0.0 RC2
4+
---------
5+
6+
### Enhancements
7+
8+
* Maintainability
9+
* Added support for PHP 7.4.
10+
* Removed support for PHP 7.2.
11+
* Added support for PHPUnit 9.
12+
* Improved assertion actions to support PHPUnit 9 changes. [See assertions page for details](./docs/test/assertions.md)
13+
* Added new actions: `assertEqualsWithDelta`, `assertNotEqualsWithDelta`, `assertEqualsCanonicalizing`, `assertNotEqualsCanonicalizing`, `assertEqualsIgnoringCase`, `assertNotEqualsIgnoringCase`.
14+
* Added new actions: `assertStringContainsString`, `assertStringNotContainsString`, `assertStringContainsStringIgnoringCase`, `assertStringNotContainsStringIgnoringCase` for string haystacks.
15+
* Removed actions: `assertInternalType`, `assertNotInternalType`, `assertArraySubset`.
16+
* Removed delta option from `assertEquals` and `assertNotEquals`.
17+
* Removed action `pauseExecution` and added `pause`. [See actions page for details](./docs/test/actions.md#pause)
18+
* Removed action `formatMoney` and added `formatCurrency`. [See actions page for details](./docs/test/actions.md#formatcurrency)
19+
* Added new static check that checks and reports references to deprecated test entities.
20+
* Bumped dependencies to support PHP/PHPUnit upgrade.
21+
22+
* Traceability
23+
* Introduced new `.env` configuration `VERBOSE_ARTIFACTS` to toggle saving attachments in Allure. [See configuration page for details](./docs/configuration.md)
24+
25+
### Fixes
26+
27+
* Fixed issue of resolving arguments of type `entity` in action groups within a custom helper.
28+
* Fixed reporting issue in output file for `testDependencies` static check.
29+
* Fixed a bug in `actionGroupArguments` static check when action group filename is missing `ActionGroup`.
30+
* Fixed issue of running suites under root `_suite` directory in Standalone MFTF.
31+
32+
### GitHub Issues/Pull Requests
33+
34+
* [#567](https://github.com/magento/magento2-functional-testing-framework/pull/567) -- log attachments for failed requests.
35+
336
3.0.0 RC1
437
---------
538

639
### Enhancements
40+
741
* Customizability
842
* Introduced MFTF helpers `<helper>` to create custom actions outside of MFTF.
943
* Removed deprecated actions `<executeSelenium>` and `<performOn>`.
@@ -23,11 +57,13 @@ Magento Functional Testing Framework Changelog
2357
* Bumped dependencies to latest possible versions.
2458

2559
### Fixes
60+
2661
* Throw exception during generation when leaving out .url for `amOnPage`.
2762
* `request_timeout` and `connection_timeout` added to functional.suite.yml.dist.
2863
* Fixed `ModuleResolver` to resolve test modules moved out of deprecated path.
2964

3065
### Upgrade Instructions
66+
3167
* Run `bin/mftf reset --hard` to remove old generated configurations.
3268
* Run `bin/mftf build:project` to generate new configurations.
3369
* Run `bin/mftf upgrade:tests`. [See command page for details](./docs/commands/mftf.md#upgradetests).

docs/commands/mftf.md

Lines changed: 41 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -430,8 +430,12 @@ The example parameters are taken from the `etc/config/.env.example` file.
430430

431431
### `static-checks`
432432

433-
Runs all or specific MFTF static-checks on the test codebase that MFTF is currently attached to.
434-
If no script name argument is specified, all existing static check scripts will run.
433+
Runs all or specific MFTF static-checks on the test codebase that MFTF is currently attached to.
434+
Behavior for determining what tests to run is as follows:
435+
436+
* If test names are specified, only those tests are run.
437+
* If no test names are specified, tests are run according to `staticRuleset.json`.
438+
* If no `staticRuleset.json` is found, all tests are run.
435439

436440
#### Usage
437441

@@ -464,18 +468,50 @@ To run specific static check scripts
464468
```bash
465469
vendor/bin/mftf static-checks testDependencies
466470
```
471+
467472
```bash
468473
vendor/bin/mftf static-checks actionGroupArguments
469474
```
475+
476+
```bash
477+
vendor/bin/mftf static-checks deprecatedEntityUsage
478+
```
479+
480+
```bash
481+
vendor/bin/mftf static-checks deprecatedEntityUsage -p path/to/mftf/test/module
482+
```
483+
470484
```bash
471485
vendor/bin/mftf static-checks testDependencies actionGroupArguments
472486
```
473487

474488
#### Existing static checks
475489

476-
* Test Dependency: Checks that test dependencies do not violate Magento module's composer dependencies.
477-
* Action Group Unused Arguments: Checks that action groups do not have unused arguments.
478-
490+
| Argument | Description |
491+
|-----------------------|-----------------------------------------------------------------------------------------------------------|
492+
|`testDependencies` | Checks that test dependencies do not violate Magento module's composer dependencies.|
493+
|`actionGroupArguments` | Checks that action groups do not have unused arguments.|
494+
|`deprecatedEntityUsage`| Checks that deprecated test entities are not being referenced.|
495+
496+
#### Defining ruleset
497+
498+
The `static-checks` command will look for a `staticRuleset.json` file under either:
499+
500+
* `dev/tests/acceptance/staticRuleset.json`, if embedded with Magento2
501+
* `dev/staticRuleset.json`, if standalone
502+
503+
This file works as the default configuration to easily allow for the integration of `static-checks` in a CI environment.
504+
Currently, the ruleset only defines the tests to run. Here is an example of the expected format:
505+
506+
```json
507+
{
508+
"tests": [
509+
"actionGroupArguments",
510+
"anotherTest"
511+
]
512+
}
513+
```
514+
479515
### `upgrade:tests`
480516

481517
When the path argument is specified, this `upgrade` command applies all the major version MFTF upgrade scripts to a `Test Module` in the given path.

docs/custom-helpers.md

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Custom Helpers
2+
3+
<div class="bs-callout-warning">
4+
Due to complexity, you should only write new custom helpers as a last resort, after trying to implement your test using built-in actions.
5+
</div>
6+
7+
Custom Helpers allow test writers to write custom test actions to solve advanced requirements beyond what MFTF offers out of the box.
8+
9+
In MFTF version 3.0.0, the following test actions were removed:
10+
11+
* `<executeInSelenium>`
12+
* `<performOn>`
13+
14+
These actions were removed because they allowed custom PHP code to be written inline inside of XML files. This code was difficult to read. It had no proper syntax highlighting and no linting. It was difficult to maintain, troubleshoot, and modify.
15+
16+
However, sometimes custom logic beyond what MFTF offers is necessary so we have provided an alternative solution: the `<helper>` action.
17+
18+
## Example
19+
20+
Custom helpers are implemented in PHP files that must be placed in this directory:
21+
22+
```text
23+
<ModuleName>/Test/Mftf/Helper
24+
```
25+
26+
This custom helper selects text on the page with this approach:
27+
28+
1. Move to a very specific X,Y starting position.
29+
1. Click and hold the mouse button down.
30+
1. Move to another specific X,Y position.
31+
1. Release the mouse button.
32+
33+
This functionality is used to select text on the page and cannot be accomplished using built-in test actions.
34+
35+
### PHP File
36+
37+
```php
38+
<?php
39+
/**
40+
* Copyright © Magento, Inc. All rights reserved.
41+
* See COPYING.txt for license details.
42+
*/
43+
44+
namespace Magento\PageBuilder\Test\Mftf\Helper;
45+
46+
use Magento\FunctionalTestingFramework\Helper\Helper;
47+
48+
/**
49+
* Class SelectText provides an ability to select needed text.
50+
*/
51+
class SelectText extends Helper
52+
{
53+
/**
54+
* Select needed text.
55+
*
56+
* @param string $context
57+
* @param int $startX
58+
* @param int $startY
59+
* @param int $endX
60+
* @param int $endY
61+
* @return void
62+
*/
63+
public function selectText(string $context, int $startX, int $startY, int $endX, int $endY)
64+
{
65+
try {
66+
/** @var \Magento\FunctionalTestingFramework\Module\MagentoWebDriver $webDriver */
67+
$webDriver = $this->getModule('\Magento\FunctionalTestingFramework\Module\MagentoWebDriver');
68+
69+
$contextElement = $webDriver->webDriver->findElement(\Facebook\WebDriver\WebDriverBy::xpath($context));
70+
$actions = new \Facebook\WebDriver\Interactions\WebDriverActions($webDriver->webDriver);
71+
$actions->moveToElement($contextElement, $startX, $startY)
72+
->clickAndHold()
73+
->moveToElement($contextElement, $endX, $endY)
74+
->release()
75+
->perform();
76+
} catch (\Exception $e) {
77+
$this->fail($e->getMessage());
78+
}
79+
}
80+
}
81+
```
82+
83+
### Notes about this PHP file
84+
85+
The following details are important about the above file:
86+
87+
1. The `namespace` must match the file location: `namespace Magento\PageBuilder\Test\Mftf\Helper;`
88+
2. The class must `extends Helper` and have the corresponding `use` statement to match.
89+
3. You may access the WebDriver object via `$this->getModule('\Magento\FunctionalTestingFramework\Module\MagentoWebDriver')`.
90+
4. You may implement multiple related methods within the same class.
91+
5. Specify the correct function argument types to match the type of values you want to pass in. In this case, we specified `string $context, int $startX, int $startY, int $endX, int $endY`. In the XML we will match these types.
92+
93+
You should follow the same patterns in any custom helpers that you write yourself, but you may implement any logic or iteration that you need to solve for your use case.
94+
95+
### Referencing in a test
96+
97+
Once you have implemented something like the above PHP file, reference it in a test:
98+
99+
```xml
100+
<helper class="\Magento\PageBuilder\Test\Mftf\Helper\SelectText" method="selectText" stepKey="selectHeadingTextInTinyMCE">
101+
<argument name="context">//div[contains(@class, 'inline-wysiwyg')]//h2</argument>
102+
<argument name="startX">{{TinyMCEPartialHeadingSelection.startX}}</argument>
103+
<argument name="startY">{{TinyMCEPartialHeadingSelection.startY}}</argument>
104+
<argument name="endX">{{TinyMCEPartialHeadingSelection.endX}}</argument>
105+
<argument name="endY">{{TinyMCEPartialHeadingSelection.endY}}</argument>
106+
</helper>
107+
```
108+
109+
### Notes about the XML
110+
111+
1. Specify an argument value for every argument that matches our PHP implementation. This allows us to pass other test data to the Custom Helper.
112+
1. The `class` attribute matches the namespace specified in the PHP file.
113+
1. Specify the method from the class via the `method` attribute.
114+
1. If the function has a return value, it will be assigned to the `stepKey` variable. In this case, `$selectHeadingTextInTinyMCE` holds the return value.
115+
1. The types of argument values must match the PHP implementation's expected types.
116+
117+
## Key takeaways
118+
119+
Custom helpers allow you to solve complex use cases such as conditional logic, iteration, or complex WebDriver usage.
120+
121+
With access to the WebDriver object, you have a lot of flexibility available to you. See the [Codeception WebDriver](https://github.com/Codeception/module-webdriver/blob/master/src/Codeception/Module/WebDriver.php) for technical details and functionality available for use.
122+
123+
A custom helper is written in a PHP file and then referenced in test XML, like other actions.
124+
125+
You should only use these as a last resort after trying to implement your test using built-in actions.
126+
127+
## References
128+
129+
[Codeception WebDriver source code](https://github.com/Codeception/module-webdriver/blob/master/src/Codeception/Module/WebDriver.php) - Reference for using the WebDriver Object

docs/test.md

Lines changed: 15 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
# Test
22

33
Test cases in the Magento Functional Testing Framework (MFTF) are defined in XML as [`<tests>`].
4-
`<tests>` is a [Codeception test container][Codeception] that contains multiple individual tests with test metadata and before and after actions.
4+
`<tests>` is a [Codeception test container][Codeception] that contains individual test [`<test>`] with its metadata ([`<annotations>`]), before ([`<before>`]) and after ([`<after>`]) section.
55

6-
MFTF `<tests>` is considered a sequence of actions with associated parameters.
6+
MFTF `<test>` is considered a sequence of actions with associated parameters.
77
Any failed [assertion] within a test constitutes a failed test.
88

99
<div class="bs-callout bs-callout-info" markdown="1">
@@ -18,7 +18,7 @@ The following diagram shows the structure of an MFTF test case:
1818

1919
## Format
2020

21-
The format of `<tests>` is:
21+
The format of a test XML file is:
2222

2323
```xml
2424
<?xml version="1.0" encoding="UTF-8"?>
@@ -44,26 +44,23 @@ The format of `<tests>` is:
4444

4545
The following conventions apply to MFTF tests:
4646

47-
* All names within the framework are in the CamelCase format.
48-
* `<test>` name must be alphanumeric.
49-
* Each action and action group has its own identifier `<stepKey>` for reference purposes.
50-
* A test may have any number of [assertions][assertion] at any point within the `<test>`.
51-
* If `<test>` is included in `<suite>`, it **cannot be generated in isolation** to the rest of the contents of the suite (see [suites] for details).
5247
* One `<test>` tag is allowed per test XML file.
48+
* All names within the framework are in the **PascalCase** format and must be alphanumeric.
49+
* Each action and action group call should have its own identifier `<stepKey>`.
50+
* A test may have any number of [assertions][assertion] at any point within the `<test>`.
51+
* If `<test>` is included in [`<suite>`][suites], it **cannot be generated in isolation** from `<before>` and `<after>` section of the suite (see [suites] for details).
5352

5453
## Elements reference
5554

56-
There are several XML elements that are used in `<tests>` in the MFTF.
55+
There are several XML elements that are used within `<test>` in the MFTF.
5756

5857
### tests {#tests-tag}
5958

60-
`<tests>` is a container for multiple tests. It is a group of test methods that define test flows within a test case.
61-
62-
`<tests>` must contain at least one [`<test>`].
59+
`<tests>` is a container for test and must contain exactly one [`<test>`].
6360

6461
### test {#test-tag}
6562

66-
`<test>` is a set of steps, including [actions] and [assertions][assertion]. It is a sequence of test steps that define test flow within a test method.
63+
`<test>` is a set of steps, including [actions], [assertions][assertion] and Action Group calls. It is a sequence of test steps that define test flow within a test method.
6764

6865
Attribute|Type|Use|Description
6966
---|---|---|---
@@ -85,21 +82,20 @@ Allure annotations provide metadata for reporting.
8582

8683
### before {#before-tag}
8784

88-
`<before>` wraps the steps to perform before the [`<test>`].
85+
`<before>` wraps the steps that are preconditions for the [`<test>`]. For example: Change configuration, create Customer Account, Create Category and Product.
8986

9087
`<before>` may contain these child elements:
9188

92-
* Any [`<action>`][actions]
93-
* [`<actionGroup>`]
89+
* Any [Action][actions]
90+
* [`<actionGroup>`]s
9491

9592
### after {#after-tag}
9693

97-
`<after>` wraps the steps to perform after the [`<test>`].
98-
The steps are run in both successful **and** failed test runs.
94+
`<after>` wraps the steps to perform after the [`<test>`]. The steps are run in both successful **and** failed test runs. The goal of this section is to perform cleanup (revert the environment to the pre-test state).
9995

10096
`<after>` may contain:
10197

102-
* Any [`<action>`][actions]
98+
* Any [Action][actions]
10399
* [`<actionGroup>`]
104100

105101
### actionGroup {#actiongroup-tag}

src/Magento/FunctionalTestingFramework/Allure/Adapter/MagentoAllureAdapter.php

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -398,7 +398,7 @@ private function retrieveStepKey($stepLine)
398398
private function removeAttachments($step, $testFailed)
399399
{
400400
//Remove Attachments if verbose flag is not true AND test did not fail
401-
if (getenv('VERBOSE_ARTIFACTS') !== true && $testFailed === null) {
401+
if (getenv('VERBOSE_ARTIFACTS') !== "true" && $testFailed === null) {
402402
foreach ($step->getAttachments() as $index => $attachment) {
403403
$step->removeAttachment($index);
404404
unlink(Provider::getOutputDirectory() . DIRECTORY_SEPARATOR . $attachment->getSource());

0 commit comments

Comments
 (0)