Skip to content

Commit 531673e

Browse files
philimon-resetphilimon-reset
authored
Merge pull request #654 from mozilla/philimon/l10n_docs
Philimon/L10N_README
2 parents d48c29f + 31beb69 commit 531673e

File tree

1 file changed

+161
-0
lines changed

1 file changed

+161
-0
lines changed

‎l10n_CM/README.md

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
# Test Harness Documentation: L10N Test Framework
2+
3+
## Overview
4+
5+
This test harness is designed to automate testing of the Credential Management Autofill feature across different websites and regions combinations. It supports running tests for different combinations of sites (like Amazon, Walmart, etc.) and regions (US, CA, DE, FR), with optional configuration flags.
6+
7+
## Key Features
8+
9+
- **Multi-Region Testing**: Run tests across US, CA, DE, and FR regions
10+
- **Multi-Site Testing**: Support for multiple sites including demo, amazon, walmart, mediamarkt, and others
11+
- **Local Server**: Automatically starts and manages a local HTTP server for testing
12+
- **Test Filtering**: Automatically skips tests based on configuration in JSON files
13+
- **Configurable Test Execution**: Supports various pytest flags and options
14+
15+
## Configuration Files
16+
17+
The harness relies on several configuration files:
18+
19+
- **Region Test Configuration**: Located in `region/{region_name}.json`, defining tests for specific regions and also the sites supported for the region.
20+
```aiignore
21+
# example for CA region
22+
{
23+
"region": "CA",
24+
"sites": [
25+
"amazon",
26+
"walmart",
27+
"demo",
28+
"etsy"
29+
],
30+
"tests": [
31+
]
32+
}
33+
```
34+
- **Site Test Configuration**: Located in `constants/{site_name}.json` or `constants/{site_name}/{region}.json`, defining site-specific test configurations and skipped tests. Here we map the attributes for either `AutofillAddressBase` or `CreditCardBase` to the corresponding selectors.
35+
36+
```aiignore
37+
{
38+
# example for calvinklein address site
39+
"url": "http://127.0.0.1:8080/calvinklein_ad.html",
40+
# map class attributes to selector values.
41+
"field_mapping": {
42+
"given_name": "7d7a6c7c-7084-477e-afda-b27c826032a8",
43+
"family_name": "179a5ec2-4a98-45e9-9806-abb5496700d0",
44+
"street_address": "a9506204-8cee-4723-acb0-91307b9ae8ef",
45+
"address_level_2": "3fdac58a-13b3-4ca5-a5fb-8b5cc9712360",
46+
"address_level_1": "d7171e42-f617-4093-bca7-d655ac58f2c9",
47+
"postal_code": "a2d9e97a-4dab-44cf-960f-f5bbc1db447a",
48+
"email": "e4ebe951-0ed1-4ea9-b2c2-518b7b2ec034",
49+
"telephone": "57aab032-b31d-4bf9-8b7a-3fc5f265f6fc"
50+
},
51+
# define the selector key
52+
"form_field": "*[data-moz-autofill-inspect-id='{given_name}']",
53+
# state whether to skip all tests for this section (address or credit card)
54+
"skip": "True",
55+
# list of selector values for easy access
56+
"fields": [
57+
"7d7a6c7c-7084-477e-afda-b27c826032a8",
58+
"179a5ec2-4a98-45e9-9806-abb5496700d0",
59+
"a9506204-8cee-4723-acb0-91307b9ae8ef",
60+
"3fdac58a-13b3-4ca5-a5fb-8b5cc9712360",
61+
"d7171e42-f617-4093-bca7-d655ac58f2c9",
62+
"a2d9e97a-4dab-44cf-960f-f5bbc1db447a",
63+
"e4ebe951-0ed1-4ea9-b2c2-518b7b2ec034",
64+
"57aab032-b31d-4bf9-8b7a-3fc5f265f6fc"
65+
],
66+
# list of individual tests to skip for this configuration run.
67+
"skipped": []
68+
}
69+
70+
```
71+
## Usage
72+
73+
```bash
74+
python run_l10n.py [FLAGS] [REGIONS] [SITES]
75+
```
76+
77+
### Parameters
78+
79+
- **FLAGS**: Additional flags for customizing test execution
80+
- **REGIONS**: One or more region codes (US, CA, DE, FR)
81+
- **SITES**: One or more site names (demo, amazon, walmart, etc.)
82+
83+
### Supported Flags
84+
85+
- `--run-headless`: Run tests in headless mode
86+
- `-n [num]`: Specify number of parallel workers
87+
- `--reruns [num]`: Specify number of test reruns on failure
88+
- `--fx-executable`: Specify Firefox executable path
89+
- `--ci`: Enable CI mode
90+
91+
### Examples
92+
93+
```bash
94+
# Run tests for all regions and sites
95+
python run_l10n.py
96+
97+
# Run all tests for US region only across all available sites
98+
python run_l10n.py US
99+
100+
# Run all tests for amazon site only across all regions
101+
python run_l10n.py US
102+
103+
# Run tests for US region on amazon site
104+
python run_l10n.py US amazon
105+
106+
# Run tests for US and CA regions with 4 parallel workers
107+
python run_l10n.py -n 4 US CA
108+
109+
# Run tests for US region on amazon site in headless mode
110+
python run_l10n.py --run-headless US amazon
111+
```
112+
113+
## Architecture
114+
115+
### Core Components
116+
117+
1. **Test Runner**: Orchestrates test execution with proper environment variables and pytest flags
118+
2. **Local HTTP Server**: Serves site-specific HTML files for testing
119+
3. **Test Selector**: Selects appropriate tests based on region and site configuration
120+
4. **Flag Processor**: Validates and processes command-line arguments
121+
122+
### Test Execution Flow
123+
124+
1. Command-line arguments are parsed and validated
125+
2. Site and region combinations are determined
126+
3. For each combination:
127+
- Required tests are identified from configuration files
128+
- Tests to be skipped are filtered out
129+
- A local HTTP server is started (except for "demo" site)
130+
- Tests are executed with specified flags
131+
- Results are logged
132+
133+
## Environment Variables
134+
135+
The framework sets the following environment variables during test execution:
136+
137+
- `CM_SITE`: The site being tested
138+
- `FX_REGION`: The region being tested
139+
- `TEST_EXIT_CODE`: Exit code of the test execution
140+
141+
## Adding New Tests
142+
143+
1. Create test files in the `Unified` directory
144+
2. Update the region configuration in `region/{region_name}.json` if new test is region specific.
145+
3. If needed, update site-specific configurations to handle skipped tests.
146+
147+
## Adding New Regions/Sites
148+
149+
1. Add the new region/site to `valid_region`/`valid_sites` in the script
150+
2. If new region, add a new mapping file to `region/` (must be capitalized abbreviation of the region `US.json`).
151+
3. If new site, create a directory with the name of the site in `constants/` and another subdirectory for the region supported for that site.
152+
4. Create necessary HTML files in `sites/{site_name}/{region}/` directory.
153+
5. Create the necessary mapping files in `constants/{site_name}/{region}/`
154+
* The naming convention for both the mapping files and the HTML files are `{site}_ad` for Address Pages and `{site}_cc` for Credit Card Pages.
155+
156+
## Troubleshooting
157+
158+
- **Invalid Arguments**: Ensure all region codes, site names, and flags are valid
159+
- **Missing HTML Files**: Verify that HTML files exist at `sites/{site_name}/{region}/`
160+
- **Missing JSON Files**: Check for proper configuration files in `region/` and `constants/` directories
161+
- **Test Failures**: Review test execution logs for details on failures

0 commit comments

Comments
 (0)