|
| 1 | +# Enterprise QA Framework: ETL, API, UI Automation |
| 2 | + |
| 3 | +This project demonstrates an enterprise-grade End-to-End (E2E) Quality Assurance (QA) framework for a banking domain, covering ETL, API, and UI testing. It's designed for functional and integration testing, incorporating data-driven variability and CI/CD integration. |
| 4 | + |
| 5 | +## Features |
| 6 | + |
| 7 | +- **ETL Testing (MySQL):** |
| 8 | + - Validates data transformations and loads between `src`, `stg`, and `dw` (data warehouse) layers. |
| 9 | + - Uses `pytest-bdd` for Gherkin-style feature definitions (`.feature` files) for clear, business-readable test cases. |
| 10 | + - Includes scenarios for positive data flow and graceful handling of missing raw data. |
| 11 | +- **API Testing (FastAPI & `pytest-bdd`):** |
| 12 | + - Tests a mock banking API (FastAPI) that interacts with the `dw` layer. |
| 13 | + - Verifies customer balances and transaction retrieval. |
| 14 | + - Includes positive scenarios and negative testing (e.g., 404 for unknown customers). |
| 15 | +- **UI Testing (Playwright Python):** |
| 16 | + - Automates interactions with a public banking demo application ([GlobalSQA BankingProject](https://www.globalsqa.com/angularJs-protractor/BankingProject/#/login)). |
| 17 | + - Covers key workflows like Bank Manager login, adding new customers, opening accounts, and verifying customer lists. |
| 18 | + - Designed for cross-browser execution (though currently configured for Chromium in CI). |
| 19 | +- **E2E Pipeline Test:** |
| 20 | + - A single test demonstrating a full vertical slice: seeding data in `src`, simulating ETL, querying the API, and performing a basic UI check. |
| 21 | +- **Data-Driven Testing:** |
| 22 | + - Uses external YAML files (e.g., `data/login_cases.yaml`, `data/ach_cases.yaml`) for test data variability (though not fully implemented in all current tests, the structure is in place). |
| 23 | +- **CI/CD Integration (GitHub Actions):** |
| 24 | + - Automated workflow (`.github/workflows/ci.yml`) to run all tests on `push` and `pull_request`. |
| 25 | + - Sets up a MySQL service, Python environment, installs dependencies (including Playwright browsers), and executes ETL, API, and UI tests. |
| 26 | + |
| 27 | +## Project Structure |
| 28 | +. |
| 29 | +├── .github/ |
| 30 | +│ └── workflows/ |
| 31 | +│ └── ci.yml # GitHub Actions workflow for CI |
| 32 | +├── api/ |
| 33 | +│ └── main.py # FastAPI application for banking API |
| 34 | +├── data/ |
| 35 | +│ ├── ach_cases.yaml # Example data for ACH scenarios |
| 36 | +│ └── login_cases.yaml # Example data for login scenarios |
| 37 | +├── db/ |
| 38 | +│ └── init_mysql.sql # SQL script to initialize MySQL schema for CI |
| 39 | +├── features/ |
| 40 | +│ ├── api_validation.feature # Gherkin features for API tests |
| 41 | +│ └── etl_validation.feature # Gherkin features for ETL tests |
| 42 | +├── pages/ # Playwright Page Objects for UI interactions |
| 43 | +│ ├── add_customer_page.py |
| 44 | +│ ├── base_page.py |
| 45 | +│ ├── dashboard_page.py |
| 46 | +│ ├── login_page.py |
| 47 | +│ ├── open_account_page.py |
| 48 | +│ └── customers_page.py |
| 49 | +├── tests/ |
| 50 | +│ ├── api/ |
| 51 | +│ │ └── test_api_steps.py # Step definitions for API BDD tests |
| 52 | +│ ├── e2e/ |
| 53 | +│ │ ├── test_end_to_end_pipeline.py # E2E test across layers |
| 54 | +│ │ └── test_treasury_workflow.py # E2E UI workflow (placeholder) |
| 55 | +│ ├── etl/ |
| 56 | +│ │ └── test_etl_steps.py # Step definitions for ETL BDD tests |
| 57 | +│ └── ui/ |
| 58 | +│ ├── test_banking_manager_login.py # UI tests for banking app |
| 59 | +│ └── test_google_smoke.py # Playwright smoke test (can be removed) |
| 60 | +├── utils/ |
| 61 | +│ ├── config.py # Configuration (DB, API_BASE_URL) |
| 62 | +│ ├── data_loader.py # Utility for loading data files |
| 63 | +│ └── db_client.py # Database client for MySQL interactions |
| 64 | +├── .gitignore # Specifies intentionally untracked files |
| 65 | +├── pytest.ini # Pytest configuration |
| 66 | +└── requirements.txt # Python dependencies |
| 67 | + |
| 68 | + |
| 69 | +## Setup and Local Execution |
| 70 | + |
| 71 | +### Prerequisites |
| 72 | + |
| 73 | +- Python 3.11+ |
| 74 | +- MySQL 8.0+ (running locally on port 3307, with root user and password '080675' or configured in `utils/config.py`) |
| 75 | +- PyCharm (recommended IDE) |
| 76 | + |
| 77 | +### 1. Clone the repository |
| 78 | + |
| 79 | +```bash |
| 80 | +git clone https://github.com/<your-username>/data-qa-etl-api-ui-framework.git |
| 81 | +cd data-qa-etl-api-ui-framework |
| 82 | +2. Set up Python Virtual Environment and Install Dependencies |
| 83 | + |
| 84 | +python -m venv venv |
| 85 | +.\venv\Scripts\activate # On Windows |
| 86 | +# source venv/bin/activate # On macOS/Linux |
| 87 | + |
| 88 | +pip install -r requirements.txt |
| 89 | +playwright install --with-deps |
| 90 | +3. Initialize MySQL Databases |
| 91 | +Ensure your MySQL server is running. Then, execute the schema initialization script: |
| 92 | + |
| 93 | + |
| 94 | +mysql -h 127.0.0.1 -P 3307 -uroot -proot < db/init_mysql.sql |
| 95 | +4. Run the FastAPI Application (API Layer) |
| 96 | +Open a terminal and start the FastAPI server. Keep this running while executing API and E2E tests. |
| 97 | + |
| 98 | + |
| 99 | +uvicorn api.main:app --reload --port 8000 |
| 100 | +5. Run Tests Locally |
| 101 | +Open a separate terminal (with your virtual environment activated) and run pytest: |
| 102 | + |
| 103 | + |
| 104 | +# Run all tests |
| 105 | +pytest |
| 106 | + |
| 107 | +# Run specific test suites |
| 108 | +pytest tests/etl |
| 109 | +pytest tests/api |
| 110 | +pytest tests/ui |
| 111 | +pytest tests/e2e |
| 112 | + |
| 113 | +# Run UI tests in headed mode (browser visible) |
| 114 | +pytest tests/ui --headed |
| 115 | +CI/CD with GitHub Actions |
| 116 | +The .github/workflows/ci.yml workflow automatically runs all tests on every push and pull_request to the main branch. It sets up a MySQL service, installs dependencies, and executes the test suite in a clean environment. |
0 commit comments