Skip to content

Commit 464139c

Browse files
Harsh-MicrosoftHarsh-Microsoft
authored
test: Migrate test automation scripts, llm evaluator and pipeline for CWYD (#1818)
1 parent 69669bb commit 464139c

28 files changed

+1319
-1
lines changed

‎.github/workflows/test-automation.yml

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
name: Test Automation CWYD
2+
on:
3+
push:
4+
branches:
5+
- main
6+
- dev
7+
paths:
8+
- 'tests/e2e-test/**'
9+
schedule:
10+
- cron: '0 13 * * 3' # Runs at 1 PM UTC once a week on Wednesday
11+
workflow_dispatch:
12+
13+
env:
14+
web_url: ${{ vars.CWYD_WEB_URL }}
15+
admin_url: ${{ vars.CWYD_ADMIN_URL }}
16+
accelerator_name: "Chat with your Data"
17+
18+
jobs:
19+
test:
20+
runs-on: ubuntu-latest
21+
steps:
22+
- name: Checkout repository
23+
uses: actions/checkout@v4
24+
25+
- name: Set up Python
26+
uses: actions/setup-python@v4
27+
with:
28+
python-version: '3.13'
29+
30+
- name: Install dependencies
31+
run: |
32+
python -m pip install --upgrade pip
33+
pip install -r tests/e2e-test/requirements.txt
34+
35+
- name: Ensure browsers are installed
36+
run: python -m playwright install --with-deps chromium
37+
38+
- name: Run tests(1)
39+
id: test1
40+
run: |
41+
xvfb-run pytest --headed --html=report/report.html --self-contained-html
42+
working-directory: tests/e2e-test
43+
continue-on-error: true
44+
45+
- name: Sleep for 30 seconds
46+
if: ${{ steps.test1.outcome == 'failure' }}
47+
run: sleep 30s
48+
shell: bash
49+
50+
- name: Run tests(2)
51+
id: test2
52+
if: ${{ steps.test1.outcome == 'failure' }}
53+
run: |
54+
xvfb-run pytest --headed --html=report/report.html --self-contained-html
55+
working-directory: tests/e2e-test
56+
continue-on-error: true
57+
58+
- name: Sleep for 60 seconds
59+
if: ${{ steps.test2.outcome == 'failure' }}
60+
run: sleep 60s
61+
shell: bash
62+
63+
- name: Run tests(3)
64+
id: test3
65+
if: ${{ steps.test2.outcome == 'failure' }}
66+
run: |
67+
xvfb-run pytest --headed --html=report/report.html --self-contained-html
68+
working-directory: tests/e2e-test
69+
70+
- name: Upload test report
71+
id: upload_report
72+
uses: actions/upload-artifact@v4
73+
if: ${{ !cancelled() }}
74+
with:
75+
name: cwyd-test-report
76+
path: tests/e2e-test/report/*
77+
78+
- name: Send Notification
79+
if: always()
80+
run: |
81+
RUN_URL="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
82+
REPORT_URL=${{ steps.upload_report.outputs.artifact-url }}
83+
IS_SUCCESS=${{ steps.test1.outcome == 'success' || steps.test2.outcome == 'success' || steps.test3.outcome == 'success' }}
84+
# Construct the email body
85+
if [ "$IS_SUCCESS" = "true" ]; then
86+
EMAIL_BODY=$(cat <<EOF
87+
{
88+
"body": "<p>Dear Team,</p><p>We would like to inform you that the ${{ env.accelerator_name }} Test Automation process has completed successfully.</p><p><strong>Run URL:</strong> <a href=\"${RUN_URL}\">${RUN_URL}</a><br></p><p><strong>Test Report:</strong> <a href=\"${REPORT_URL}\">${REPORT_URL}</a></p><p>Best regards,<br>Your Automation Team</p>",
89+
"subject": "${{ env.accelerator_name }} Test Automation - Success"
90+
}
91+
EOF
92+
)
93+
else
94+
EMAIL_BODY=$(cat <<EOF
95+
{
96+
"body": "<p>Dear Team,</p><p>We would like to inform you that the ${{ env.accelerator_name }} Test Automation process has encountered an issue and has failed to complete successfully.</p><p><strong>Run URL:</strong> <a href=\"${RUN_URL}\">${RUN_URL}</a><br> ${OUTPUT}</p><p><strong>Test Report:</strong> <a href=\"${REPORT_URL}\">${REPORT_URL}</a></p><p>Please investigate the matter at your earliest convenience.</p><p>Best regards,<br>Your Automation Team</p>",
97+
"subject": "${{ env.accelerator_name }} Test Automation - Failure"
98+
}
99+
EOF
100+
)
101+
fi
102+
103+
# Send the notification
104+
curl -X POST "${{ secrets.EMAILNOTIFICATION_LOGICAPP_URL_TA }}" \
105+
-H "Content-Type: application/json" \
106+
-d "$EMAIL_BODY" || echo "Failed to send notification"

‎.github/workflows/tests.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ jobs:
7878
7979
echo "MIN_COVERAGE=$MIN_COVERAGE" >> "$GITHUB_OUTPUT"
8080
- name: Run Python Tests
81-
run: make python-test optional_args="--junitxml=coverage-junit.xml --cov=. --cov-report xml:coverage.xml --cov-fail-under ${{ steps.coverage-value.outputs.MIN_COVERAGE }}"
81+
run: make python-test optional_args="--junitxml=coverage-junit.xml --cov=. --cov-report xml:coverage.xml --cov-fail-under ${{ steps.coverage-value.outputs.MIN_COVERAGE }} ./code/tests"
8282
- uses: actions/upload-artifact@v4
8383
if: ${{ !cancelled() }}
8484
with:

‎.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -418,6 +418,8 @@ temp/
418418
# so that Azure App Service can install the dependencies
419419
requirements.txt
420420
!infra/prompt-flow/cwyd/requirements.txt
421+
!tests/e2e-test/requirements.txt
422+
!tests/llm-evaluator/requirements.txt
421423

422424
# Cypress UI tests screenshots folder
423425
tests/integration/ui/cypress/screenshots/

‎tests/e2e-test/.gitignore

Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
build/
12+
develop-eggs/
13+
dist/
14+
downloads/
15+
eggs/
16+
.eggs/
17+
lib/
18+
lib64/
19+
parts/
20+
sdist/
21+
var/
22+
wheels/
23+
share/python-wheels/
24+
*.egg-info/
25+
.installed.cfg
26+
*.egg
27+
MANIFEST
28+
29+
# PyInstaller
30+
# Usually these files are written by a python script from a template
31+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
32+
*.manifest
33+
*.spec
34+
35+
# Installer logs
36+
pip-log.txt
37+
pip-delete-this-directory.txt
38+
39+
# Unit test / coverage reports
40+
htmlcov/
41+
.tox/
42+
.nox/
43+
.coverage
44+
.coverage.*
45+
.cache
46+
nosetests.xml
47+
coverage.xml
48+
*.cover
49+
*.py,cover
50+
.hypothesis/
51+
.pytest_cache/
52+
cover/
53+
report.html
54+
55+
# Translations
56+
*.mo
57+
*.pot
58+
59+
# Django stuff:
60+
*.log
61+
local_settings.py
62+
db.sqlite3
63+
db.sqlite3-journal
64+
65+
# Flask stuff:
66+
instance/
67+
.webassets-cache
68+
69+
# Scrapy stuff:
70+
.scrapy
71+
72+
# Sphinx documentation
73+
docs/_build/
74+
75+
# PyBuilder
76+
.pybuilder/
77+
target/
78+
79+
# Jupyter Notebook
80+
.ipynb_checkpoints
81+
82+
# IPython
83+
profile_default/
84+
ipython_config.py
85+
86+
# pyenv
87+
# For a library or package, you might want to ignore these files since the code is
88+
# intended to run in multiple environments; otherwise, check them in:
89+
# .python-version
90+
91+
# pipenv
92+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
93+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
94+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
95+
# install all needed dependencies.
96+
#Pipfile.lock
97+
98+
# poetry
99+
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
100+
# This is especially recommended for binary packages to ensure reproducibility, and is more
101+
# commonly ignored for libraries.
102+
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
103+
#poetry.lock
104+
105+
# pdm
106+
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
107+
#pdm.lock
108+
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
109+
# in version control.
110+
# https://pdm.fming.dev/latest/usage/project/#working-with-version-control
111+
.pdm.toml
112+
.pdm-python
113+
.pdm-build/
114+
115+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
116+
__pypackages__/
117+
118+
# Celery stuff
119+
celerybeat-schedule
120+
celerybeat.pid
121+
122+
# SageMath parsed files
123+
*.sage.py
124+
125+
# Environments
126+
.env
127+
.venv
128+
env/
129+
venv/
130+
ENV/
131+
env.bak/
132+
venv.bak/
133+
microsoft/
134+
135+
# Spyder project settings
136+
.spyderproject
137+
.spyproject
138+
139+
# Rope project settings
140+
.ropeproject
141+
142+
# mkdocs documentation
143+
/site
144+
145+
# mypy
146+
.mypy_cache/
147+
.dmypy.json
148+
dmypy.json
149+
150+
# Pyre type checker
151+
.pyre/
152+
153+
# pytype static type analyzer
154+
.pytype/
155+
156+
# Cython debug symbols
157+
cython_debug/
158+
159+
# PyCharm
160+
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
161+
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
162+
# and can be added to the global gitignore or merged into this file. For a more nuclear
163+
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
164+
.idea/
165+
archive/
166+
report/
167+
screenshots/
168+
videos/

‎tests/e2e-test/README.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Test Automation for Chat with your Data Accelerator
2+
3+
4+
5+
Write end-to-end tests for your web apps with [Playwright](https://github.com/microsoft/playwright-python) and [pytest](https://docs.pytest.org/en/stable/).
6+
7+
- Support for **all modern browsers** including Chromium, WebKit and Firefox.
8+
- Support for **headless and headed** execution.
9+
- **Built-in fixtures** that provide browser primitives to test functions.
10+
11+
Pre-Requisites:
12+
- Install Visual Studio Code: Download and Install Visual Studio Code(VSCode).
13+
- Install NodeJS: Download and Install Node JS
14+
15+
Create and Activate Python Virtual Environment
16+
- From your directory open and run cmd : "python -m venv microsoft"
17+
This will create a virtual environment directory named microsoft inside your current directory
18+
- To enable virtual environment, copy location for "microsoft\Scripts\activate.bat" and run from cmd
19+
20+
21+
Installing Playwright Pytest from Virtual Environment
22+
- To install libraries run "pip install -r requirements.txt"
23+
- Install the required browsers "playwright install"
24+
25+
Run test cases
26+
- To run test cases from your 'tests' folder : "pytest --headed --html=report/report.html"
27+
28+
Steps need to be followed to enable Access Token and Client Credentials
29+
- Go to App Service from the resource group and select the Access Tokens check box in 'Manage->Authentication' tab
30+
![img.png](img.png)
31+
- Go to Manage->Certificates & secrets tab to generate Client Secret value
32+
![img_1.png](img_1.png)
33+
- Go to Overview tab to get the client id and tenant id.
34+
35+
Create .env file in project root level with web app url and client credentials
36+
- create a .env file in project root level and add your user_name, pass_word, client_id,client_secret,
37+
tenant_id, web_url and admin_url for the resource group. please refer 'sample_dotenv_file.txt' file.
38+
39+
## Documentation
40+
41+
See on [playwright.dev](https://playwright.dev/python/docs/test-runners) for examples and more detailed information.

‎tests/e2e-test/base/__init__.py

Whitespace-only changes.

‎tests/e2e-test/base/base.py

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
import json
2+
import os
3+
import uuid
4+
5+
from dotenv import load_dotenv
6+
7+
8+
class BasePage:
9+
10+
def __init__(self, page):
11+
self.page = page
12+
13+
def scroll_into_view(self, locator):
14+
reference_list = locator
15+
locator.nth(reference_list.count() - 1).scroll_into_view_if_needed()
16+
17+
def select_an_element(self, locator, text):
18+
elements = locator.all()
19+
for element in elements:
20+
clientele = element.text_content()
21+
if clientele == text:
22+
element.click()
23+
break
24+
25+
def is_visible(self, locator):
26+
locator.is_visible()
27+
28+
def validate_response_status(self, questions):
29+
load_dotenv()
30+
WEB_URL = os.getenv("web_url")
31+
32+
url = f"{WEB_URL}/api/conversation"
33+
34+
user_message_id = str(uuid.uuid4())
35+
assistant_message_id = str(uuid.uuid4())
36+
conversation_id = str(uuid.uuid4())
37+
38+
payload = {
39+
"messages": [{"role": "user", "content": questions, "id": user_message_id}],
40+
"conversation_id": conversation_id,
41+
}
42+
# Serialize the payload to JSON
43+
payload_json = json.dumps(payload)
44+
headers = {"Content-Type": "application/json", "Accept": "*/*"}
45+
response = self.page.request.post(url, headers=headers, data=payload_json)
46+
# Check the response status code
47+
assert response.status == 200, (
48+
"response code is " + str(response.status) + " " + str(response.json())
49+
)
50+
51+
def wait_for_load(self, wait_time):
52+
self.page.wait_for_timeout(wait_time)

0 commit comments

Comments
 (0)