Skip to content

Commit 640b829

Browse files
filip-michalskymiguelg719
filip-michalsky
and
miguelg719
authored
Fm/stg 474 more tests (#83)
* update readme * update env example * update * update to pip * update examples and README * do not require bb key for local runs * update example * format * formatting * format; git commit -m ;format * remove saving the tree * one shot test structure * fixing tests * fixing more tests * fix more tests * update tests * fixing tests * all tests pass * remove warnings * fix formatting * fix: update deprecated GitHub Actions upload/download-artifact from v3 to v4 * make google cua optional import, fix stagehand import * update tests * format * update cua to CI * fix linter * remove min coverage threshold * run tests in CI attempt * remove tests from ruff * more debug to pass ci * more ci fixes * cleaning up some unit tests * remove stuff from publish yaml * revert * fix test * revert types back from schema * add note todo * select tests based on PR labels * first pass on integration tests * updates to integration tests * fix unit tests * revert pr template, extract handler, remove test readme * reverting more files except test folder * revert more files * revert more files * revert examples * revert example * trim unit tests * trim tests * fix unit test * fix smoke test warnings * update tests * update test CI workflow * updates * update core integration test * change local integration test * ci passing --------- Co-authored-by: miguel <[email protected]>
1 parent 4986c27 commit 640b829

30 files changed

+5714
-1278
lines changed

‎ .env.example renamed to ‎.env.example

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
1-
MODEL_API_KEY = "anthropic-or-openai-api-key"
1+
MODEL_API_KEY = "your-favorite-llm-api-key"
22
BROWSERBASE_API_KEY = "browserbase-api-key"
33
BROWSERBASE_PROJECT_ID = "browserbase-project-id"
44
STAGEHAND_API_URL = "api_url"
5+
STAGEHAND_ENV= "LOCAL or BROWSERBASE"

‎.github/workflows/test.yml

Lines changed: 292 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,292 @@
1+
name: Test Suite
2+
3+
on:
4+
push:
5+
branches: [ main, develop ]
6+
pull_request:
7+
branches: [ main, develop ]
8+
types: [opened, synchronize, reopened, labeled, unlabeled]
9+
10+
jobs:
11+
test-unit:
12+
name: Unit Tests
13+
runs-on: ubuntu-latest
14+
strategy:
15+
matrix:
16+
python-version: ["3.9", "3.10", "3.11", "3.12"]
17+
18+
steps:
19+
- uses: actions/checkout@v4
20+
21+
- name: Set up Python ${{ matrix.python-version }}
22+
uses: actions/setup-python@v4
23+
with:
24+
python-version: ${{ matrix.python-version }}
25+
26+
- name: Cache pip dependencies
27+
uses: actions/cache@v3
28+
with:
29+
path: ~/.cache/pip
30+
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt', '**/pyproject.toml') }}
31+
restore-keys: |
32+
${{ runner.os }}-pip-
33+
34+
- name: Install dependencies
35+
run: |
36+
python -m pip install --upgrade pip
37+
pip install -e ".[dev]"
38+
# Install jsonschema for schema validation tests
39+
pip install jsonschema
40+
# Install temporary Google GenAI wheel
41+
pip install temp/google_genai-1.14.0-py3-none-any.whl
42+
43+
- name: Run unit tests
44+
run: |
45+
pytest tests/unit/ -v --junit-xml=junit-unit-${{ matrix.python-version }}.xml
46+
47+
- name: Upload unit test results
48+
uses: actions/upload-artifact@v4
49+
if: always()
50+
with:
51+
name: unit-test-results-${{ matrix.python-version }}
52+
path: junit-unit-${{ matrix.python-version }}.xml
53+
54+
test-integration-local:
55+
name: Local Integration Tests
56+
runs-on: ubuntu-latest
57+
needs: test-unit
58+
59+
steps:
60+
- uses: actions/checkout@v4
61+
62+
- name: Set up Python 3.11
63+
uses: actions/setup-python@v4
64+
with:
65+
python-version: "3.11"
66+
67+
- name: Install system dependencies
68+
run: |
69+
sudo apt-get update
70+
sudo apt-get install -y xvfb
71+
72+
- name: Install dependencies
73+
run: |
74+
python -m pip install --upgrade pip
75+
pip install -e ".[dev]"
76+
pip install jsonschema
77+
# Install temporary Google GenAI wheel
78+
pip install temp/google_genai-1.14.0-py3-none-any.whl
79+
# Install Playwright browsers for integration tests
80+
playwright install chromium
81+
playwright install-deps chromium
82+
83+
- name: Run local integration tests
84+
run: |
85+
# Run integration tests marked as 'integration' and 'local'
86+
xvfb-run -a pytest tests/integration/ -v \
87+
--cov=stagehand \
88+
--cov-report=xml \
89+
--junit-xml=junit-integration-local.xml \
90+
-m "integration and local" \
91+
--tb=short \
92+
--maxfail=5
93+
env:
94+
MODEL_API_KEY: ${{ secrets.MODEL_API_KEY || 'mock-model-key' }}
95+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY || 'mock-openai-key' }}
96+
DISPLAY: ":99"
97+
98+
- name: Upload integration test results
99+
uses: actions/upload-artifact@v4
100+
if: always()
101+
with:
102+
name: integration-test-results-local
103+
path: junit-integration-local.xml
104+
105+
- name: Upload coverage data
106+
uses: actions/upload-artifact@v4
107+
if: always()
108+
with:
109+
name: coverage-data-integration-local
110+
path: |
111+
.coverage
112+
coverage.xml
113+
114+
test-integration-api:
115+
name: API Integration Tests
116+
runs-on: ubuntu-latest
117+
needs: test-unit
118+
119+
steps:
120+
- uses: actions/checkout@v4
121+
122+
- name: Set up Python 3.11
123+
uses: actions/setup-python@v4
124+
with:
125+
python-version: "3.11"
126+
127+
- name: Install dependencies
128+
run: |
129+
python -m pip install --upgrade pip
130+
pip install -e ".[dev]"
131+
pip install jsonschema
132+
# Install temporary Google GenAI wheel
133+
pip install temp/google_genai-1.14.0-py3-none-any.whl
134+
135+
- name: Run API integration tests
136+
run: |
137+
pytest tests/integration/ -v \
138+
--cov=stagehand \
139+
--cov-report=xml \
140+
--junit-xml=junit-integration-api.xml \
141+
-m "integration and api" \
142+
--tb=short
143+
env:
144+
BROWSERBASE_API_KEY: ${{ secrets.BROWSERBASE_API_KEY }}
145+
BROWSERBASE_PROJECT_ID: ${{ secrets.BROWSERBASE_PROJECT_ID }}
146+
MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
147+
STAGEHAND_API_URL: ${{ secrets.STAGEHAND_API_URL }}
148+
149+
- name: Upload API integration test results
150+
uses: actions/upload-artifact@v4
151+
if: always()
152+
with:
153+
name: api-integration-test-results
154+
path: junit-integration-api.xml
155+
156+
smoke-tests:
157+
name: Smoke Tests
158+
runs-on: ubuntu-latest
159+
160+
steps:
161+
- uses: actions/checkout@v4
162+
163+
- name: Set up Python 3.11
164+
uses: actions/setup-python@v4
165+
with:
166+
python-version: "3.11"
167+
168+
- name: Install dependencies
169+
run: |
170+
python -m pip install --upgrade pip
171+
pip install -e ".[dev]"
172+
pip install jsonschema
173+
pip install temp/google_genai-1.14.0-py3-none-any.whl
174+
175+
- name: Run smoke tests
176+
run: |
177+
pytest tests/ -v \
178+
--junit-xml=junit-smoke.xml \
179+
-m "smoke" \
180+
--tb=line \
181+
--maxfail=5
182+
183+
- name: Upload smoke test results
184+
uses: actions/upload-artifact@v4
185+
if: always()
186+
with:
187+
name: smoke-test-results
188+
path: junit-smoke.xml
189+
190+
test-e2e:
191+
name: End-to-End Tests
192+
runs-on: ubuntu-latest
193+
needs: test-unit
194+
if: |
195+
contains(github.event.pull_request.labels.*.name, 'test-e2e') ||
196+
contains(github.event.pull_request.labels.*.name, 'e2e')
197+
198+
steps:
199+
- uses: actions/checkout@v4
200+
201+
- name: Set up Python 3.11
202+
uses: actions/setup-python@v4
203+
with:
204+
python-version: "3.11"
205+
206+
- name: Install system dependencies
207+
run: |
208+
sudo apt-get update
209+
sudo apt-get install -y xvfb
210+
211+
- name: Install dependencies
212+
run: |
213+
python -m pip install --upgrade pip
214+
pip install -e ".[dev]"
215+
pip install jsonschema
216+
# Install temporary Google GenAI wheel
217+
pip install temp/google_genai-1.14.0-py3-none-any.whl
218+
playwright install chromium
219+
playwright install-deps chromium
220+
221+
- name: Run E2E tests
222+
run: |
223+
xvfb-run -a pytest tests/ -v \
224+
--cov=stagehand \
225+
--cov-report=xml \
226+
--junit-xml=junit-e2e.xml \
227+
-m "e2e" \
228+
--tb=short
229+
env:
230+
BROWSERBASE_API_KEY: ${{ secrets.BROWSERBASE_API_KEY || 'mock-api-key' }}
231+
BROWSERBASE_PROJECT_ID: ${{ secrets.BROWSERBASE_PROJECT_ID || 'mock-project-id' }}
232+
MODEL_API_KEY: ${{ secrets.MODEL_API_KEY || 'mock-model-key' }}
233+
STAGEHAND_API_URL: ${{ secrets.STAGEHAND_API_URL || 'http://localhost:3000' }}
234+
DISPLAY: ":99"
235+
236+
- name: Upload E2E test results
237+
uses: actions/upload-artifact@v4
238+
if: always()
239+
with:
240+
name: e2e-test-results
241+
path: junit-e2e.xml
242+
243+
test-all:
244+
name: Complete Test Suite
245+
runs-on: ubuntu-latest
246+
needs: test-unit
247+
if: |
248+
contains(github.event.pull_request.labels.*.name, 'test-all') ||
249+
contains(github.event.pull_request.labels.*.name, 'full-test')
250+
251+
steps:
252+
- uses: actions/checkout@v4
253+
254+
- name: Set up Python 3.11
255+
uses: actions/setup-python@v4
256+
with:
257+
python-version: "3.11"
258+
259+
- name: Install dependencies
260+
run: |
261+
python -m pip install --upgrade pip
262+
pip install -e ".[dev]"
263+
pip install jsonschema
264+
# Install temporary Google GenAI wheel
265+
pip install temp/google_genai-1.14.0-py3-none-any.whl
266+
playwright install chromium
267+
268+
- name: Run complete test suite
269+
run: |
270+
pytest tests/ -v \
271+
--cov=stagehand \
272+
--cov-report=xml \
273+
--cov-report=html \
274+
--junit-xml=junit-all.xml \
275+
--maxfail=10 \
276+
--tb=short
277+
env:
278+
BROWSERBASE_API_KEY: ${{ secrets.BROWSERBASE_API_KEY }}
279+
BROWSERBASE_PROJECT_ID: ${{ secrets.BROWSERBASE_PROJECT_ID }}
280+
MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
281+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
282+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
283+
STAGEHAND_API_URL: ${{ secrets.STAGEHAND_API_URL }}
284+
285+
- name: Upload complete test results
286+
uses: actions/upload-artifact@v4
287+
if: always()
288+
with:
289+
name: complete-test-results
290+
path: |
291+
junit-all.xml
292+
htmlcov/

0 commit comments

Comments
 (0)