Skip to content

Commit 2bdf2e9

Browse files
patchmemorypatchmemory
committed
CI: make E2E Playwright job strict (remove continue-on-error); verify Ready Queue no longer includes scaffold task at top. Ensure Dev CLI timestamps align (complete_task uses timezone-aware ISO8601).
1 parent cb51da3 commit 2bdf2e9

File tree

543 files changed

+154687
-15
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

543 files changed

+154687
-15
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [ main, master, '**' ]
6+
pull_request:
7+
branches: [ main, master, '**' ]
8+
9+
jobs:
10+
test-python:
11+
name: Python tests (pytest)
12+
runs-on: ubuntu-latest
13+
steps:
14+
- name: Checkout
15+
uses: actions/checkout@v4
16+
17+
- name: Set up Python
18+
uses: actions/setup-python@v5
19+
with:
20+
python-version: '3.11'
21+
22+
- name: Install Python dependencies
23+
run: |
24+
python -m pip install --upgrade pip
25+
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
26+
if [ -f pyproject.toml ]; then pip install -e . || true; fi
27+
28+
- name: Run pytest
29+
run: |
30+
python -m pytest -q
31+
32+
e2e-playwright:
33+
name: E2E smoke (Playwright)
34+
runs-on: ubuntu-latest
35+
needs: test-python
36+
steps:
37+
- name: Checkout
38+
uses: actions/checkout@v4
39+
40+
- name: Set up Node.js
41+
uses: actions/setup-node@v4
42+
with:
43+
node-version: '18'
44+
45+
- name: Install Node dependencies
46+
run: |
47+
if [ -f package.json ]; then npm ci || npm install; fi
48+
49+
- name: Install Playwright browsers and dependencies
50+
run: |
51+
npx playwright install --with-deps
52+
53+
- name: Run Playwright smoke tests
54+
env:
55+
# limit providers to local filesystem to avoid external deps during CI
56+
SCIDK_PROVIDERS: local_fs
57+
run: |
58+
npm run e2e
59+

‎README.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -132,16 +132,18 @@ Rclone provider (optional):
132132
## Testing
133133
We use pytest for unit and API tests.
134134

135-
Pytest is included in requirements.txt; after installing dependencies, run:
135+
Quick start:
136136
```
137137
python3 -m pytest -q
138138
```
139139
Notes:
140140
- If your shell doesn't expose a global `pytest` command (common in fish), using `python3 -m pytest` is the most reliable.
141141
- You can still run `pytest -q` if your PATH includes the virtualenv's bin directory.
142142

143-
Conventions:
144-
- Tests live in tests/ and rely on pytest fixtures in tests/conftest.py (e.g., Flask app and client).
143+
Details:
144+
- Full testing overview: docs/testing.md (tools, fixtures, mocking strategy, env vars, and how to run subsets)
145+
- Pytest configuration lives in pyproject.toml (testpaths=tests, addopts=-q)
146+
- Tests live in tests/ and rely on pytest fixtures in tests/conftest.py (e.g., Flask app and client)
145147
- Add tests alongside new features in future cycles; see dev/cycles.md for cycle protocol.
146148

147149
## Notes

‎docs/testing.md‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
SciDK testing overview
2+
3+
This repository uses pytest for unit, API, and integration-like tests that are hermetic by default (no external services required). The goal is fast feedback with realistic behavior via controlled monkeypatching.
4+
5+
How to run
6+
- Preferred: python3 -m pytest -q
7+
- Virtualenv: If you use .venv, activate it first; pytest is in requirements.txt and [project] dependencies.
8+
- Dev CLI: python3 -m dev.cli test (calls pytest with sensible fallbacks). Some subcommands also run tests as part of DoD checks.
9+
- Pytest config: Defined in pyproject.toml
10+
- testpaths = ["tests"]
11+
- addopts = "-q"
12+
13+
Quickstart: API contracts (phase 00)
14+
- Minimal contracts live under tests/contracts/.
15+
- Example: tests/contracts/test_api_contracts.py::test_providers_contract
16+
- Run: python -m pytest tests/contracts/test_api_contracts.py -q
17+
18+
Quickstart: Playwright E2E smoke (phase 02)
19+
- Requires Node.js. Install Playwright deps once:
20+
- npm install
21+
- npm run e2e:install
22+
- Run smoke locally:
23+
- npm run e2e
24+
- The Playwright config uses e2e/global-setup.ts to spawn the Flask server and exports BASE_URL. See e2e/smoke.spec.ts for the first spec.
25+
26+
Dev CLI flows (validate/context/start)
27+
- Inspect Ready Queue ordering (E2E tasks are top via RICE=999 and DoR):
28+
- python -m dev.cli ready-queue
29+
- Validate Definition of Ready for a task:
30+
- python -m dev.cli validate task:e2e:02-playwright-scaffold
31+
- Print the context for prompting/PR:
32+
- python -m dev.cli context task:e2e:02-playwright-scaffold
33+
- Start the task (creates branch if in git, updates status to In Progress with a timezone-aware ISO8601 timestamp):
34+
- python -m dev.cli start task:e2e:02-playwright-scaffold
35+
36+
Test layout and conventions
37+
- Location: tests/
38+
- Shared fixtures: tests/conftest.py
39+
- app() -> Flask app with TESTING=True, created via scidk.app.create_app
40+
- client(app) -> Flask test client
41+
- sample_py_file/tmp files -> helper fixtures for interpreter tests
42+
- Style: Each test file focuses on a feature area (API endpoints, providers, interpreters, scan/index, graph/neo4j, tasks, UI smoke).
43+
- Naming: test_*.py, functions starting with test_*.
44+
45+
External dependency strategy (mock-first)
46+
Many features integrate with tools/services such as rclone and Neo4j. The test suite isolates these by mocking at process or module boundaries:
47+
- rclone
48+
- Enable provider via env: SCIDK_PROVIDERS=local_fs,mounted_fs,rclone
49+
- Pretend binary exists: monkeypatch shutil.which('rclone') to a fake path
50+
- Simulate commands: monkeypatch subprocess.run to return canned outputs for
51+
- rclone version
52+
- rclone listremotes
53+
- rclone lsjson <target> [--max-depth N | --recursive]
54+
- Tests verify API behavior (providers list, roots, browse) and error messages when rclone is “not installed”. No real rclone needed.
55+
- Neo4j
56+
- Fake driver module by injecting a stub into sys.modules['neo4j'] with GraphDatabase.driver → fake driver/session
57+
- Session.run records Cypher and returns synthetic rows for verification queries
58+
- Tests assert that commit flow fires expected Cypher and that post-commit verification reports counts/flags
59+
- Tests can set env like NEO4J_URI, NEO4J_AUTH=none for the app to attempt a Neo4j path without requiring the real driver
60+
- SQLite and filesystem
61+
- Uses tmp_path for isolated file trees
62+
- Batch inserts and migrations exercised against ephemeral databases; WAL mode is default in app config
63+
64+
What the tests cover (representative)
65+
- API surface: /api/providers, /api/provider_roots, /api/browse, /api/scan, /api/scans/<id>/status|fs|commit, /api/graph/*, files/folders/instances exports, health/metrics
66+
- Providers: local_fs, mounted_fs, rclone (mocked subprocess), rclone scan ingest and recursive hierarchy
67+
- Interpreters: Python, CSV, IPYNB basic parsing and UI rendering
68+
- Graph: in-memory schema endpoints; optional Neo4j schema and commit verification (mocked driver)
69+
- Tasks: background task queue limits, cancel, status
70+
- UI smoke: basic route existence for map/interpreters pages
71+
72+
Environment variables commonly used in tests
73+
- SCIDK_PROVIDERS: Feature-flag providers set (e.g., local_fs,mounted_fs,rclone)
74+
- NEO4J_URI / NEO4J_USER / NEO4J_PASSWORD / NEO4J_AUTH: Used to steer code paths; tests often set NEO4J_AUTH=none with a fake neo4j module
75+
- SCIDK_RCLONE_MOUNTS or SCIDK_FEATURE_RCLONE_MOUNTS: Enables rclone mount manager endpoints (tests mock subprocess)
76+
77+
Running subsets and debugging
78+
- Run a single file: python3 -m pytest tests/test_rclone_provider.py -q
79+
- Run a test node: python3 -m pytest tests/test_neo4j_commit.py::test_standard_scan_and_commit_with_mock_neo4j -q
80+
- Increase verbosity temporarily: add -vv; drop -q if needed
81+
82+
Notes and tips
83+
- The test suite avoids network or real external binaries by default. If you wish to run against real services, do so manually in an isolated environment; this is outside normal CI/local flows.
84+
- Cached artifacts under pytest-of-patch/ are output from past runs and are not part of the active suite.
85+
- If your shell lacks a pytest command, always prefer python3 -m pytest.
86+
87+
Maintenance guidelines
88+
- When adding new features, create tests in tests/ alongside related areas and reuse existing fixtures/mocking patterns
89+
- Prefer monkeypatch at the highest useful boundary (subprocess/module) rather than deep internals to keep tests robust
90+
- Keep tests deterministic and independent; rely on tmp_path and in-memory/synthetic data
91+
92+
UI selectors for E2E
93+
- Stable hooks are provided via data-testid attributes on key elements:
94+
- Header/nav/main in scidk/ui/templates/base.html (e.g., [data-testid="nav-files"]).
95+
- Home page recent scans section in scidk/ui/templates/index.html (data-testid="home-recent-scans").
96+
- Files page root container and title in scidk/ui/templates/datasets.html (data-testid="files-root", "files-title").
97+
- In Playwright, prefer page.getByTestId('nav-files') etc. over brittle CSS paths.
98+
99+
SciDK testing overview
100+
101+
This repository uses pytest for unit, API, and integration-like tests that are hermetic by default (no external services required). The goal is fast feedback with realistic behavior via controlled monkeypatching.
102+
103+
How to run
104+
- Preferred: python3 -m pytest -q
105+
- Virtualenv: If you use .venv, activate it first; pytest is in requirements.txt and [project] dependencies.
106+
- Dev CLI: python3 -m dev.cli test (calls pytest with sensible fallbacks). Some subcommands also run tests as part of DoD checks.
107+
- Pytest config: Defined in pyproject.toml
108+
- testpaths = ["tests"]
109+
- addopts = "-q"
110+
111+
Quickstart: API contracts (phase 00)
112+
- Minimal contracts live under tests/contracts/.
113+
- Example: tests/contracts/test_api_contracts.py::test_providers_contract
114+
- Run: python -m pytest tests/contracts/test_api_contracts.py -q
115+
116+
Quickstart: Playwright E2E smoke (phase 02)
117+
- Requires Node.js. Install Playwright deps once:
118+
- npm install
119+
- npm run e2e:install
120+
- Run smoke locally:
121+
- npm run e2e
122+
- The Playwright config uses e2e/global-setup.ts to spawn the Flask server and exports BASE_URL. See e2e/smoke.spec.ts for the first spec.
123+
124+
Dev CLI flows (validate/context/start)
125+
- Inspect Ready Queue ordering (E2E tasks are top via RICE=999 and DoR):
126+
- python -m dev.cli ready-queue
127+
- Validate Definition of Ready for a task:
128+
- python -m dev.cli validate task:e2e:02-playwright-scaffold
129+
- Print the context for prompting/PR:
130+
- python -m dev.cli context task:e2e:02-playwright-scaffold
131+
- Start the task (creates branch if in git, updates status to In Progress with a timezone-aware ISO8601 timestamp):
132+
- python -m dev.cli start task:e2e:02-playwright-scaffold
133+
134+
Test layout and conventions
135+
- Location: tests/
136+
- Shared fixtures: tests/conftest.py
137+
- app() -> Flask app with TESTING=True, created via scidk.app.create_app
138+
- client(app) -> Flask test client
139+
- sample_py_file/tmp files -> helper fixtures for interpreter tests
140+
- Style: Each test file focuses on a feature area (API endpoints, providers, interpreters, scan/index, graph/neo4j, tasks, UI smoke).
141+
- Naming: test_*.py, functions starting with test_*.
142+
143+
External dependency strategy (mock-first)
144+
Many features integrate with tools/services such as rclone and Neo4j. The test suite isolates these by mocking at process or module boundaries:
145+
- rclone
146+
- Enable provider via env: SCIDK_PROVIDERS=local_fs,mounted_fs,rclone
147+
- Pretend binary exists: monkeypatch shutil.which('rclone') to a fake path
148+
- Simulate commands: monkeypatch subprocess.run to return canned outputs for
149+
- rclone version
150+
- rclone listremotes
151+
- rclone lsjson <target> [--max-depth N | --recursive]
152+
- Tests verify API behavior (providers list, roots, browse) and error messages when rclone is “not installed”. No real rclone needed.
153+
- Neo4j
154+
- Fake driver module by injecting a stub into sys.modules['neo4j'] with GraphDatabase.driver → fake driver/session
155+
- Session.run records Cypher and returns synthetic rows for verification queries
156+
- Tests assert that commit flow fires expected Cypher and that post-commit verification reports counts/flags
157+
- Tests can set env like NEO4J_URI, NEO4J_AUTH=none for the app to attempt a Neo4j path without requiring the real driver
158+
- SQLite and filesystem
159+
- Uses tmp_path for isolated file trees
160+
- Batch inserts and migrations exercised against ephemeral databases; WAL mode is default in app config
161+
162+
What the tests cover (representative)
163+
- API surface: /api/providers, /api/provider_roots, /api/browse, /api/scan, /api/scans/<id>/status|fs|commit, /api/graph/*, files/folders/instances exports, health/metrics
164+
- Providers: local_fs, mounted_fs, rclone (mocked subprocess), rclone scan ingest and recursive hierarchy
165+
- Interpreters: Python, CSV, IPYNB basic parsing and UI rendering
166+
- Graph: in-memory schema endpoints; optional Neo4j schema and commit verification (mocked driver)
167+
- Tasks: background task queue limits, cancel, status
168+
- UI smoke: basic route existence for map/interpreters pages
169+
170+
Environment variables commonly used in tests
171+
- SCIDK_PROVIDERS: Feature-flag providers set (e.g., local_fs,mounted_fs,rclone)
172+
- NEO4J_URI / NEO4J_USER / NEO4J_PASSWORD / NEO4J_AUTH: Used to steer code paths; tests often set NEO4J_AUTH=none with a fake neo4j module
173+
- SCIDK_RCLONE_MOUNTS or SCIDK_FEATURE_RCLONE_MOUNTS: Enables rclone mount manager endpoints (tests mock subprocess)
174+
175+
Running subsets and debugging
176+
- Run a single file: python3 -m pytest tests/test_rclone_provider.py -q
177+
- Run a test node: python3 -m pytest tests/test_neo4j_commit.py::test_standard_scan_and_commit_with_mock_neo4j -q
178+
- Increase verbosity temporarily: add -vv; drop -q if needed
179+
180+
Notes and tips
181+
- The test suite avoids network or real external binaries by default. If you wish to run against real services, do so manually in an isolated environment; this is outside normal CI/local flows.
182+
- Cached artifacts under pytest-of-patch/ are output from past runs and are not part of the active suite.
183+
- If your shell lacks a pytest command, always prefer python3 -m pytest.
184+
185+
Maintenance guidelines
186+
- When adding new features, create tests in tests/ alongside related areas and reuse existing fixtures/mocking patterns
187+
- Prefer monkeypatch at the highest useful boundary (subprocess/module) rather than deep internals to keep tests robust
188+
- Keep tests deterministic and independent; rely on tmp_path and in-memory/synthetic data
189+
190+
UI selectors for E2E
191+
- Stable hooks are provided via data-testid attributes on key elements:
192+
- Header/nav/main in scidk/ui/templates/base.html (e.g., [data-testid="nav-files"]).
193+
- Home page recent scans section in scidk/ui/templates/index.html (data-testid="home-recent-scans").
194+
- Files page root container and title in scidk/ui/templates/datasets.html (data-testid="files-root", "files-title").
195+
- In Playwright, prefer page.getByTestId('nav-files') etc. over brittle CSS paths.
196+
197+
CI (phase 05)
198+
- A GitHub Actions workflow is provided at .github/workflows/ci.yml
199+
- Jobs:
200+
- Python tests (pytest): sets up Python 3.11, installs deps (requirements.txt / pyproject), runs python -m pytest -q
201+
- E2E smoke (Playwright): sets up Node 18, installs deps, installs browsers with npx playwright install --with-deps, runs npm run e2e
202+
- Environment: SCIDK_PROVIDERS=local_fs is used during E2E to avoid external dependencies.
203+
- Continue-on-error: E2E job is marked continue-on-error: true during bring-up; tighten later when stable.
204+
- To run the same locally:
205+
- python -m pytest -q
206+
- npm install && npx playwright install --with-deps && npm run e2e

‎e2e/global-setup.ts‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
import { FullConfig } from '@playwright/test';
2+
import { spawn, ChildProcessWithoutNullStreams } from 'node:child_process';
3+
import http from 'node:http';
4+
5+
function waitForReady(url: string, timeout = 20000): Promise<void> {
6+
const start = Date.now();
7+
return new Promise((resolve, reject) => {
8+
const tick = () => {
9+
http
10+
.get(url, (res) => {
11+
res.resume();
12+
res.destroy();
13+
resolve();
14+
})
15+
.on('error', () => {
16+
if (Date.now() - start > timeout) reject(new Error('Server not ready'));
17+
else setTimeout(tick, 250);
18+
});
19+
};
20+
tick();
21+
});
22+
}
23+
24+
let proc: ChildProcessWithoutNullStreams | null = null;
25+
26+
export default async function globalSetup(config: FullConfig) {
27+
const port = 5010 + Math.floor(Math.random() * 500);
28+
const env = { ...process.env, PORT: String(port), FLASK_ENV: 'development' };
29+
30+
// Prefer running the Flask app directly via Python to avoid Flask CLI dependency
31+
const pyCode = [
32+
"from scidk.app import create_app",
33+
"app=create_app()",
34+
"app.run(host='127.0.0.1', port=int(__import__('os').environ.get('PORT','5000')), use_reloader=False)"
35+
].join('; ');
36+
37+
proc = spawn('python', ['-c', pyCode], {
38+
env,
39+
stdio: ['ignore', 'pipe', 'pipe'],
40+
});
41+
42+
proc.stdout?.on('data', (d) => process.stdout.write(`[server] ${d}`));
43+
proc.stderr?.on('data', (d) => process.stderr.write(`[server] ${d}`));
44+
45+
const baseUrl = `http://127.0.0.1:${port}`;
46+
(process as any).env.BASE_URL = baseUrl;
47+
48+
await waitForReady(baseUrl);
49+
}
50+
51+
export async function teardown() {
52+
if (proc && !proc.killed) {
53+
try { proc.kill(); } catch {}
54+
}
55+
}

‎e2e/playwright.config.ts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
import { defineConfig } from '@playwright/test';
2+
3+
export default defineConfig({
4+
retries: 1,
5+
timeout: 30_000,
6+
use: {
7+
baseURL: process.env.BASE_URL || 'http://127.0.0.1:5000',
8+
trace: 'on-first-retry',
9+
headless: true,
10+
},
11+
globalSetup: require.resolve('./global-setup'),
12+
});

‎e2e/smoke.spec.ts‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
import { test, expect } from '@playwright/test';
2+
3+
// Basic smoke: load home page and ensure no severe console errors
4+
5+
test('home loads without console errors and has stable hooks', async ({ page, baseURL }) => {
6+
const consoleMessages: { type: string; text: string }[] = [];
7+
page.on('console', (msg) => {
8+
const type = msg.type();
9+
const text = msg.text();
10+
consoleMessages.push({ type, text });
11+
});
12+
13+
const url = baseURL || process.env.BASE_URL || 'http://127.0.0.1:5000';
14+
await page.goto(url);
15+
16+
// Basic page sanity
17+
await expect(page).toHaveTitle(/SciDK/i, { timeout: 10_000 });
18+
19+
// Stable selector hooks should exist
20+
await expect(page.getByTestId('nav-files')).toBeVisible();
21+
await expect(page.getByTestId('header')).toBeVisible();
22+
await expect(page.getByTestId('home-recent-scans')).toBeVisible();
23+
24+
// Allow some network/idling time
25+
await page.waitForLoadState('networkidle');
26+
27+
// No error-level logs
28+
const errors = consoleMessages.filter((m) => m.type === 'error');
29+
if (errors.length) {
30+
console.error('Console errors observed:', errors);
31+
}
32+
expect(errors.length).toBe(0);
33+
});

‎node_modules/.bin/playwright‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎node_modules/.bin/playwright-core‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)