[pull] dev from ArchiveBox:dev #72
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Fix conftest.py: use subprocess for init, remove unused cli_env fixture - Update all test files to use data_dir parameter instead of env - Remove mock-based TestJSONLOutput class from tests_piping.py - Remove unused imports (MagicMock, patch) - Fix file permissions for cli_utils.py All tests now use real subprocess calls per CLAUDE.md guidelines: - NO MOCKS - tests exercise real code paths - NO SKIPS - every test runs
… search backends This adds new test coverage for previously untested areas: Machine module (archivebox/machine/tests/): - Machine, NetworkInterface, Binary, Process model tests - BinaryMachine and ProcessMachine state machine tests - JSONL serialization/deserialization tests - Manager method tests Workers module (archivebox/workers/tests/): - PID file utility tests (write, read, cleanup) - Orchestrator lifecycle and queue management tests - Worker spawning logic tests - Idle detection and exit condition tests Search backends: - SQLite FTS5 search tests with real indexed content - Phrase search, stemming, and unicode support - Ripgrep search tests with archive directory structure - Environment variable configuration tests Binary provider plugins: - pip provider hook tests - npm provider hook tests with PATH updates - apt provider hook tests
- Remove pid_utils tests (module deleted in dev) - Update orchestrator tests to use Process model for tracking - Add tests for Process.current(), cleanup_stale_running(), terminate() - Add tests for Process hierarchy (parent/child, root, depth) - Add tests for Process.get_running(), get_running_count() - Add tests for ProcessMachine state machine - Update machine model tests to match current API (from_jsonl vs from_json)
- Add real integration tests for SSL, redirects, and SEO plugins using Chrome session helpers for live URL testing - Remove fake "format" tests that just created dicts and asserted on them (apt, pip, npm provider output format tests) - Remove npm integration test that created dirs then checked they existed - Fix SQLite search test to use SQLITEFTS_DB constant instead of hardcoded value
Records hostname → IP resolutions during page load using Chrome CDP. Uses Network.responseReceived events to capture DNS resolution data and writes one JSON line per record to dns.jsonl. Features: - Captures hostname to IP address mappings (A/AAAA records) - Records failed DNS lookups (NXDOMAIN) - Deduplicates resolution records per page load - Integrates with existing Chrome plugin infrastructure
Import shared utilities (getEnv, getEnvBool, getEnvInt) from chrome_utils.js instead of duplicating them locally. Also use DNS_TIMEOUT config for dynamic timeout calculations.
- Add `archivebox persona create/list/update/delete` commands - Support `--import=chrome|firefox|brave` to copy browser profile - Extract cookies via CDP to generate cookies.txt for non-browser tools - Fix JSDoc comment parsing issue in chrome_utils.js
Add shared snapshot hook utilities to chrome_utils.js: - parseArgs(): CLI argument parsing - waitForChromeSession(): Wait for CDP session files - readCdpUrl(): Read CDP WebSocket URL - readTargetId(): Read target page ID - connectToPage(): High-level browser/page connection - waitForPageLoaded(): Wait for navigation completion Refactor ssl, responses, and dns plugins to use shared utilities, eliminating ~100 lines of duplicated code across plugins.
The assertion was checking 'has_seo_data or seo_data' inside an 'if seo_data:' block, making it always truthy. Changed to just check 'has_seo_data' to properly verify that expected SEO keys were extracted. Co-authored-by: Nick Sweeting <[email protected]>
… search backends (#1745)
…order-plugin-dNbxC
Real integration tests using Chrome sessions with example.com: - accessibility: Tests page outline and accessibility tree extraction - parse_dom_outlinks: Tests link extraction and categorization - consolelog: Tests console output capture
- responses: Tests network response capture during page load - staticfile: Tests static file detection and download skip for HTML - env: Tests PATH-based binary discovery (python3, bash)
- merkletree: Tests merkle tree generation with real files, empty directory handling, and disabled mode - custom: Tests custom bash command execution and binary discovery
Add input validation and path safety checks to prevent path traversal attacks in persona name handling: - Add validate_persona_name() to block dangerous characters (/, \, .., etc) - Add ensure_path_within_personas_dir() to verify resolved paths stay within PERSONAS_DIR - Apply validation at persona creation, renaming, and deletion operations Fixes security issues identified by cubic-dev-ai in PR review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]> Co-authored-by: Nick Sweeting <[email protected]>
Refactored 8 plugins to import shared utilities instead of duplicating code locally: - consolelog, redirects: Complete rewrite using shared utils - modalcloser, staticfile: Use readCdpUrl, readTargetId, parseArgs - dom, screenshot, pdf: Remove local parseArgs/getCdpUrl - headers: Import getEnv, getEnvBool, getEnvInt, parseArgs Removes ~380 lines of duplicated boilerplate code.
- test_seo.py: Add assertIsNotNone before conditional to catch SEO extraction failures - test_ssl.py: Add assertIsNotNone to ensure SSL data is captured from HTTPS URLs - test_pip_provider.py: Assert jsonl_found variable to verify binary discovery - dns plugin: Deduplicate NXDOMAIN records using seenResolutions map Tests now fail when functionality doesn't work (no cheating). Co-authored-by: Nick Sweeting <[email protected]>
- Add assertIsNotNone for accessibility_data to ensure test fails if no data generated - Capture and report JSON decode errors in parse_dom_outlinks test - Add assertIsNotNone for outlinks_data with error details - Removes conditional checks that allowed tests to pass without verifying functionality Addresses review comments from cubic-dev-ai Co-authored-by: Nick Sweeting <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )