Skip to content

feat(phpt): add PHP .phpt filter for php-src run-tests.php (-99%)#1503

Open
iliaal wants to merge 1 commit intortk-ai:developfrom
iliaal:phpt-support
Open

feat(phpt): add PHP .phpt filter for php-src run-tests.php (-99%)#1503
iliaal wants to merge 1 commit intortk-ai:developfrom
iliaal:phpt-support

Conversation

@iliaal
Copy link
Copy Markdown

@iliaal iliaal commented Apr 24, 2026

Summary

Adds rtk phpt, a filter for php run-tests.php. It's the driver php-src and PHP extensions use for their .phpt test suite.

A full php-src run emits one status line per test plus a ========DIFF======== block per failure. On a 5322-test run that's about 1.3 MB of output, 99%+ of which is PASS chatter. After filtering:

Scenario Raw Filtered Reduction
php-src full run (4738 PASS, 467 FAIL, 116 SKIP, 1 XFAIL) 1,301,105 B 6,279 B 99.5%
Single passing test ~200 B 2 lines ~85%
Small all-pass dir (11 tests) ~4 KB 2 lines ~97%

Filtered output keeps what the agent needs: aggregate counts, PHP version/SAPI/OS, and up to 20 failures with the first 6 diff lines each.

Example

phpt: 4738 passed, 467 failed, 116 skipped, 1 xfail  (5322 total, 391.4s)
PHP 8.4.20  SAPI cli  OS Linux

FAILURES (467):
  Zend/tests/__debugInfo_reference.phpt -- __debugInfo with reference return
    001- object(Test)#1 (1) {
    002-   ["x"]=>
    003-   int(1)
    004- }
    001+ Fatal error: __debuginfo() must return an array in /home/.../__debugInfo_reference.php on line 12

  Zend/tests/abstract_implicit.phpt -- Abstract methods not allowed in classes (GH-16067)
    001- Fatal error: Class NotAbstract declares abstract method bar() ... in %s on line %d
    001+ Fatal error: Class NotAbstract contains 1 abstract method ... on line 11

... +447 more failures

Usage

cd /path/to/php-src
rtk phpt                           # full suite
rtk phpt Zend/tests/                # directory
rtk phpt Zend/tests/67468.phpt     # single test
rtk phpt -j8 ext/standard/         # args pass through to run-tests.php

Implementation

src/cmds/php/phpt_cmd.rs is a new filter module modeled on python/pytest_cmd.rs: stateful line scan, regex anchors, inline tests.

Status tokens match via (?:^|\])(PASS|FAIL|SKIP|BORK|WARN|LEAK|XFAIL)\s+(.*?)\s*\[([^\[\]]+\.phpt)\]. The (?:^|\]) anchor is what prevents the trailing FAILED TEST SUMMARY block from being re-counted. Its lines are description [path] with no status tag.

ANSI is stripped via the existing utils::strip_ansi. Execution goes through runner::run_filtered with stdout_only().tee("phpt"), same as pytest. The command is added to the is_operational_command whitelist so it flows through the hook pipeline.

Tests

12 inline unit tests:

  • All-pass run
  • Single FAIL with DIFF block
  • Diff truncation (more than 6 lines)
  • Failure-list truncation (more than 20 failures)
  • SKIP and XFAIL don't produce a FAILURES section
  • BORK counts as a failure
  • Empty run (no tests ran)
  • FAILED TEST SUMMARY block doesn't re-count failures
  • ANSI stripping
  • Compression ratio on a synthesized 1000-test pass run (>99%)
  • PHP_OS kernel info trimmed to just the OS family
  • Summary-line counts override scanned counts when capture is truncated

Test plan

  • cargo test -p rtk phpt passes 12/12
  • cargo fmt --all -- --check clean
  • cargo clippy -p rtk no new warnings in diff (pre-existing errors unchanged)
  • Live rtk phpt Zend/tests/67468.phpt on real php-src, correct output
  • Live rtk phpt Zend/tests/ArrayAccess/ (11 tests), correct output
  • Captured 5322-test output run through filter: 99.5% reduction, 467 failures with diffs

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Apr 24, 2026

CLA assistant check
All committers have signed the CLA.

@pszymkowiak pszymkowiak added effort-medium 1-2 jours, quelques fichiers enhancement New feature or request labels Apr 24, 2026
@pszymkowiak
Copy link
Copy Markdown
Collaborator

[w] wshm · Automated triage by AI

📊 Automated PR Analysis

Type feature
🟢 Risk low

Summary

Adds a new rtk phpt filter command for PHP's run-tests.php test runner, reducing output by ~99% by keeping only aggregate counts, environment info, and bounded failure diffs. The implementation follows the existing pytest filter pattern with a stateful line scanner, regex-based parsing, and 12 inline unit tests.

Review Checklist

  • Tests present
  • Breaking change
  • Docs updated

Analyzed automatically by wshm · This is an automated analysis, not a human review.

Wraps `php run-tests.php` and collapses its per-test PASS chatter into
one summary line plus a bounded list of failure diffs. On php-src's
5322-test suite with 467 failures, output drops from 1.3 MB to 6 KB
(99.5% reduction). An all-pass run collapses to two lines.

Handles PASS/FAIL/SKIP/BORK/WARN/LEAK/XFAIL, ANSI color codes, the
========DIFF======== / ========DONE======== blocks, and the trailing
FAILED TEST SUMMARY block (which must not be re-counted).
@iliaal iliaal changed the base branch from master to develop April 24, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort-medium 1-2 jours, quelques fichiers enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants