Skip to content

Commit cae0df1

Browse files
rodrigoprimorodrigoprimo
committed
Add script to validate code examples in sniff documentation
This commit introduces a new script that verifies code examples in PHPCS sniff documentation. The script checks that "Valid" examples don't trigger the sniff and "Invalid" examples do trigger the sniff, ensuring code examples correctly demonstrate the behavior of the sniff. Noteworthy implementation decisions: - The script will be available in the Composer vendor/bin directory as `phpcs-check-sniff-doc-code-examples`. - To check for syntax errors in the code examples, the code uses `token_get_all()` with the flag TOKEN_PARSE. This means that PHP >= 7.0 is required to run this script while the rest of the repository requires PHP >= 5.4. A requirement check was added to `phpcs-check-sniff-doc-code-examples` to alert users if they are running PHP < 7.0, and a warning was added to the top of the file informing that `phpcs-check-sniff-doc-code-examples` must remain compatible with PHP 5.4 for the requirement check to work. - Since the code uses DOM and libxml, those two PHP extensions were added as requirements to the composer.json file. - The script is unable to differentiate between multiple code examples in the same <code> block. This means that for invalid examples, the script might miss ones that don't trigger the sniff if, in the same <code> block, there is at least one example that does trigger the sniff. - The tests for this script run on a separate GitHub action job as they need to be tested against a different set of PHP versions than the rest of the tests in this repository, and they need to be tested against multiple PHPCS versions and OSes. The DebugSniff tests need to be tested against multiple PHPCS versions but not multiple OSes, while the tests for the other tools need to be tested against multiple OSes but not multiple PHPCS versions. How to use the script: - Run it from the root directory of a PHPCS standard: ``` $ /path/to/script/phpcs-check-sniff-doc-code-examples ``` The script will search for all sniff documentation files and check their code examples. For other ways to run the script check the help page: ``` $ /path/to/script/phpcs-check-sniff-doc-code-examples --help ```
1 parent b59c267 commit cae0df1

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+3224
-7
lines changed

‎.github/workflows/quicktest.yml‎

Lines changed: 89 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -68,10 +68,14 @@ jobs:
6868
if: ${{ matrix.php >= '7.2' }}
6969
run: composer lint
7070

71-
- name: Lint against parse errors (PHP < 7.2)
72-
if: ${{ matrix.php < '7.2' }}
71+
- name: Lint against parse errors (PHP < 7.2 && PHP >= 7.0)
72+
if: ${{ matrix.php < '7.2' && matrix.php >= '7.0' }}
7373
run: composer lintlt72
7474

75+
- name: Lint against parse errors (PHP < 7.0)
76+
if: ${{ matrix.php < '7.0' }}
77+
run: composer lintlt70
78+
7579
# Check that any sniffs available are feature complete.
7680
# This acts as an integration test for the feature completeness script,
7781
# which is why it is run against various PHP versions.
@@ -173,3 +177,86 @@ jobs:
173177
174178
- name: Run the unit tests for the PHPCSDebug sniff
175179
run: composer test-sniff${{ steps.phpunit_script.outputs.SUFFIX }}
180+
181+
# Run CI checks/tests for the DocCodeExamples script as it requires PHP >= 7.0.
182+
quicktest-doc-code-examples:
183+
runs-on: ${{ matrix.os }}
184+
185+
strategy:
186+
matrix:
187+
os: ['ubuntu-latest', 'windows-latest']
188+
php: ['7.0', 'latest']
189+
phpcs_version: ['dev-master']
190+
191+
include:
192+
- os: 'ubuntu-latest'
193+
php: '7.0'
194+
phpcs_version: '3.1.0'
195+
- os: 'windows-latest'
196+
php: '7.0'
197+
phpcs_version: '3.1.0'
198+
199+
name: "QTest: PHP ${{ matrix.php }} - PHPCS ${{ matrix.phpcs_version }} (${{ matrix.os == 'windows-latest' && 'Win' || 'Linux' }})"
200+
201+
steps:
202+
- name: Prepare git to leave line endings alone
203+
run: git config --global core.autocrlf input
204+
205+
- name: Checkout code
206+
uses: actions/checkout@v4
207+
208+
# On stable PHPCS versions, allow for PHP deprecation notices.
209+
# Unit tests don't need to fail on those for stable releases where those issues won't get fixed anymore.
210+
- name: Setup ini config
211+
id: set_ini
212+
shell: bash
213+
run: |
214+
if [ "${{ matrix.phpcs_version }}" != "dev-master" ]; then
215+
echo 'PHP_INI=error_reporting=E_ALL & ~E_DEPRECATED, display_errors=On' >> "$GITHUB_OUTPUT"
216+
else
217+
echo 'PHP_INI=error_reporting=-1, display_errors=On' >> "$GITHUB_OUTPUT"
218+
fi
219+
220+
- name: Install PHP
221+
uses: shivammathur/setup-php@v2
222+
with:
223+
php-version: ${{ matrix.php }}
224+
ini-values: ${{ steps.set_ini.outputs.PHP_INI }}
225+
coverage: none
226+
227+
- name: 'Composer: adjust dependencies'
228+
run: |
229+
# Set the PHPCS version to be used in the tests.
230+
composer require --no-update squizlabs/php_codesniffer:"${{ matrix.phpcs_version }}" --no-scripts --no-interaction
231+
# Remove the PHPCSDevCS dependency as it has different PHPCS requirements and would block installs.
232+
composer remove --no-update --dev phpcsstandards/phpcsdevcs --no-scripts --no-interaction
233+
234+
# Install dependencies and handle caching in one go.
235+
# @link https://github.com/marketplace/actions/install-php-dependencies-with-composer
236+
- name: Install Composer dependencies
237+
uses: "ramsey/composer-install@v3"
238+
with:
239+
# Bust the cache at least once a month - output format: YYYY-MM.
240+
custom-cache-suffix: $(date -u "+%Y-%m")
241+
242+
- name: Grab PHPUnit version
243+
id: phpunit_version
244+
shell: bash
245+
# yamllint disable rule:line-length
246+
run: echo "VERSION=$(vendor/bin/phpunit --version | grep --only-matching --max-count=1 --extended-regexp '\b[0-9]+\.[0-9]+')" >> "$GITHUB_OUTPUT"
247+
# yamllint enable rule:line-length
248+
249+
- name: Determine PHPUnit composer script to use
250+
id: phpunit_script
251+
shell: bash
252+
run: |
253+
if [ "${{ startsWith( steps.phpunit_version.outputs.VERSION, '11.' ) }}" == "true" ]; then
254+
echo 'SUFFIX=' >> "$GITHUB_OUTPUT"
255+
elif [ "${{ startsWith( steps.phpunit_version.outputs.VERSION, '10.' ) }}" == "true" ]; then
256+
echo 'SUFFIX=' >> "$GITHUB_OUTPUT"
257+
else
258+
echo 'SUFFIX=-lte9' >> "$GITHUB_OUTPUT"
259+
fi
260+
261+
- name: Run the unit tests for the DocCodeExamples script
262+
run: composer test-doc-code-examples${{ steps.phpunit_script.outputs.SUFFIX }}

‎.github/workflows/test.yml‎

Lines changed: 196 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -73,9 +73,13 @@ jobs:
7373
if: ${{ matrix.php >= '7.2' }}
7474
run: composer lint -- --checkstyle | cs2pr
7575

76-
- name: Lint against parse errors (PHP < 7.2)
77-
if: ${{ matrix.php < '7.2' }}
78-
run: composer lintlt72 -- --checkstyle | cs2pr
76+
- name: Lint against parse errors (PHP < 7.2 && PHP >= 7.0)
77+
if: ${{ matrix.php < '7.2' && matrix.php >= '7.0' }}
78+
run: composer lintlt72
79+
80+
- name: Lint against parse errors (PHP < 7.0)
81+
if: ${{ matrix.php < '7.0' }}
82+
run: composer lintlt70
7983

8084
# Check that any sniffs available are feature complete.
8185
# This also acts as an integration test for the feature completeness script,
@@ -230,3 +234,192 @@ jobs:
230234
231235
- name: Run the unit tests for the PHPCSDebug sniff
232236
run: composer test-sniff${{ steps.phpunit_script.outputs.SUFFIX }}
237+
238+
# Run CI checks/tests for the DocCodeExamples script as it requires PHP >= 7.0.
239+
test-doc-code-examples:
240+
runs-on: ${{ matrix.os }}
241+
242+
strategy:
243+
matrix:
244+
os: ['ubuntu-latest', 'windows-latest']
245+
# IMPORTANT: test runs shouldn't fail because of PHPCS being incompatible with a PHP version.
246+
# - PHPCS will run without errors on PHP 7.0 - 7.2 on any version.
247+
# - PHP 7.3 needs PHPCS 3.3.1+ to run without errors.
248+
# - PHP 7.4 needs PHPCS 3.5.0+ to run without errors.
249+
# - PHP 8.0 needs PHPCS 3.5.7+ to run without errors.
250+
# - PHP 8.1 needs PHPCS 3.6.1+ to run without errors.
251+
# - PHP 8.2 needs PHPCS 3.6.1+ to run without errors.
252+
# - PHP 8.3 needs PHPCS 3.8.0+ to run without errors (though the errors don't affect this package).
253+
# - PHP 8.4 needs PHPCS 3.8.0+ to run without errors (officially 3.11.0, but 3.8.0 will work fine).
254+
php: ['7.0', '7.1', '7.2']
255+
phpcs_version: ['3.1.0', 'dev-master']
256+
257+
include:
258+
# Complete the matrix, while preventing issues with PHPCS versions incompatible with certain PHP versions.
259+
- os: 'ubuntu-latest'
260+
php: '7.3'
261+
phpcs_version: 'dev-master'
262+
- os: 'windows-latest'
263+
php: '7.3'
264+
phpcs_version: 'dev-master'
265+
- os: 'ubuntu-latest'
266+
php: '7.3'
267+
phpcs_version: '3.3.1'
268+
- os: 'windows-latest'
269+
php: '7.3'
270+
phpcs_version: '3.3.1'
271+
272+
- os: 'ubuntu-latest'
273+
php: '7.4'
274+
phpcs_version: 'dev-master'
275+
- os: 'windows-latest'
276+
php: '7.4'
277+
phpcs_version: 'dev-master'
278+
- os: 'ubuntu-latest'
279+
php: '7.4'
280+
phpcs_version: '3.5.0'
281+
- os: 'windows-latest'
282+
php: '7.4'
283+
phpcs_version: '3.5.0'
284+
285+
- os: 'ubuntu-latest'
286+
php: '8.0'
287+
phpcs_version: 'dev-master'
288+
- os: 'windows-latest'
289+
php: '8.0'
290+
phpcs_version: 'dev-master'
291+
- os: 'ubuntu-latest'
292+
php: '8.0'
293+
phpcs_version: '3.5.7'
294+
- os: 'windows-latest'
295+
php: '8.0'
296+
phpcs_version: '3.5.7'
297+
298+
- os: 'ubuntu-latest'
299+
php: '8.1'
300+
phpcs_version: 'dev-master'
301+
- os: 'windows-latest'
302+
php: '8.1'
303+
phpcs_version: 'dev-master'
304+
- os: 'ubuntu-latest'
305+
php: '8.1'
306+
phpcs_version: '3.6.1'
307+
- os: 'windows-latest'
308+
php: '8.1'
309+
phpcs_version: '3.6.1'
310+
311+
- os: 'ubuntu-latest'
312+
php: '8.2'
313+
phpcs_version: 'dev-master'
314+
- os: 'windows-latest'
315+
php: '8.2'
316+
phpcs_version: 'dev-master'
317+
- os: 'ubuntu-latest'
318+
php: '8.2'
319+
phpcs_version: '3.6.1'
320+
- os: 'windows-latest'
321+
php: '8.2'
322+
phpcs_version: '3.6.1'
323+
324+
- os: 'ubuntu-latest'
325+
php: '8.3'
326+
phpcs_version: 'dev-master'
327+
- os: 'windows-latest'
328+
php: '8.3'
329+
phpcs_version: 'dev-master'
330+
- os: 'ubuntu-latest'
331+
php: '8.3'
332+
phpcs_version: '3.8.0'
333+
- os: 'windows-latest'
334+
php: '8.3'
335+
phpcs_version: '3.8.0'
336+
337+
- os: 'ubuntu-latest'
338+
php: '8.4'
339+
phpcs_version: 'dev-master'
340+
- os: 'windows-latest'
341+
php: '8.4'
342+
phpcs_version: 'dev-master'
343+
- os: 'ubuntu-latest'
344+
php: '8.4'
345+
phpcs_version: '3.8.0'
346+
- os: 'windows-latest'
347+
php: '8.4'
348+
phpcs_version: '3.8.0'
349+
350+
# Experimental builds. These are allowed to fail.
351+
- os: 'ubuntu-latest'
352+
php: '7.4'
353+
phpcs_version: '4.0.x-dev'
354+
355+
- os: 'ubuntu-latest'
356+
php: '8.5' # Nightly.
357+
phpcs_version: 'dev-master'
358+
359+
name: "Test: PHP ${{ matrix.php }} - PHPCS ${{ matrix.phpcs_version }} (${{ matrix.os == 'windows-latest' && 'Win' || 'Linux' }})"
360+
361+
continue-on-error: ${{ matrix.php == '8.5' || matrix.phpcs_version == '4.0.x-dev' }}
362+
363+
steps:
364+
- name: Prepare git to leave line endings alone
365+
run: git config --global core.autocrlf input
366+
367+
- name: Checkout code
368+
uses: actions/checkout@v4
369+
370+
- name: Setup ini config
371+
id: set_ini
372+
shell: bash
373+
run: |
374+
# On stable PHPCS versions, allow for PHP deprecation notices.
375+
# Unit tests don't need to fail on those for stable releases where those issues won't get fixed anymore.
376+
if [[ "${{ matrix.phpcs_version }}" != "dev-master" && "${{ matrix.phpcs_version }}" != "4.0.x-dev" ]]; then
377+
echo 'PHP_INI=error_reporting=E_ALL & ~E_DEPRECATED, display_errors=On' >> "$GITHUB_OUTPUT"
378+
else
379+
echo 'PHP_INI=error_reporting=-1, display_errors=On' >> "$GITHUB_OUTPUT"
380+
fi
381+
382+
- name: Install PHP
383+
uses: shivammathur/setup-php@v2
384+
with:
385+
php-version: ${{ matrix.php }}
386+
ini-values: ${{ steps.set_ini.outputs.PHP_INI }}
387+
coverage: none
388+
389+
- name: 'Composer: adjust dependencies'
390+
run: |
391+
# Set the PHPCS version to be used in the tests.
392+
composer require --no-update squizlabs/php_codesniffer:"${{ matrix.phpcs_version }}" --no-scripts --no-interaction
393+
# Remove the PHPCSDevCS dependency as it has different PHPCS requirements and would block installs.
394+
composer remove --no-update --dev phpcsstandards/phpcsdevcs --no-scripts --no-interaction
395+
396+
# Install dependencies and handle caching in one go.
397+
# @link https://github.com/marketplace/actions/install-php-dependencies-with-composer
398+
- name: Install Composer dependencies
399+
uses: "ramsey/composer-install@v3"
400+
with:
401+
composer-options: ${{ matrix.php == '8.5' && '--ignore-platform-req=php+' || '' }}
402+
# Bust the cache at least once a month - output format: YYYY-MM.
403+
custom-cache-suffix: $(date -u "+%Y-%m")
404+
405+
- name: Grab PHPUnit version
406+
id: phpunit_version
407+
shell: bash
408+
# yamllint disable rule:line-length
409+
run: echo "VERSION=$(vendor/bin/phpunit --version | grep --only-matching --max-count=1 --extended-regexp '\b[0-9]+\.[0-9]+')" >> "$GITHUB_OUTPUT"
410+
# yamllint enable rule:line-length
411+
412+
- name: Determine PHPUnit composer script to use
413+
id: phpunit_script
414+
shell: bash
415+
run: |
416+
if [ "${{ startsWith( steps.phpunit_version.outputs.VERSION, '11.' ) }}" == "true" ]; then
417+
echo 'SUFFIX=' >> "$GITHUB_OUTPUT"
418+
elif [ "${{ startsWith( steps.phpunit_version.outputs.VERSION, '10.' ) }}" == "true" ]; then
419+
echo 'SUFFIX=' >> "$GITHUB_OUTPUT"
420+
else
421+
echo 'SUFFIX=-lte9' >> "$GITHUB_OUTPUT"
422+
fi
423+
424+
- name: Run the unit tests for the DocCodeExamples script
425+
run: composer test-doc-code-examples${{ steps.phpunit_script.outputs.SUFFIX }}

‎README.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ This is a set of tools to assist developers of sniffs for [PHP CodeSniffer][phpc
2727
+ [Stand-alone Installation](#stand-alone-installation)
2828
* [Features](#features)
2929
+ [Checking whether all sniffs in a PHPCS standard are feature complete](#checking-whether-all-sniffs-in-a-phpcs-standard-are-feature-complete)
30+
+ [Checking code examples in sniff documentation are correct](#checking-code-examples-in-sniff-documentation-are-correct)
3031
+ [Sniff Debugging](#sniff-debugging)
3132
+ [Documentation XSD Validation](#documentation-xsd-validation)
3233
* [Contributing](#contributing)
@@ -133,6 +134,44 @@ directories <dir> One or more specific directories to examine.
133134
-V, --version Display the current version of this script.
134135
```
135136

137+
### Checking code examples in sniff documentation are correct
138+
139+
You can verify that the code examples in your sniff documentation XML files correctly demonstrate the behavior of the sniffs. In other words, check that "Valid" examples don't trigger the sniff and "Invalid" examples do trigger the sniff.
140+
141+
Note that while this repository requires PHP >= 5.4, this script requires PHP >= 7.0.
142+
143+
Be aware that this script is unable to differentiate between multiple code examples in the same `<code>` block. This means that for invalid examples, the script might miss ones that don't trigger the sniff if, in the same `<code>` block, there is at least one example that does trigger the sniff.
144+
145+
To use the tool, run it from the root of your standards repo like so:
146+
```bash
147+
# When installed as a project dependency:
148+
vendor/bin/phpcs-check-sniff-doc-code-examples
149+
150+
# When installed globally with Composer:
151+
phpcs-check-sniff-doc-code-examples
152+
153+
# When installed as a git clone or otherwise:
154+
php path/to/PHPCSDevTools/bin/phpcs-check-sniff-doc-code-examples
155+
```
156+
157+
If all is good, the script will exit with code 0 and no output.
158+
159+
If there are issues with the code examples, you will see error messages for each problematic example, like so:
160+
```text
161+
Errors found while processing path/to/project/StandardName/Docs/Category/SniffNameStandard.xml:
162+
- Invalid example #1 does not trigger any errors, but it should.
163+
- Valid example #2 triggers errors, but it shouldn't.
164+
```
165+
166+
#### Options
167+
```text
168+
directories|files One or more specific directories or files to examine.
169+
Defaults to the directory from which the script is run.
170+
--exclude=<dir1,dir2> Comma-delimited list of relative paths of directories to
171+
exclude from the scan.
172+
--ignore-sniffs=<sniff1,sniff2> Comma-delimited list of sniffs to ignore.
173+
--help Print the script help page.
174+
```
136175

137176
### Sniff Debugging
138177

0 commit comments

Comments
 (0)