Skip to content

Commit 80d1314

Browse files
alfsbAndré L F S Baccikamil-tekiela
authored
Argument parsing refactor and urgent mode (#221)
* Coordinate command line on ArgvParser class * Urgent option to focus on avoiding or fixing translations manual builds * Command line option to ignore entity by name Co-authored-by: André L F S Bacci <[email protected]> Co-authored-by: Kamil Tekiela <[email protected]>
1 parent c63809f commit 80d1314

File tree

7 files changed

+266
-109
lines changed

7 files changed

+266
-109
lines changed

‎scripts/translation/README.md

Lines changed: 60 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,69 @@
1+
# Scripts to check consistency of manual translations
12

2-
# Useful scripts for maintaining translation consistency of manual
3+
After a normal `doc-base/configure.php --with-lang=$LANG`, it is possible to
4+
run the command line tools below to check translated source files
5+
for inconsistencies. These tools check for structural differences
6+
that may cause translation build failures or non-validating DocBook XML
7+
results, and fixing these issues will help avoid build failures.
38

4-
Some of these scripts only test some file contents or XML structure
5-
of translated files against their equivalents on `en/` directory.
6-
Others will try to modify the translations in place, changing the
7-
translated files. Use with care.
9+
Some checks are less structural, and as not all translations are identical,
10+
or use the same conventions, they may not be entirely applicable in all
11+
languages. Even two translators working on one language may have different
12+
opinions on how much synchronization is wanted, so not all scripts will be of
13+
use for all translations.
814

9-
Not all translations are identical, or use the same conventions.
10-
Even two translators working on one language may havedifferent
11-
opinions on how much synchronization is wanted. So not all scripts
12-
will be of use for all translations.
15+
Because of the above, it's possible to silence each alert indempendly. These
16+
scripts will output `--add-ignore` commands that, if executed, will omit the
17+
specific alerts in future executions.
1318

14-
Because of aboce, it's possible to silence each alert indempendly.
15-
These scripts will output `--add-ignore` commands that, if executed,
16-
will omit the specific warming in future executions.
19+
## First execution
1720

18-
The `lib/` directory contains common code and functionality
19-
across these scripts.
21+
The first execution of these scripts may generate an inordinate amount of
22+
alerts. It's advised to initially run each command separately, and work the
23+
alerts on a case by case basis. After all interesting cases are fixed,
24+
it's possible to rerun the command and `grep` the output for `--add-ignore`
25+
lines, run these commands, and by so, mass ignore the residual alerts.
26+
27+
## qaxml-attributes.php (structural)
28+
29+
`doc-base/scripts/translation/qaxml-attributes.php` checks if all translated
30+
files have the same tag-attribute-value triplets. Tag's attributes are
31+
extensively utilized in manual for linking and XIncludes. Translated files
32+
with missing or mistyped attributes may cause build failures or missing parts.
33+
34+
This script accepts an `--urgent` option, to filter alerts related to `xml:id`
35+
attributes. This will help translators on languages that are failing to build,
36+
to focus on mismatches that are probably most related with build fails.
37+
38+
## qaxml-entities.php (structural)
39+
40+
`doc-base/scripts/translation/qaxml-entities.php` checks if all translated
41+
files contain the same XML Entities References as the original files.
42+
Unbalanced entities may indicate mistyped or wrongly translated parts. This
43+
is problematic because some of these entities are "file
44+
entities", that is, entities that include entire files and even directories,
45+
so missing or misplaced file entity references almost always cause build
46+
failures.
47+
48+
This script accepts an `--urgent` option, to filter alerts related to file
49+
entities. This will help translators on languages that are failing to build,
50+
to focus on mismatches that are probably most related with build fails.
51+
52+
This script also accepts `-entity` options that will ignore the informed
53+
entities when generating alerts. This is handy in languages that use some
54+
"leaf" entities differently than `doc-en`. For example, `doc-de` uses a lot of
55+
`&zb;` and `&dh;` entities, and could run with `-zb -dh` to avoid generating
56+
alerts for these entities' differences.
2057

21-
Before using the scripts, it need be configured:
58+
## Old tools (below)
59+
60+
The tools on `doc-base/scripts/translation/` are slowly being rewritten. While
61+
this effort is not complete, the previous tools, document below, could be used
62+
to supply for features yet not completed.
63+
64+
---
65+
66+
Before using the old scripts, they need be configured:
2267
```
2368
php doc-base/scripts/translation/configure.php $LANG_DIR
2469
```
@@ -103,12 +148,3 @@ php doc-base/scripts/translation/qaxml.t.php filename
103148
php doc-base/scripts/translation/qaxml.t.php literal
104149
php doc-base/scripts/translation/qaxml.t.php varname
105150
```
106-
107-
## Initial alerts execution
108-
109-
The first execution of these scripts may generate an inordinate amount of
110-
alerts. It's advised to initially run each command separately, and work the
111-
alerts on a case by case basis. After all interesting cases are observed,
112-
it's possible to rerun the command, and `grep` the output for `--add-ignore`
113-
lines, and to mass ignore the residual alerts.
114-

‎scripts/translation/libqa/ArgvParser.php

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
<?php /*
2+
+----------------------------------------------------------------------+
3+
| Copyright (c) 1997-2025 The PHP Group |
4+
+----------------------------------------------------------------------+
5+
| This source file is subject to version 3.01 of the PHP license, |
6+
| that is bundled with this package in the file LICENSE, and is |
7+
| available through the world-wide-web at the following url: |
8+
| https://www.php.net/license/3_01.txt. |
9+
| If you did not receive a copy of the PHP license and are unable to |
10+
| obtain it through the world-wide-web, please send a note to |
11+
| [email protected], so we can mail you a copy immediately. |
12+
+----------------------------------------------------------------------+
13+
| Authors: André L F S Bacci <ae php.net> |
14+
+----------------------------------------------------------------------+
15+
16+
# Description
17+
18+
This class coordinates and centrailzes control for $argv command line
19+
parameters, used between vairous classes. */
20+
21+
class ArgvParser
22+
{
23+
private array $argv;
24+
private array $used;
25+
26+
public function __construct( array $argv )
27+
{
28+
$this->argv = array_values( array_filter( $argv ) );
29+
$this->used = [];
30+
$this->used = array_fill( 0 , count( $argv ) , false );
31+
}
32+
33+
public function use( string $arg ) : void
34+
{
35+
foreach ( $this->argv as $pos => $value )
36+
if ( $arg == $value && $this->used[ $pos ] == false )
37+
{
38+
$this->used[ $pos ] = true;
39+
return;
40+
}
41+
throw new Exception( "Unused '$arg' not found." );
42+
}
43+
44+
public function consume( string $equals = null , string $prefix = null , int $position = -1 ) : string|null
45+
{
46+
$args = $this->argv;
47+
foreach ( $args as $pos => $arg )
48+
{
49+
if ( $arg == null )
50+
continue;
51+
52+
$foundByEquals = $equals != null && $arg == $equals;
53+
$foundByPrefix = $prefix != null && str_starts_with( $arg , $prefix );
54+
$foundByPosition = $position == $pos;
55+
56+
if ( $foundByEquals || $foundByPrefix || $foundByPosition )
57+
{
58+
$this->argv[ $pos ] = null;
59+
$this->used[ $pos ] = true;
60+
61+
return $arg;
62+
}
63+
}
64+
65+
return null;
66+
}
67+
68+
public function complete() : void
69+
{
70+
foreach ( $this->argv as $pos => $arg )
71+
if ( $this->used[ $pos ] == false )
72+
fwrite( STDERR , "Unknown argument: {$arg}\n\n" );
73+
}
74+
75+
public function residual() : array
76+
{
77+
return array_filter( $this->argv );
78+
}
79+
}

‎scripts/translation/libqa/OutputBuffer.php

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -35,10 +35,7 @@ public function __construct( string $header , string $filename , OutputIgnore $i
3535
$this->header = $header . ": " . $filename . "\n\n";
3636
$this->filename = $filename;
3737
$this->ignore = $ignore;
38-
39-
$copy = $ignore->residualArgv;
40-
array_shift( $copy );
41-
$this->options = implode( " " , $copy );
38+
$this->options = implode( " " , $ignore->argv->residual() );
4239
}
4340

4441
public function add( string $text )
@@ -67,7 +64,7 @@ public function addDiff( string $text , int $sourceCount , int $targetCount )
6764

6865
public function addFooter( string $text )
6966
{
70-
// $this->footer[] = $text;
67+
$this->footer[] = $text;
7168
}
7269

7370
public function addLine()
@@ -76,15 +73,24 @@ public function addLine()
7673
$this->add( "\n" );
7774
}
7875

76+
public function contains( string $text ) : bool
77+
{
78+
foreach( $this->matter as $line )
79+
if ( str_contains( $line , $text ) )
80+
return true;
81+
return false;
82+
}
83+
7984
public function print( bool $useAlternatePrinting = false )
8085
{
8186
if ( count( $this->matter ) == 0 && count( $this->footer ) == 0 )
8287
return;
8388

89+
$hashFile = hash( "crc32b" , $this->filename );
8490
$hashHead = $this->hash( false );
8591
$hashFull = $this->hash( true );
8692

87-
if ( $this->ignore->shouldIgnore( $this , $this->filename , $hashHead , $hashFull ) )
93+
if ( $this->ignore->shouldIgnore( $this , $hashFile , $hashHead , $hashFull ) )
8894
return;
8995

9096
print $this->header;
@@ -103,6 +109,8 @@ public function print( bool $useAlternatePrinting = false )
103109

104110
if ( count( $this->footer ) )
105111
print "\n";
112+
113+
print "\n";
106114
}
107115

108116
private function printMatterAlternate() : void

‎scripts/translation/libqa/OutputIgnore.php

Lines changed: 44 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -20,57 +20,53 @@
2020

2121
class OutputIgnore
2222
{
23-
public array $residualArgv;
24-
2523
private bool $appendIgnores = true;
2624
private bool $showIgnore = true;
27-
private string $filename = ".syncxml.ignores";
25+
private string $filename = ".qaxml.ignores";
2826
private string $argv0 = "";
2927

30-
public function __construct( array & $argv )
28+
public ArgvParser $argv;
29+
30+
public function __construct( ArgvParser $argv )
3131
{
32-
$this->argv0 = escapeshellarg( $argv[0] );
32+
$this->argv = $argv;
33+
$this->argv0 = escapeshellarg( $argv->consume( position: 0 ) );
3334

34-
foreach( $argv as $key => $arg )
35-
{
36-
if ( str_starts_with( $arg , "--add-ignore=" ) )
37-
{
38-
$list = $this->loadIgnores();
39-
$line = substr( $arg , 13 );
40-
if ( ! in_array( $line , $list ) )
41-
{
42-
$list[] = $line;
43-
$this->saveIgnores( $list );
44-
}
45-
exit;
46-
}
35+
$arg = $argv->consume( prefix: "--add-ignore=" );
4736

48-
if ( str_starts_with( $arg , "--del-ignore=" ) )
37+
if ( $arg != null )
38+
{
39+
$item = substr( $arg , 13 );
40+
$list = $this->loadIgnores();
41+
if ( ! in_array( $item , $list ) )
4942
{
50-
$list = $this->loadIgnores();
51-
$line = substr( $arg , 13 );
52-
$dels = 0;
53-
while ( in_array( $line , $list ) )
54-
{
55-
$key = array_search( $line , $list );
56-
unset( $list[$key] );
57-
$dels++;
58-
}
59-
if ( $dels == 0 )
60-
print "Ignore mark not found.\n";
61-
else
62-
$this->saveIgnores( $list );
63-
exit;
43+
$list[] = $item;
44+
$this->saveIgnores( $list );
6445
}
46+
exit;
47+
}
6548

66-
if ( $arg == "--disable-ignore" )
49+
$arg = $argv->consume( prefix: "--del-ignore=" );
50+
if ( $arg != null )
51+
{
52+
$item = substr( $arg , 13 );
53+
$list = $this->loadIgnores();
54+
$dels = 0;
55+
while ( in_array( $item , $list ) )
6756
{
68-
$this->showIgnore = false;
69-
unset( $argv[$key] );
57+
$key = array_search( $item , $list );
58+
unset( $list[$key] );
59+
$dels++;
7060
}
61+
if ( $dels == 0 )
62+
print "Ignore mark not found.\n";
63+
else
64+
$this->saveIgnores( $list );
65+
exit;
7166
}
7267

73-
$this->residualArgv = $argv;
68+
if ( $argv->consume( "--disable-ignore" ) != null )
69+
$this->showIgnore = false;
7470
}
7571

7672
private function loadIgnores()
@@ -87,36 +83,28 @@ public function saveIgnores( $data )
8783
file_put_contents( $this->filename , $contents );
8884
}
8985

90-
public function shouldIgnore( OutputBuffer $output , string $filename , string $hashHeader , string $hashMatter )
86+
public function shouldIgnore( OutputBuffer $output , string $hashFile , string $hashHeader , string $hashMatter )
9187
{
9288
$ret = false;
9389

94-
$prefix = "{$filename}:{$hashHeader}:";
95-
$ignore = "{$filename}:{$hashHeader}:{$hashMatter}";
90+
$prefix = "{$hashFile}-{$hashHeader}-";
91+
$active = "{$hashFile}-{$hashHeader}-{$hashMatter}";
9692
$marks = $this->loadIgnores();
9793

9894
// --add-ignore command
9995

100-
if ( in_array( $ignore , $marks ) )
101-
$ret = true; // is already ignored
102-
else //
103-
if ( $this->showIgnore ) // show add command
104-
$output->addFooter( " php {$this->argv0} --add-ignore=$ignore\n" );
105-
106-
// Remove valid ignores, leaves outdated ones for listing
107-
108-
while ( in_array( $ignore , $marks ) )
109-
{
110-
$key = array_search( $ignore , $marks );
111-
unset( $marks[$key] );
112-
}
96+
if ( in_array( $active , $marks ) )
97+
$ret = true;
98+
else
99+
if ( $this->showIgnore )
100+
$output->addFooter( " php {$this->argv0} --add-ignore=$active\n" );
113101

114102
// --del-ignore command
115103

116-
if ( $this->showIgnore ) // show del commands (for this file/prefix)
104+
if ( $this->showIgnore )
117105
foreach ( $marks as $mark )
118-
if ( $mark != null )
119-
if ( str_starts_with( $mark , $prefix ) )
106+
if ( str_starts_with( $mark , $prefix ) )
107+
if ( $mark != $active )
120108
$output->addFooter( " php {$this->argv0} --del-ignore=$mark\n" );
121109

122110
return $ret;

‎scripts/translation/libqa/all.php

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
ini_set( 'display_startup_errors' , 1 );
1919
error_reporting( E_ALL );
2020

21+
require_once __DIR__ . '/ArgvParser.php';
2122
require_once __DIR__ . '/OutputBuffer.php';
2223
require_once __DIR__ . '/OutputIgnore.php';
2324
require_once __DIR__ . '/SyncFileList.php';

0 commit comments

Comments
 (0)