-
Notifications
You must be signed in to change notification settings - Fork 254
Expand file tree
/
Copy pathpcre2test.html
More file actions
2289 lines (2284 loc) · 98.6 KB
/
pcre2test.html
File metadata and controls
2289 lines (2284 loc) · 98.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<html>
<head>
<title>pcre2test specification</title>
</head>
<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB">
<h1>pcre2test man page</h1>
<p>
Return to the <a href="index.html">PCRE2 index page</a>.
</p>
<p>
This page is part of the PCRE2 HTML documentation. It was generated
automatically from the original man page. If there is any nonsense in it,
please consult the man page, in case the conversion went wrong.
<br>
<ul>
<li><a name="TOC1" href="#SEC1">SYNOPSIS</a>
<li><a name="TOC2" href="#SEC2">PCRE2's 8-BIT, 16-BIT AND 32-BIT LIBRARIES</a>
<li><a name="TOC3" href="#SEC3">INPUT ENCODING</a>
<li><a name="TOC4" href="#SEC4">COMMAND LINE OPTIONS</a>
<li><a name="TOC5" href="#SEC5">DESCRIPTION</a>
<li><a name="TOC6" href="#SEC6">COMMAND LINES</a>
<li><a name="TOC7" href="#SEC7">MODIFIER SYNTAX</a>
<li><a name="TOC8" href="#SEC8">PATTERN SYNTAX</a>
<li><a name="TOC9" href="#SEC9">SUBJECT LINE SYNTAX</a>
<li><a name="TOC10" href="#SEC10">PATTERN MODIFIERS</a>
<li><a name="TOC11" href="#SEC11">SUBJECT MODIFIERS</a>
<li><a name="TOC12" href="#SEC12">THE ALTERNATIVE MATCHING FUNCTION</a>
<li><a name="TOC13" href="#SEC13">DEFAULT OUTPUT FROM pcre2test</a>
<li><a name="TOC14" href="#SEC14">OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION</a>
<li><a name="TOC15" href="#SEC15">RESTARTING AFTER A PARTIAL MATCH</a>
<li><a name="TOC16" href="#SEC16">CALLOUTS</a>
<li><a name="TOC17" href="#SEC17">NON-PRINTING CHARACTERS</a>
<li><a name="TOC18" href="#SEC18">SAVING AND RESTORING COMPILED PATTERNS</a>
<li><a name="TOC19" href="#SEC19">SEE ALSO</a>
<li><a name="TOC20" href="#SEC20">AUTHOR</a>
<li><a name="TOC21" href="#SEC21">REVISION</a>
</ul>
<br><a name="SEC1" href="#TOC1">SYNOPSIS</a><br>
<p>
<b>pcre2test [options] [input file [output file]]</b>
<br>
<br>
<b>pcre2test</b> is a test program for the PCRE2 regular expression libraries,
but it can also be used for experimenting with regular expressions. This
document describes the features of the test program; for details of the regular
expressions themselves, see the
<a href="pcre2pattern.html"><b>pcre2pattern</b></a>
documentation. For details of the PCRE2 library function calls and their
options, see the
<a href="pcre2api.html"><b>pcre2api</b></a>
documentation.
</p>
<p>
The input for <b>pcre2test</b> is a sequence of regular expression patterns and
subject strings to be matched. There are also command lines for setting
defaults and controlling some special actions. The output shows the result of
each match attempt. Modifiers on external or internal command lines, the
patterns, and the subject lines specify PCRE2 function options, control how the
subject is processed, and what output is produced.
</p>
<p>
There are many obscure modifiers, some of which are specifically designed for
use in conjunction with the test script and data files that are distributed as
part of PCRE2. All the modifiers are documented here, some without much
justification, but many of them are unlikely to be of use except when testing
the libraries.
</p>
<br><a name="SEC2" href="#TOC1">PCRE2's 8-BIT, 16-BIT AND 32-BIT LIBRARIES</a><br>
<p>
Different versions of the PCRE2 library can be built to support character
strings that are encoded in 8-bit, 16-bit, or 32-bit code units. One, two, or
all three of these libraries may be simultaneously installed. The
<b>pcre2test</b> program can be used to test all the libraries. However, its own
input and output are always in 8-bit format. When testing the 16-bit or 32-bit
libraries, patterns and subject strings are converted to 16-bit or 32-bit
format before being passed to the library functions. Results are converted back
to 8-bit code units for output.
</p>
<p>
In the rest of this document, the names of library functions and structures
are given in generic form, for example, <b>pcre2_compile()</b>. The actual
names used in the libraries have a suffix _8, _16, or _32, as appropriate.
<a name="inputencoding"></a></p>
<br><a name="SEC3" href="#TOC1">INPUT ENCODING</a><br>
<p>
Input to <b>pcre2test</b> is processed line by line, either by calling the C
library's <b>fgets()</b> function, or via the <b>libreadline</b> or <b>libedit</b>
library. In some Windows environments character 26 (hex 1A) causes an immediate
end of file, and no further data is read, so this character should be avoided
unless you really want that action.
</p>
<p>
The input is processed using C's string functions, so must not contain binary
zeros, even though in Unix-like environments, <b>fgets()</b> treats any bytes
other than newline as data characters. An error is generated if a binary zero
is encountered. By default subject lines are processed for backslash escapes,
which makes it possible to include any data value in strings that are passed to
the library for matching. For patterns, there is a facility for specifying some
or all of the 8-bit input characters as hexadecimal pairs, which makes it
possible to include binary zeros.
</p>
<br><b>
Input for the 16-bit and 32-bit libraries
</b><br>
<p>
When testing the 16-bit or 32-bit libraries, there is a need to be able to
generate character code points greater than 255 in the strings that are passed
to the library. For subject lines and some patterns, backslash escapes can be
used. In addition, when the <b>utf</b> modifier (see
<a href="#optionmodifiers">"Setting compilation options"</a>
below) is set, the pattern and any following subject lines are interpreted as
UTF-8 strings and translated to UTF-16 or UTF-32 as appropriate.
</p>
<p>
For non-UTF testing of wide characters, the <b>utf8_input</b> modifier can be
used. This is mutually exclusive with <b>utf</b>, and is allowed only in 16-bit
or 32-bit mode. It causes the pattern and following subject lines to be treated
as UTF-8 according to the original definition (RFC 2279), which allows for
character values up to 0x7fffffff. Each character is placed in one 16-bit or
32-bit code unit (in the 16-bit case, values greater than 0xffff cause an error
to occur).
</p>
<p>
UTF-8 (in its original definition) is not capable of encoding values greater
than 0x7fffffff, but such values can be handled by the 32-bit library. When
testing this library in non-UTF mode with <b>utf8_input</b> set, if any
character is preceded by the byte 0xff (which is an invalid byte in UTF-8)
0x80000000 is added to the character's value. For subject strings, using an
escape sequence is preferable.
</p>
<br><a name="SEC4" href="#TOC1">COMMAND LINE OPTIONS</a><br>
<p>
<b>-8</b>
If the 8-bit library has been built, this option causes it to be used (this is
the default). If the 8-bit library has not been built, this option causes an
error.
</p>
<p>
<b>-16</b>
If the 16-bit library has been built, this option causes it to be used. If the
8-bit library has not been built, this is the default. If the 16-bit library
has not been built, this option causes an error.
</p>
<p>
<b>-32</b>
If the 32-bit library has been built, this option causes it to be used. If no
other library has been built, this is the default. If the 32-bit library has
not been built, this option causes an error.
</p>
<p>
<b>-ac</b>
Behave as if each pattern has the <b>auto_callout</b> modifier, that is, insert
automatic callouts into every pattern that is compiled.
</p>
<p>
<b>-AC</b>
As for <b>-ac</b>, but in addition behave as if each subject line has the
<b>callout_extra</b> modifier, that is, show additional information from
callouts.
</p>
<p>
<b>-b</b>
Behave as if each pattern has the <b>fullbincode</b> modifier; the full
internal binary form of the pattern is output after compilation.
</p>
<p>
<b>-C</b>
Output the version number of the PCRE2 library, and all available information
about the optional features that are included, and then exit with zero exit
code. All other options are ignored. If both -C and -LM are present, whichever
is first is recognized.
</p>
<p>
<b>-C</b> <i>option</i>
Output information about a specific build-time option, then exit. This
functionality is intended for use in scripts such as <b>RunTest</b>. The
following options output the value and set the exit code as indicated:
<pre>
linksize the configured internal link size (2, 3, or 4)
exit code is set to the link size
newline the default newline setting:
CR, LF, CRLF, ANYCRLF, ANY, or NUL
exit code is always 0
bsr the default setting for what \R matches:
ANYCRLF or ANY
exit code is always 0
</pre>
The following options output 1 for true or 0 for false, and set the exit code
to the same value:
<pre>
backslash-C \C is supported (not locked out)
ebcdic compiled for an EBCDIC environment
ebcdic-io if PCRE2 is compiled for EBCDIC, whether pcre2test's input and
output is EBCDIC or ASCII
ebcdic-nl25 if PCRE2 is compiled for EBCDIC, whether NL (= LF) is 0x25
(otherwise it is 0x15, the default)
jit just-in-time support is available
pcre2-16 the 16-bit library was built
pcre2-32 the 32-bit library was built
pcre2-8 the 8-bit library was built
unicode Unicode support is available
</pre>
Note that the availability of JIT support in the library does not guarantee
that it can actually be used because in some environments it is unable to
allocate executable memory. The option "jitusable" gives more detailed
information. It returns one of the following values:
<pre>
0 JIT is available and usable
1 JIT is available but cannot allocate executable memory
2 JIT is not available
3 Unexpected return from test call to <b>pcre2_jit_compile()</b>
</pre>
If an unknown option is given, an error message is output; the exit code is 0.
</p>
<p>
<b>-d</b>
Behave as if each pattern has the <b>debug</b> modifier; the internal
form and information about the compiled pattern is output after compilation;
<b>-d</b> is equivalent to <b>-b -i</b>.
</p>
<p>
<b>-dfa</b>
Behave as if each subject line has the <b>dfa</b> modifier; matching is done
using the <b>pcre2_dfa_match()</b> function instead of the default
<b>pcre2_match()</b>.
</p>
<p>
<b>-E</b>
Run in "preprocess only" mode (similar to "gcc -E"). The "#if ... #endif"
commands are processed, and all other lines are printed verbatim.
</p>
<p>
<b>-error</b> <i>number[,number,...]</i>
Call <b>pcre2_get_error_message()</b> for each of the error numbers in the
comma-separated list, display the resulting messages on the standard output,
then exit with zero exit code. The numbers may be positive or negative. This is
a convenience facility for PCRE2 maintainers.
</p>
<p>
<b>-help</b>
Output a brief summary these options and then exit.
</p>
<p>
<b>-i</b>
Behave as if each pattern has the <b>info</b> modifier; information about the
compiled pattern is given after compilation.
</p>
<p>
<b>-jit</b>
Behave as if each pattern line has the <b>jit</b> modifier; after successful
compilation, each pattern is passed to the just-in-time compiler, if available.
</p>
<p>
<b>-jitfast</b>
Behave as if each pattern line has the <b>jitfast</b> modifier; after
successful compilation, each pattern is passed to the just-in-time compiler, if
available, and each subject line is passed directly to the JIT matcher via its
"fast path".
</p>
<p>
<b>-jitverify</b>
Behave as if each pattern line has the <b>jitverify</b> modifier; after
successful compilation, each pattern is passed to the just-in-time compiler, if
available, and the use of JIT for matching is verified.
</p>
<p>
<b>-LM</b>
List modifiers: write a list of available pattern and subject modifiers to the
standard output, then exit with zero exit code. All other options are ignored.
If both -C and any -Lx options are present, whichever is first is recognized.
</p>
<p>
<b>-LP</b>
List properties: write a list of recognized Unicode properties to the standard
output, then exit with zero exit code. All other options are ignored. If both
-C and any -Lx options are present, whichever is first is recognized.
</p>
<p>
<b>-LS</b>
List scripts: write a list of recognized Unicode script names to the standard
output, then exit with zero exit code. All other options are ignored. If both
-C and any -Lx options are present, whichever is first is recognized.
</p>
<p>
<b>-pattern</b> <i>modifier-list</i>
Behave as if each pattern line contains the given modifiers.
</p>
<p>
<b>-q</b>
Do not output the version number of <b>pcre2test</b> at the start of execution.
</p>
<p>
<b>-S</b> <i>size</i>
On Unix-like systems, set the size of the run-time stack to <i>size</i>
mebibytes (units of 1024*1024 bytes).
</p>
<p>
<b>-subject</b> <i>modifier-list</i>
Behave as if each subject line contains the given modifiers.
</p>
<p>
<b>-t</b>
Run each compile and match many times with a timer, and output the resulting
times per compile or match. When JIT is used, separate times are given for the
initial compile and the JIT compile. You can control the number of iterations
that are used for timing by following <b>-t</b> with a number (as a separate
item on the command line). For example, "-t 1000" iterates 1000 times. The
default is to iterate 500,000 times.
</p>
<p>
<b>-tm</b>
This is like <b>-t</b> except that it times only the matching phase, not the
compile phase.
</p>
<p>
<b>-T</b> <b>-TM</b>
These behave like <b>-t</b> and <b>-tm</b>, but in addition, at the end of a run,
the total times for all compiles and matches are output.
</p>
<p>
<b>-version</b>
Output the PCRE2 version number and then exit.
</p>
<br><a name="SEC5" href="#TOC1">DESCRIPTION</a><br>
<p>
If <b>pcre2test</b> is given two filename arguments, it reads from the first and
writes to the second. If the first name is "-", input is taken from the
standard input. If <b>pcre2test</b> is given only one argument, it reads from
that file and writes to stdout. Otherwise, it reads from stdin and writes to
stdout.
</p>
<p>
When <b>pcre2test</b> is built, a configuration option can specify that it
should be linked with the <b>libreadline</b> or <b>libedit</b> library. When this
is done, if the input is from a terminal, it is read using the <b>readline()</b>
function. This provides line-editing and history facilities. The output from
the <b>-help</b> option states whether or not <b>readline()</b> will be used.
</p>
<p>
The program handles any number of tests, each of which consists of a set of
input lines. Each set starts with a regular expression pattern, followed by any
number of subject lines to be matched against that pattern. In between sets of
test data, command lines that begin with # may appear. This file format, with
some restrictions, can also be processed by the <b>perltest.sh</b> script that
is distributed with PCRE2 as a means of checking that the behaviour of PCRE2
and Perl is the same. For a specification of <b>perltest.sh</b>, see the
comments near its beginning. See also the #perltest command below.
</p>
<p>
When the input is a terminal, <b>pcre2test</b> prompts for each line of input,
using "re>" to prompt for regular expression patterns, and "data>" to prompt
for subject lines. Command lines starting with # can be entered only in
response to the "re>" prompt.
</p>
<p>
Each subject line is matched separately and independently. If you want to do
multi-line matches, you have to use the \n escape sequence (or \r or \r\n,
etc., depending on the newline setting) in a single line of input to encode the
newline sequences. There is no limit on the length of subject lines; the input
buffer is automatically extended if it is too small. There are replication
features that makes it possible to generate long repetitive pattern or subject
lines without having to supply them explicitly.
</p>
<p>
An empty line or the end of the file signals the end of the subject lines for a
test, at which point a new pattern or command line is expected if there is
still input to be read.
</p>
<br><a name="SEC6" href="#TOC1">COMMAND LINES</a><br>
<p>
In between sets of test data, a line that begins with # is interpreted as a
command line. If the first character is followed by white space or an
exclamation mark, the line is treated as a comment, and ignored. Otherwise, the
following commands are recognized:
<pre>
#forbid_utf
</pre>
Subsequent patterns automatically have the PCRE2_NEVER_UTF and PCRE2_NEVER_UCP
options set, which locks out the use of the PCRE2_UTF and PCRE2_UCP options and
the use of (*UTF) and (*UCP) at the start of patterns. This command also forces
an error if a subsequent pattern contains any occurrences of \P, \p, or \X,
which are still supported when PCRE2_UTF is not set, but which require Unicode
property support to be included in the library.
</p>
<p>
This is a trigger guard that is used in test files to ensure that UTF or
Unicode property tests are not accidentally added to files that are used when
Unicode support is not included in the library. Setting PCRE2_NEVER_UTF and
PCRE2_NEVER_UCP as a default can also be obtained by the use of <b>#pattern</b>;
the difference is that <b>#forbid_utf</b> cannot be unset, and the automatic
options are not displayed in pattern information, to avoid cluttering up test
output.
<pre>
#load <filename>
</pre>
This command is used to load a set of precompiled patterns from a file, as
described in the section entitled "Saving and restoring compiled patterns"
<a href="#saverestore">below.</a>
<pre>
#loadtables <filename>
</pre>
This command is used to load a set of binary character tables that can be
accessed by the tables=3 qualifier. Such tables can be created by the
<b>pcre2_dftables</b> program with the -b option.
<pre>
#newline_default [<newline-list>]
</pre>
When PCRE2 is built, a default newline convention can be specified. This
determines which characters and/or character pairs are recognized as indicating
a newline in a pattern or subject string. The default can be overridden when a
pattern is compiled. The standard test files contain tests of various newline
conventions, but the majority of the tests expect a single linefeed to be
recognized as a newline by default. Without special action the tests would fail
when PCRE2 is compiled with either CR or CRLF as the default newline.
</p>
<p>
The #newline_default command specifies a list of newline types that are
acceptable as the default. The types must be one of CR, LF, CRLF, ANYCRLF,
ANY, or NUL (in upper or lower case), for example:
<pre>
#newline_default LF Any anyCRLF
</pre>
If the default newline is in the list, this command has no effect. Otherwise,
except when testing the POSIX API, a <b>newline</b> modifier that specifies the
first newline convention in the list (LF in the above example) is added to any
pattern that does not already have a <b>newline</b> modifier. If the newline
list is empty, the feature is turned off. This command is present in a number
of the standard test input files.
</p>
<p>
When the POSIX API is being tested there is no way to override the default
newline convention, though it is possible to set the newline convention from
within the pattern. A warning is given if the <b>posix</b> or <b>posix_nosub</b>
modifier is used when <b>#newline_default</b> would set a default for the
non-POSIX API.
<pre>
#pattern <modifier-list>
</pre>
This command sets a default modifier list that applies to all subsequent
patterns. Modifiers on a pattern can change these settings.
<pre>
#perltest
</pre>
This line is used in test files that can also be processed by <b>perltest.sh</b>
to confirm that Perl gives the same results as PCRE2. Subsequent tests are
checked for the use of <b>pcre2test</b> features that are incompatible with the
<b>perltest.sh</b> script.
</p>
<p>
Patterns must use '/' as their delimiter, and only certain modifiers are
supported. Comment lines, #pattern commands, and #subject commands that set or
unset "mark" are recognized and acted on. The #perltest, #forbid_utf, and
#newline_default commands, which are needed in the relevant pcre2test files,
are silently ignored. All other command lines are ignored, but give a warning
message. The <b>#perltest</b> command helps detect tests that are accidentally
put in the wrong file or use the wrong delimiter. For more details of the
<b>perltest.sh</b> script see the comments it contains.
<pre>
#pop [<modifiers>]
#popcopy [<modifiers>]
</pre>
These commands are used to manipulate the stack of compiled patterns, as
described in the section entitled "Saving and restoring compiled patterns"
<a href="#saverestore">below.</a>
<pre>
#save <filename>
</pre>
This command is used to save a set of compiled patterns to a file, as described
in the section entitled "Saving and restoring compiled patterns"
<a href="#saverestore">below.</a>
<pre>
#subject <modifier-list>
</pre>
This command sets a default modifier list that applies to all subsequent
subject lines. Modifiers on a subject line can change these settings.
<pre>
#if CONDITION
...
#endif
</pre>
If CONDITION is true, then the command is printed, and its contents are
processed as normal, including printing the commandlines to the output. If
CONDITION is false, then all lines between the "#if" and "#endif" are skipped
and not printed. The CONDITION can be any of the conditions which are tested by
the "-C" commandline option and which set pcre2test's exit code to a boolean
value. The CONDITION may also be preceded by "!".
</p>
<br><a name="SEC7" href="#TOC1">MODIFIER SYNTAX</a><br>
<p>
Modifier lists are used with both pattern and subject lines. Items in a list
are separated by commas followed by optional white space. Trailing white space
in a modifier list is ignored. Some modifiers may be given for both patterns
and subject lines, whereas others are valid only for one or the other. Each
modifier has a long name, for example "anchored", and some of them must be
followed by an equals sign and a value, for example, "offset=12". Values cannot
contain comma characters, but may contain spaces. Modifiers that do not take
values may be preceded by a minus sign to turn off a previous setting.
</p>
<p>
A few of the more common modifiers can also be specified as single letters, for
example "i" for "caseless". In documentation, following the Perl convention,
these are written with a slash ("the /i modifier") for clarity. Abbreviated
modifiers must all be concatenated in the first item of a modifier list. If the
first item is not recognized as a long modifier name, it is interpreted as a
sequence of these abbreviations. For example:
<pre>
/abc/ig,newline=cr,jit=3
</pre>
This is a pattern line whose modifier list starts with two one-letter modifiers
(/i and /g). The lower-case abbreviated modifiers are the same as used in Perl.
</p>
<br><a name="SEC8" href="#TOC1">PATTERN SYNTAX</a><br>
<p>
A pattern line must start with one of the following characters (common symbols,
excluding pattern meta-characters):
<pre>
/ ! " ' ` - = _ : ; , % & @ ~
</pre>
This is interpreted as the pattern's delimiter. A regular expression may be
continued over several input lines, in which case the newline characters are
included within it. It is possible to include the delimiter as a literal within
the pattern by escaping it with a backslash, for example
<pre>
/abc\/def/
</pre>
If you do this, the escape and the delimiter form part of the pattern, but
since the delimiters are all non-alphanumeric, the inclusion of the backslash
does not affect the pattern's interpretation. Note, however, that this trick
does not work within \Q...\E literal bracketing because the backslash will
itself be interpreted as a literal. If the terminating delimiter is immediately
followed by a backslash, for example,
<pre>
/abc/\
</pre>
a backslash is added to the end of the pattern. This is done to provide a way
of testing the error condition that arises if a pattern finishes with a
backslash, because
<pre>
/abc\/
</pre>
is interpreted as the first line of a pattern that starts with "abc/", causing
pcre2test to read the next line as a continuation of the regular expression.
</p>
<p>
A pattern can be followed by a modifier list (details below).
</p>
<br><a name="SEC9" href="#TOC1">SUBJECT LINE SYNTAX</a><br>
<p>
Before each subject line is passed to <b>pcre2_match()</b>,
<b>pcre2_dfa_match()</b>, or <b>pcre2_jit_match()</b>, leading and trailing white
space is removed, and the line is scanned for backslash escapes, unless the
<b>subject_literal</b> modifier was set for the pattern. The following provide a
means of encoding non-printing characters in a visible way:
<pre>
\a alarm (BEL, \x07)
\b backspace (\x08)
\e escape (\x27)
\f form feed (\x0c)
\n newline (\x0a)
\N{U+hh...} unicode character (any number of hex digits)
\r carriage return (\x0d)
\t tab (\x09)
\v vertical tab (\x0b)
\ddd octal number (up to 3 octal digits); represent a single
code point unless larger than 255 with the 8-bit library
\o{dd...} octal number (any number of octal digits} representing a
character in UTF mode or a code point
\xhh hexadecimal byte (up to 2 hex digits)
\x{hh...} hexadecimal number (up to 8 hex digits) representing a
character in UTF mode or a code point
</pre>
Invoking \N{U+hh...} or \x{hh...} doesn't require the use of the <b>utf</b>
modifier on the pattern. It is always recognized. There may be any number of
hexadecimal digits inside the braces; invalid values provoke error messages
but when using \N{U+hh...} with some invalid unicode characters they will
be accepted with a warning instead.
</p>
<p>
Note that even in UTF-8 mode, \xhh (and depending of how large, \ddd)
describe one byte rather than one character; this makes it possible to
construct invalid UTF-8 sequences for testing purposes. On the other hand,
\x{hh...} is interpreted as a UTF-8 character in UTF-8 mode, only generating
more than one byte if the value is greater than 127. To avoid the ambiguity
it is preferred to use \N{U+hh...} when describing characters. When testing
the 8-bit library not in UTF-8 mode, \x{hh} generates one byte for values
that could fit on it, and causes an error for greater values.
</p>
<p>
When testing the 16-bit library, not in UTF-16 mode, all 4-digit \x{hhhh}
values are accepted. This makes it possible to construct invalid UTF-16
sequences for testing purposes.
</p>
<p>
When testing the 32-bit library, not in UTF-32 mode, all 4 to 8-digit \x{...}
values are accepted. This makes it possible to construct invalid UTF-32
sequences for testing purposes.
</p>
<p>
There is a special backslash sequence that specifies replication of one or more
characters:
<pre>
\[<characters>]{<count>}
</pre>
This makes it possible to test long strings without having to provide them as
part of the file. For example:
<pre>
\[abc]{4}
</pre>
is converted to "abcabcabcabc". This feature does not support nesting. To
include a closing square bracket in the characters, code it as \x5D.
</p>
<p>
A backslash followed by an equals sign marks the end of the subject string and
the start of a modifier list. For example:
<pre>
abc\=notbol,notempty
</pre>
If the subject string is empty and \= is followed by white space, the line is
treated as a comment line, and is not used for matching. For example:
<pre>
\= This is a comment.
abc\= This is an invalid modifier list.
</pre>
A backslash followed by any other non-alphanumeric character just escapes that
character. A backslash followed by anything else causes an error. However, if
the very last character in the line is a backslash (and there is no modifier
list), it is ignored. This gives a way of passing an empty line as data, since
a real empty line terminates the data input.
</p>
<p>
If the <b>subject_literal</b> modifier is set for a pattern, all subject lines
that follow are treated as literals, with no special treatment of backslashes.
No replication is possible, and any subject modifiers must be set as defaults
by a <b>#subject</b> command.
</p>
<br><a name="SEC10" href="#TOC1">PATTERN MODIFIERS</a><br>
<p>
There are several types of modifier that can appear in pattern lines. Except
where noted below, they may also be used in <b>#pattern</b> commands. A
pattern's modifier list can add to or override default modifiers that were set
by a previous <b>#pattern</b> command.
<a name="optionmodifiers"></a></p>
<br><b>
Setting compilation options
</b><br>
<p>
The following modifiers set options for <b>pcre2_compile()</b>. Most of them set
bits in the options argument of that function, but those whose names start with
PCRE2_EXTRA are additional options that are set in the compile context.
Some of these options have single-letter abbreviations. There is special
handling for /x: if a second x is present, PCRE2_EXTENDED is converted into
PCRE2_EXTENDED_MORE as in Perl. A third appearance adds PCRE2_EXTENDED as well,
though this makes no difference to the way <b>pcre2_compile()</b> behaves. See
<a href="pcre2api.html"><b>pcre2api</b></a>
for a description of the effects of these options.
<pre>
allow_empty_class set PCRE2_ALLOW_EMPTY_CLASS
allow_lookaround_bsk set PCRE2_EXTRA_ALLOW_LOOKAROUND_BSK
allow_surrogate_escapes set PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES
alt_bsux set PCRE2_ALT_BSUX
alt_circumflex set PCRE2_ALT_CIRCUMFLEX
alt_extended_class set PCRE2_ALT_EXTENDED_CLASS
alt_verbnames set PCRE2_ALT_VERBNAMES
anchored set PCRE2_ANCHORED
/a ascii_all set all ASCII options
ascii_bsd set PCRE2_EXTRA_ASCII_BSD
ascii_bss set PCRE2_EXTRA_ASCII_BSS
ascii_bsw set PCRE2_EXTRA_ASCII_BSW
ascii_digit set PCRE2_EXTRA_ASCII_DIGIT
ascii_posix set PCRE2_EXTRA_ASCII_POSIX
auto_callout set PCRE2_AUTO_CALLOUT
bad_escape_is_literal set PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL
/i caseless set PCRE2_CASELESS
/r caseless_restrict set PCRE2_EXTRA_CASELESS_RESTRICT
dollar_endonly set PCRE2_DOLLAR_ENDONLY
/s dotall set PCRE2_DOTALL
dupnames set PCRE2_DUPNAMES
endanchored set PCRE2_ENDANCHORED
escaped_cr_is_lf set PCRE2_EXTRA_ESCAPED_CR_IS_LF
/x extended set PCRE2_EXTENDED
/xx extended_more set PCRE2_EXTENDED_MORE
extra_alt_bsux set PCRE2_EXTRA_ALT_BSUX
firstline set PCRE2_FIRSTLINE
literal set PCRE2_LITERAL
match_line set PCRE2_EXTRA_MATCH_LINE
match_invalid_utf set PCRE2_MATCH_INVALID_UTF
match_unset_backref set PCRE2_MATCH_UNSET_BACKREF
match_word set PCRE2_EXTRA_MATCH_WORD
/m multiline set PCRE2_MULTILINE
never_backslash_c set PCRE2_NEVER_BACKSLASH_C
never_callout set PCRE2_EXTRA_NEVER_CALLOUT
never_ucp set PCRE2_NEVER_UCP
never_utf set PCRE2_NEVER_UTF
/n no_auto_capture set PCRE2_NO_AUTO_CAPTURE
no_auto_possess set PCRE2_NO_AUTO_POSSESS
no_bs0 set PCRE2_EXTRA_NO_BS0
no_dotstar_anchor set PCRE2_NO_DOTSTAR_ANCHOR
no_start_optimize set PCRE2_NO_START_OPTIMIZE
no_utf_check set PCRE2_NO_UTF_CHECK
python_octal set PCRE2_EXTRA_PYTHON_OCTAL
turkish_casing set PCRE2_EXTRA_TURKISH_CASING
ucp set PCRE2_UCP
ungreedy set PCRE2_UNGREEDY
use_offset_limit set PCRE2_USE_OFFSET_LIMIT
utf set PCRE2_UTF
</pre>
As well as turning on the PCRE2_UTF option, the <b>utf</b> modifier causes all
non-printing characters in output strings to be printed using the \x{hh...}
notation. Otherwise, those less than 0x100 are output in hex without the curly
brackets. Setting <b>utf</b> in 16-bit or 32-bit mode also causes pattern and
subject strings to be translated to UTF-16 or UTF-32, respectively, before
being passed to library functions.
<br>
<br>
The following modifiers enable or disable performance optimizations by
calling <b>pcre2_set_optimize()</b> before invoking the regex compiler.
<pre>
optimization_full enable all optional optimizations
optimization_none disable all optional optimizations
auto_possess auto-possessify variable quantifiers
auto_possess_off don't auto-possessify variable quantifiers
dotstar_anchor anchor patterns starting with .*
dotstar_anchor_off don't anchor patterns starting with .*
start_optimize enable pre-scan of subject string
start_optimize_off disable pre-scan of subject string
</pre>
See the
<a href="pcre2_set_optimize.html"><b>pcre2_set_optimize</b></a>
documentation for details on these optimizations.
<a name="controlmodifiers"></a></p>
<br><b>
Setting compilation controls
</b><br>
<p>
The following modifiers affect the compilation process or request information
about the pattern. There are single-letter abbreviations for some that are
heavily used in the test files.
<pre>
/B bincode show binary code without lengths
bsr=[anycrlf|unicode] specify \R handling
callout_info show callout information
convert=<options> request foreign pattern conversion
convert_glob_escape=c set glob escape character
convert_glob_separator=c set glob separator character
convert_length set convert buffer length
debug same as info,fullbincode
expand expand repetition syntax in pattern
framesize show matching frame size
fullbincode show binary code with lengths
/I info show info about compiled pattern
hex unquoted characters are hexadecimal
jit[=<number>] use JIT
jitfast use JIT fast path
jitverify verify JIT use
locale=<name> use this locale
max_pattern_compiled ) set maximum compiled pattern
_length=<n> ) length (bytes)
max_pattern_length=<n> set maximum pattern length (code units)
max_varlookbehind=<n> set maximum variable lookbehind length
memory show memory used
newline=<type> set newline type
null_context compile with a NULL context
null_pattern pass pattern as NULL
parens_nest_limit=<n> set maximum parentheses depth
posix use the POSIX API
posix_nosub use the POSIX API with REG_NOSUB
push push compiled pattern onto the stack
pushcopy push a copy onto the stack
pushtablescopy push a copy with tables onto the stack
stackguard=<number> test the stackguard feature
subject_literal treat all subject lines as literal
tables=[0|1|2|3] select internal tables
use_length do not zero-terminate the pattern
utf8_input treat input as UTF-8
</pre>
The effects of these modifiers are described in the following sections.
</p>
<br><b>
Newline and \R handling
</b><br>
<p>
The <b>bsr</b> modifier specifies what \R in a pattern should match. If it is
set to "anycrlf", \R matches CR, LF, or CRLF only. If it is set to "unicode",
\R matches any Unicode newline sequence. The default can be specified when
PCRE2 is built; if it is not, the default is set to Unicode.
</p>
<p>
The <b>newline</b> modifier specifies which characters are to be interpreted as
newlines, both in the pattern and in subject lines. The type must be one of CR,
LF, CRLF, ANYCRLF, ANY, or NUL (in upper or lower case).
</p>
<br><b>
Information about a pattern
</b><br>
<p>
The <b>debug</b> modifier is a shorthand for <b>info,fullbincode</b>, requesting
all available information.
</p>
<p>
The <b>bincode</b> modifier causes a representation of the compiled code to be
output after compilation. This information does not contain length and offset
values, which ensures that the same output is generated for different internal
link sizes and different code unit widths. By using <b>bincode</b>, the same
regression tests can be used in different environments.
</p>
<p>
The <b>fullbincode</b> modifier, by contrast, <i>does</i> include length and
offset values. This is used in a few special tests that run only for specific
code unit widths and link sizes, and is also useful for one-off tests.
</p>
<p>
The <b>info</b> modifier requests information about the compiled pattern
(whether it is anchored, has a fixed first character, and so on). The
information is obtained from the <b>pcre2_pattern_info()</b> function. Here are
some typical examples:
<pre>
re> /(?i)(^a|^b)/m,info
Capture group count = 1
Compile options: multiline
Overall options: caseless multiline
First code unit at start or follows newline
Subject length lower bound = 1
re> /(?i)abc/info
Capture group count = 0
Compile options: <none>
Overall options: caseless
First code unit = 'a' (caseless)
Last code unit = 'c' (caseless)
Subject length lower bound = 3
</pre>
"Compile options" are those specified by modifiers; "overall options" have
added options that are taken or deduced from the pattern. If both sets of
options are the same, just a single "options" line is output; if there are no
options, the line is omitted. "First code unit" is where any match must start;
if there is more than one they are listed as "starting code units". "Last code
unit" is the last literal code unit that must be present in any match. This is
not necessarily the last character. These lines are omitted if no starting or
ending code units are recorded. The subject length line is omitted when
<b>no_start_optimize</b> is set because the minimum length is not calculated
when it can never be used.
</p>
<p>
The <b>framesize</b> modifier shows the size, in bytes, of each storage frame
used by <b>pcre2_match()</b> for handling backtracking. The size depends on the
number of capturing parentheses in the pattern. A vector of these frames is
used at matching time; its overall size is shown when the <b>heaframes_size</b>
subject modifier is set.
</p>
<p>
The <b>callout_info</b> modifier requests information about all the callouts in
the pattern. A list of them is output at the end of any other information that
is requested. For each callout, either its number or string is given, followed
by the item that follows it in the pattern.
</p>
<br><b>
Passing a NULL context
</b><br>
<p>
Normally, <b>pcre2test</b> passes a context block to <b>pcre2_compile()</b>. If
the <b>null_context</b> modifier is set, however, NULL is passed. This is for
testing that <b>pcre2_compile()</b> behaves correctly in this case (it uses
default values).
</p>
<br><b>
Passing a NULL pattern
</b><br>
<p>
The <b>null_pattern</b> modifier is for testing the behaviour of
<b>pcre2_compile()</b> when the pattern argument is NULL. The length value
passed is the default PCRE2_ZERO_TERMINATED unless <b>use_length</b> is set.
Any length other than zero causes an error.
</p>
<br><b>
Specifying pattern characters in hexadecimal
</b><br>
<p>
The <b>hex</b> modifier specifies that the characters of the pattern, except for
substrings enclosed in single or double quotes, are to be interpreted as pairs
of hexadecimal digits. This feature is provided as a way of creating patterns
that contain binary zeros and other non-printing characters. White space is
permitted between pairs of digits. For example, this pattern contains three
characters:
<pre>
/ab 32 59/hex
</pre>
Parts of such a pattern are taken literally if quoted. This pattern contains
nine characters, only two of which are specified in hexadecimal:
<pre>
/ab "literal" 32/hex
</pre>
Either single or double quotes may be used. There is no way of including
the delimiter within a substring. The <b>hex</b> and <b>expand</b> modifiers are
mutually exclusive.
</p>
<br><b>
Specifying the pattern's length
</b><br>
<p>
By default, patterns are passed to the compiling functions as zero-terminated
strings but can be passed by length instead of being zero-terminated. The
<b>use_length</b> modifier causes this to happen. Using a length happens
automatically (whether or not <b>use_length</b> is set) when <b>hex</b> is set,
because patterns specified in hexadecimal may contain binary zeros.
</p>
<p>
If <b>hex</b> or <b>use_length</b> is used with the POSIX wrapper API (see
<a href="#posixwrapper">"Using the POSIX wrapper API"</a>
below), the REG_PEND extension is used to pass the pattern's length.
</p>
<br><b>
Specifying a maximum for variable lookbehinds
</b><br>
<p>
Variable lookbehind assertions are supported only if, for each one, there is a
maximum length (in characters) that it can match. There is a limit on this,
whose default can be set at build time, with an ultimate default of 255. The
<b>max_varlookbehind</b> modifier uses the <b>pcre2_set_max_varlookbehind()</b>
function to change the limit. Lookbehinds whose branches each match a fixed
length are limited to 65535 characters per branch.
</p>
<br><b>
Specifying wide characters in 16-bit and 32-bit modes
</b><br>
<p>
In 16-bit and 32-bit modes, all input is automatically treated as UTF-8 and
translated to UTF-16 or UTF-32 when the <b>utf</b> modifier is set. For testing
the 16-bit and 32-bit libraries in non-UTF mode, the <b>utf8_input</b> modifier
can be used. It is mutually exclusive with <b>utf</b>. Input lines are
interpreted as UTF-8 as a means of specifying wide characters. More details are
given in
<a href="#inputencoding">"Input encoding"</a>
above.
</p>
<br><b>
Generating long repetitive patterns
</b><br>
<p>
Some tests use long patterns that are very repetitive. Instead of creating a
very long input line for such a pattern, you can use a special repetition
feature, similar to the one described for subject lines above. If the
<b>expand</b> modifier is present on a pattern, parts of the pattern that have
the form
<pre>
\[<characters>]{<count>}
</pre>
are expanded before the pattern is passed to <b>pcre2_compile()</b>. For
example, \[AB]{6000} is expanded to "ABAB..." 6000 times. This construction
cannot be nested. An initial "\[" sequence is recognized only if "]{" followed
by decimal digits and "}" is found later in the pattern. If not, the characters
remain in the pattern unaltered. The <b>expand</b> and <b>hex</b> modifiers are
mutually exclusive.
</p>
<p>
If part of an expanded pattern looks like an expansion, but is really part of
the actual pattern, unwanted expansion can be avoided by giving two values in
the quantifier. For example, \[AB]{6000,6000} is not recognized as an
expansion item.
</p>
<p>
If the <b>info</b> modifier is set on an expanded pattern, the result of the
expansion is included in the information that is output.
</p>
<br><b>
JIT compilation
</b><br>
<p>
Just-in-time (JIT) compiling is a heavyweight optimization that can greatly
speed up pattern matching. See the
<a href="pcre2jit.html"><b>pcre2jit</b></a>
documentation for details. JIT compiling happens, optionally, after a pattern
has been successfully compiled into an internal form. The JIT compiler converts
this to optimized machine code. It needs to know whether the match-time options
PCRE2_PARTIAL_HARD and PCRE2_PARTIAL_SOFT are going to be used, because
different code is generated for the different cases. See the <b>partial</b>
modifier in "Subject Modifiers"
<a href="#subjectmodifiers">below</a>
for details of how these options are specified for each match attempt.
</p>
<p>
JIT compilation is requested by the <b>jit</b> pattern modifier, which may
optionally be followed by an equals sign and a number in the range 0 to 7.
The three bits that make up the number specify which of the three JIT operating
modes are to be compiled:
<pre>
1 compile JIT code for non-partial matching
2 compile JIT code for soft partial matching
4 compile JIT code for hard partial matching
</pre>
The possible values for the <b>jit</b> modifier are therefore:
<pre>
0 disable JIT
1 normal matching only
2 soft partial matching only
3 normal and soft partial matching
4 hard partial matching only
6 soft and hard partial matching only
7 all three modes
</pre>
If no number is given, 7 is assumed. The phrase "partial matching" means a call