No consistency in T_OPEN_TAG

[energy-coresky]

Description

Hello!
See picture..

[damianwadley]

Why does <?\n tokenize as T_OPEN_TAG then T_WHITESPACE but <?php\n tokenizes as just a T_OPEN_TAG including the newline?

[damianwadley]

(oops, early submit, but I guess this works too)

Because <? does not require whitespace after it for it to count as an opening tag, while <?php does require whitespace.

When short tags is enabled,

• <?xml is an opening tag followed by "xml"
• <?phpfoo is an opening tag followed by "phpfoo"

When short tags is disabled,

• <?xml is plain text
• <?phpfoo is plain text

[energy-coresky]

Your logic doesn't make sense in two reasons. Otherwise <? new self should be:

T_NEW --> 'new '  <---- with space at the end
T_STRING --> 'self'

But actualy work:

T_NEW --> 'new'
T_WHITESPACE --> ' '
T_STRING --> 'self'

Second: what sense to add space at the end of T_CLOSE_TAG using your logic?
By the way, imho, the new line is T_INLINE_HTML, in the above T_CLOSE_TAG

[energy-coresky]

I think it would be right not to include spaces at all in T_OPEN_TAG, T_CLOSE_TAG.
In that logic, your samples:

When short tags is enabled :
<?xml T_OPEN_TAG"<?" + T_STRING"xml"
<?phpfoo T_OPEN_TAG"<?" + T_STRING"phpfoo"
<? xml T_OPEN_TAG"<?" + T_WHITESPACE" " + T_STRING"xml"
<?php foo T_OPEN_TAG"<?php" + T_WHITESPACE" " + T_STRING"foo"
When short tags is disabled:
<?xml T_INLINE_HTML"<?xml"
<?phpfoo T_INLINE_HTML"<?phpfoo"
<? xml T_INLINE_HTML"<? xml"
<?php foo T_OPEN_TAG"<?php" + T_WHITESPACE" " + T_STRING"foo"

And what's the problem? The contents of the tokenizer tokens are a consequence of the PHP parser, not the cause. therefore, imho, your logic is not important.

Adding spaces to TAG tokens reduces the number of bugs when switching from "short_tag_open=On" and back? Doubtful. I'm writing the code. These gaps just make it more difficult to use the tokenizer. Although, of course, it was necessary from the very beginning not to include spaces in TAG tokens.

[cmb69]

The way it's currently done is to avoid even more complexity for the scanner and parser. It can be changed, but consumers of the token stream would need to adapt. All in all, I'm not convinced that this yields an overall benefit.

[energy-coresky]

Backwards incompatible changes are a real problem. I understand that. But these spaces in tag tokens are nonsense of the magic_quotes_gpc variety. The last one was successfully eliminated. It is necessary to consult with the leading developers of the main packages using the tokenizer. What impact will such changes have. And get rid of the spaces in token tags. Everyone, even the best programmers, have dark thoughts when unimportant things are considered important. It seems that these gaps belong to that category.

And the space after the closing tag can generally be considered an error.

[cmb69]

Maybe @jrfnl has thoughts about this?

[energy-coresky]

It seems clear what the matter is.. If you run this file:

<?php ?>
<?php

php file.php > z

Then the output will be an empty string. Although logically, a line break is T_INLINE_HTML. And if the file is like this:

<?php ?> <?php

Then the output will be an one char 0x20.
Once upon a time, in the early days of the interpreter, there was a frequent, hard-to-localize bug related to the fact that at the end of a file, a closing tag was specified and a line break was made. And now they've come up with a hack and have assigned the line break after the closing tag to the tag itself, and not for T_INLINE_HTML.

In general, everything is clear, it seems. But if my voice was of interest, I would remove all this. Nobody puts a closing tag at the end of a file anymore. And spaces in T_xxx_TAG are just confusion.

I guess we can close this issue.