Ability to use ParseErrorList with Parser.unescapeEntities()

[magictractor]

I am using Parser.unescapeEntities(). It is working well, but I decided I would like to use a ParseErrorList to log parser errors, such as missing a semicolon for character references.

This was tougher than I anticipated, primarily because Tokeniser is not public, so I could not make the obvious tweaks to a copy of the unescapeEntities() implementation.

I have got close using parseFragmentInput(), but it's non-trivial.

    /**
     * This replicates {@code Parser.unescapeEntities()}, but with accessible
     * {@code ParserError}s.
     */
    public String unescapeUsingParseFragment(String text) {
        // Empty string is used by Parser.unescapeEntities().
        String baseUri = "";
        // Need an RcData tag, so one of textarea or title, see TagSet. Or modify or create another.
        Element context = new PseudoTextElement(new Tag("title"), baseUri, new Attributes());

        Parser parser = Parser.htmlParser();
        parser.setTrackErrors(100);
        List<Node> nodes = parser.parseFragmentInput(text, context, baseUri);

        ParseErrorList errors = parser.getErrors();
        if (errors.isEmpty() || !errors.get(errors.size() - 1).getErrorMessage().startsWith("Unexpected EOF token []")) {
            throw new IllegalStateException("Expected an \"Unexpected EOF token\" error");
        }
        for (int i = 0; i < errors.size() - 1; i++) {
            System.out.println(errors.get(i));
        }

        if (nodes.isEmpty()) {
            // Happens when text input is an empty string,
            // but also seen for "&#x00;"
            return "";
        }
        else if (nodes.size() > 1) {
            throw new IllegalStateException("Expected a single node");
        }
        Node node = nodes.get(0);
        if (!TextNode.class.isInstance(node)) {
            throw new IllegalStateException("Expected a text node");
        }

        return node.nodeValue();
    }

It doesn't match Parser.unescapeEntities() in a couple of corner cases. I don't want to derail my own post, so I'll save those for another discussion (or a reply here).

I'd like to see a change to permit the use of ParseErrorList with Parser.unescapeEntities(). That could be done in a similar manner to Parser's static parseFragment() methods:

public static List<Node> parseFragment(String fragmentHtml, Element context, String baseUri)

public static List<Node> parseFragment(String fragmentHtml, Element context, String baseUri, ParseErrorList errorList)

suggesting that the existing:

public static String unescapeEntities(String string, boolean inAttribute)

could be overloaded with:

public static String unescapeEntities(String string, boolean inAttribute, ParseErrorList errorList)

I look forward to hearing your thoughts.

[jhy]

Hi there,

That makes sense. I wonder if it would be more flexible if we had a non-static version of unescapeEntities on Parser. That way you can configure a parser to track errors if desired and retrieve them after unescaping.

That would align more to the current API shape of tracking errors, and could include a limit on how many errors are collected, and would also allow other configuration for the method (not that I can think of what that might be right now).

[magictractor]

A non-static unescapeEntities method on Parser sounds ideal.

I've had a look at my failing tests. The corner cases I found with my workaround were:

• Any U+0000 NULL in the input is converted to U+FFFD REPLACEMENT CHARACTER.

• � and other numeric character references resolving to U+0000 result in empty output if there is nothing else in the input. Doubling up �� results in two U+0000s in the output. � with any other characters also produces U+0000 in the output.

The first of these is the correct behaviour. It's a by-product of me using a RCDATA context in my unescape code. Section 13.2.5.2 RCDATA state of the spec says that NULLs are converted to REPLACEMENT CHARACTERs.

Investigation of the second led me to raise issue #2395. The numeric character entities are converted by Tokeniser to U+0000 and then exhibit the behaviour described in the issue. Note though, that Tokeniser should convert � to U+FFFD REPLACEMENT CHARACTER. Tokeniser has comment "todo: check for extra illegal unicode points as parse errors". That is a limitation with Parser.unescapeEntities() as well as my error list workaround.

[jhy]

A non-static unescapeEntities method on Parser sounds ideal.

OK, I've added that in #2396.

Will take a look at the replacement character issue you raised, thanks.

[jhy]

Have fixed the null char remove / replace in #2395.

We should still update the unescape in Tokenizer to replace disallowed characters.

Character references spec says:

The numeric character reference forms described above are allowed to reference any code point excluding U+000D CR, noncharacters, and controls other than ASCII whitespace.