Make EncodingTester usable in testing parsed state

This change updates EncodingTester to make it test the result for cases
when the expected character encoding is not limited to what can be
determined by checking only the first 1024 bytes of the input stream.

Otherwise, without this change, EncodingTester is limited to only being
useful for testing the output of the meta prescan.

This change also allows EncodingTester to be given a directory name
rather than a list of files (or pathname with a shell wildcard). And
when given a directory name, it recurses the directory looking for *.dat
files, and then run the tests from those files.

Without that change, we can’t easily run EncodingTester from AntRun in
Maven — because we can’t use shell wildcards in the “arg” value for the
Ant “java” task, and any list of files we otherwise construct within
Maven ends up getting put into the java arg value as a single string
(single argument) — including the spaces between filenames.

Author: sideshowbarker

test-src/nu/validator/htmlparser/test/EncodingTester.java

@@ -38,6 +38,8 @@ public class EncodingTester {
 
     private static int exitStatus = 0;
 
+    protected static int SNIFFING_LIMIT = 16384;
+
     private final InputStream aggregateStream;
 
     private final StringBuilder builder = new StringBuilder();
@@ -61,7 +63,7 @@ private boolean runTest() throws IOException, SAXException {
         }
         UntilHashInputStream stream = new UntilHashInputStream(aggregateStream);
         HtmlInputStreamReader reader = new HtmlInputStreamReader(stream, null,
-                null, null, Heuristics.NONE);
+                null, null, Heuristics.NONE, SNIFFING_LIMIT);
         Charset charset = reader.getCharset();
         stream.close();
         if (skipLabel()) {