Fixed javadoc errors

[dtonhofer]

About

As the Java API doc on the ANTLR 4 site does not permit searching (it fails with 404) I tried to build the Javadoc from source.

This was done using the latest Java 17 "javadoc" command.

I am not sure how the Javadoc is built in the ANTLR 4 project, but I could not build it "out of the box". Errors were generated.

So I have tracked them all down and updated the comments accordingly. In particular, some files include images using an "embed" tag, which is unknown to "javadoc", and I have changed it to an "img" tag.

What has changed

1. runtime/Java/src/org/antlr/v4/runtime/BufferedTokenStream.java: fixed UL tag
2. runtime/Java/src/org/antlr/v4/runtime/RuleContext.java : fixed preformatted code blocks
3. runtime/Java/src/org/antlr/v4/runtime/Vocabulary.java : replaced HTML table with pure text preformatted table because making the table readable is just not worth using HTML
4. runtime/Java/src/org/antlr/v4/runtime/atn/ArrayPredictionContext.java : EMPTY has been moved to another class, breaking the doc; hopefully fixed correctly
5. runtime/Java/src/org/antlr/v4/runtime/atn/CodePointTransitions.java: Fixed HTML entities
6. runtime/Java/src/org/antlr/v4/runtime/atn/ParserATNSimulator.java: Added "code" tags around code
7. runtime/Java/src/org/antlr/v4/runtime/atn/PredicateEvalInfo.java: SemanticContext#NONE no longer exists, reference removed (not sure about this)
8. runtime/Java/src/org/antlr/v4/runtime/misc/InterpreterDataReader.java: removed stray <, added a @param, enclosed file example into pre tags
9. runtime/Java/src/org/antlr/v4/runtime/tree/ParseTreeListener.java: Fixed HTML entity

Special changes due to images:

1. runtime/Java/src/org/antlr/v4/runtime/atn/ATNState.java: embed tags unknown to javadoc replaced by img tags, `alt attributes added
2. runtime/Java/src/org/antlr/v4/runtime/atn/PredictionContext.java: embed tags unknown to javadoc replaced by img tags, `alt attributes added

How do I build the Javadoc?

Here is the filetree showing the checked-out files on the left and the generated javadoc filetree on the right.

Step 1: Preparing a new branch

1. Fork the ANTLR repository to your own repository, making sure the checkbox "only fork the master branch" is unticked
2. git clone the forked ANTLR 4 repository to your own machine.
3. We suppose the root of the ANTLR 4 repository is $HOME/branch_antlr/antlr4
4. We also have these jars which are needed for javadoc generation:
   • ANTLR 4 jar
   • ANTLR 3 jar
   • icu4j-71.1.jar jar (package com.ibm.icu, get it at Maven central, this is needed to apply javadoc to the "tools" directory)
5. We also need the graphivz package which provides the dot command which is used to generate SVG images from DOT files.
6. Switch to branch "dev": git checkout dev
7. Create a new branch: git checkout -b javadoc_update
8. Modify the javadoc files in a cycle until satisfied

I assume there is bash and Perl around to run the generation scripts.

Step 2: Generate javadoc

Run the attached bash script run_javadoc.sh which deals with all the niceties (modify the location of the needed jar files as needed).

Make sure you answer yes to the request whether the javadoc subdirectory should be deleted before the javadoc command is run.

cd $HOME/branch_antlr
./run_javadoc.sh

There should not be any errors, but there are many warnings. No matter!

The result appears in directory javadoc.

Step 3: Generate images

Run the attached Perl script generate_images.pl which calls dot on all the .dot files. The data flow is indicated on the diagram.

cd $HOME/branch_antlr
PP="org/antlr/v4/runtime/atn"
perl generate_images.pl --src "antlr4/runtime/Java/src/main/dot/$PP/images/" --tgt "javadoc/$PP/images/"

Step 4: Verify results

This should show embedded images:

cd $HOME/branch_antlr
firefox ./javadoc/org/antlr/v4/runtime/atn/PredictionContext.html
firefox ./javadoc/org/antlr/v4/runtime/atn/ATNState.html

This should show acceptable tables:

firefox ./javadoc/org/antlr/v4/runtime/Vocabulary.html

generate_images.pl.txt
run_javadoc.sh.txt

[parrt]

Hi. is this the correct alt tag? I can't tell just by looking at the diff.

[dtonhofer]

Sorry for the late reply. Yes it is, in the sense of "it describes what the image shows"

[dtonhofer]

Regarding the 404 on: https://www.antlr.org/api/Java/

It is the search box which fails.

I also vaguely remember looking for a class but not finding it. It could be javadoc did not generate the HTML page.

[dtonhofer]

Regarding the question "did you try to run mvn to get javadoc" - no, I did not think about that.

Regarding the "embed" tag, it turned out that the "javadoc" tool rejects it. However, the "img" tag is allowed. Both "embed" and "img" are HTML content for javadoc according to this page:

• https://docs.oracle.com/en/java/javase/17/docs/specs/javadoc/doc-comment-spec.html

where it says "HTML constructs should be written in HTML 5"

So I went with the "img" tag, which empirically works (tested locally) but both the img tag and the embed tag are correct HTML5:

• https://html.spec.whatwg.org/multipage/embedded-content.html#the-img-element
• https://html.spec.whatwg.org/multipage/iframe-embed-object.html#the-embed-element

"Tutorialspoint" for one says all are fine:

• https://www.tutorialspoint.com/Should-I-use-img-object-or-embed-for-SVG-files

[dtonhofer]

All right, I missed that these instructions exist:

https://github.com/antlr/antlr4/blob/master/doc/releasing-antlr.md#javadoc-for-runtime-and-tool

[dtonhofer]

Let me check the last comment later today, I will have to do some garage work first.

[parrt]

All of the links from this page work for me. Can you be more specific about what links 404? https://www.antlr.org/api/

[parrt]

did you try to run mvn to get javadoc? We have to run a special process to generate those images. Somehow it's working to get the embed tag functional. Worried about screwing up that part.

This doc https://github.com/antlr/antlr4/blob/master/doc/releasing-antlr.md#javadoc-for-runtime-and-tool shows:

cd ~/antlr/code/antlr4
mvn -DskipTests javadoc:jar -q install # get lots of errors but works

All of your stuff seems to be correct except I'm worried about embed img thing.

[parrt]

I am getting some errors from your PR:

...
/Users/parrt/antlr/code/antlr4/runtime/Java/src/org/antlr/v4/runtime/ParserInterpreter.java:72: error: bad use of '>'
	/** We need a map from (decision,inputIndex)->forced alt for computing ambiguous
	                                             ^
/Users/parrt/antlr/code/antlr4/runtime/Java/src/org/antlr/v4/runtime/atn/PredictionContext.java:178: error: self-closing element not allowed
	 * <img src="images/SingletonMerge_SameRootSamePar.svg" 
	   ^
...

I also get a whole bunch of these errors during javadoc gen:

  symbol:   class ANTLRParser
  location: package org.antlr.v4.parse
/Users/parrt/antlr/code/antlr4/tool/src/org/antlr/v4/Tool.java:22: error: cannot find symbol
import org.antlr.v4.parse.ANTLRParser;
                         ^

Not sure why mvn can't see those. I ran mvn install first.

[parrt]

ping @dtonhofer