postagger.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"[
]>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

<chapter id="tools.postagger">
<title>Part-of-Speech Tagger</title>
	<section id="tools.postagger.tagging">
		<title>Tagging</title>
		<para>
		The Part of Speech Tagger marks tokens with their corresponding word type
		based on the token itself and the context of the token. A token might have
		multiple pos tags depending on the token and the context. The OpenNLP POS Tagger
		uses a probability model to predict the correct pos tag out of the tag set.
		To limit the possible tags for a token a tag dictionary can be used which increases
		the tagging and runtime performance of the tagger.
		</para>
			<section id="tools.postagger.tagging.cmdline">
		<title>POS Tagger Tool</title>
		<para>
		The easiest way to try out the POS Tagger is the command line tool. The tool is
		only intended for demonstration and testing.
		Download the English maxent pos model and start the POS Tagger Tool with this command:
		<screen>
			<![CDATA[
$ opennlp POSTagger opennlp-en-ud-ewt-pos-1.3-2.5.4.bin]]>
		 </screen>
		The POS Tagger now reads a tokenized sentence per line from stdin.
		Copy these two sentences to the console:
		<screen>
			<![CDATA[
Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .
Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .]]>
		 </screen>
		 The POS Tagger will now echo the sentences with pos tags to the console:
		<screen>
			<![CDATA[
Pierre_PROPN Vinken_PROPN ,_PUNCT 61_NUM years_NOUN old_ADJ ,_PUNCT will_AUX join_VERB the_DET board_NOUN as_ADP
		a_DET nonexecutive_ADJ director_NOUN Nov._PROPN 29_NUM ._PUNCT
Mr._PROPN Vinken_PROPN is_AUX chairman_NOUN of_ADP Elsevier_ADJ N.V._PROPN ,_PUNCT the_DET Dutch_PROPN publishing_VERB group_NOUN .]]>
		 </screen>
		 The tag set used by the English pos model is the <ulink url="https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html">Penn Treebank tag set</ulink>.
		</para>
      </section>
      
		<section id="tools.postagger.tagging.api">
		<title>POS Tagger API</title>
		<para>
		    The POS Tagger can be embedded into an application via its API.
			First the pos model must be loaded into memory from disk or another source.
			In the sample below it is loaded from disk.
			<programlisting language="java">
				<![CDATA[
try (InputStream modelIn = new FileInputStream("opennlp-en-ud-ewt-pos-1.3-2.5.4.bin"){
  POSModel model = new POSModel(modelIn);
}]]>
			</programlisting>
			After the model is loaded the POSTaggerME can be instantiated.
			<programlisting language="java">
				<![CDATA[
POSTaggerME tagger = new POSTaggerME(model);]]>
			</programlisting>
			The POS Tagger instance is now ready to tag data. It expects a tokenized sentence
			as input, which is represented as a String array, each String object in the array
			is one token.
	   </para>
	   <para>
	   The following code shows how to determine the most likely pos tag sequence for a sentence.
	   	<programlisting language="java">
		  <![CDATA[
String[] sent = new String[]{"Most", "large", "cities", "in", "the", "US", "had",
                             "morning", "and", "afternoon", "newspapers", "."};		  
String[] tags = tagger.tag(sent);]]>
			</programlisting>
			The tags array contains one part-of-speech tag for each token in the input array. The corresponding
			tag can be found at the same index as the token has in the input array.
			The confidence scores for the returned tags can be easily retrieved from
			a POSTaggerME with the following method call:
				   	<programlisting language="java">
		  <![CDATA[
double[] probs = tagger.probs();]]>
			</programlisting>
			The call to probs is stateful and will always return the probabilities of the last
			tagged sentence. The probs method should only be called when the tag method
			was called before, otherwise the behavior is undefined.
			</para>
			<para>
			Some applications need to retrieve the n-best pos tag sequences and not
			only the best sequence.
			The topKSequences method is capable of returning the top sequences.
			It can be called in a similar way as tag.
			<programlisting language="java">
		  <![CDATA[
Sequence[] topSequences = tagger.topKSequences(sent);]]>
			</programlisting>	
			Each Sequence object contains one sequence. The sequence can be retrieved
			via Sequence.getOutcomes() which returns a tags array 
			and Sequence.getProbs() returns the probability array for this sequence.
	  		 </para>
	</section>
	</section>
		<section id="tools.postagger.training">
		<title>Training</title>
		<para>
			The POS Tagger can be trained on annotated training material. The training material
			is a collection of tokenized sentences where each token has the assigned part-of-speech tag.
			The native POS Tagger training material looks like this:
			<screen>
		  <![CDATA[
About_ADV 10_NUM Euro_PROPN ,_PUNCT I_PRON reckon._PUNCT
That_PRON sounds_VERB good_ADJ ._PUNCT]]>
			</screen>
			Each sentence must be in one line. The token/tag pairs are combined with "_".
			The token/tag pairs are whitespace separated. The data format does not
			define a document boundary. If a document boundary should be included in the
			training material it is suggested to use an empty line.
		</para>
		<para>The Part-of-Speech Tagger can either be trained with a command line tool,
		or via a training API.
		</para>
		
		<section id="tools.postagger.training.tool">
		<title>Training Tool</title>
		<para>
			OpenNLP has a command line tool which is used to train the models available from the model
			download page on various corpora.
		</para>
		<para>
		    Usage of the tool:
            <screen>
				<![CDATA[
$ opennlp POSTaggerTrainer
Usage: opennlp POSTaggerTrainer[.conllx] [-type maxent|perceptron|perceptron_sequence] \
               [-dict dictionaryPath] [-ngram cutoff] [-params paramsFile] [-iterations num] \
               [-cutoff num] -model modelFile -lang language -data sampleData \
               [-encoding charsetName]

Arguments description:
        -type maxent|perceptron|perceptron_sequence
                The type of the token name finder model. One of maxent|perceptron|perceptron_sequence.
        -dict dictionaryPath
                The XML tag dictionary file
        -ngram cutoff
                NGram cutoff. If not specified will not create ngram dictionary.
        -params paramsFile
                training parameters file.
        -iterations num
                number of training iterations, ignored if -params is used.
        -cutoff num
                minimal number of times a feature must be seen, ignored if -params is used.
        -model modelFile
                output model file.
        -lang language
                language which is being processed.
        -data sampleData
                data to be used, usually a file name.
        -encoding charsetName
                encoding for reading and writing text, if absent the system default is used.]]>
			 </screen>
		</para>
		<para>
		    The following command illustrates how an English part-of-speech model can be trained:
		    <screen>
		  <![CDATA[
$ opennlp POSTaggerTrainer -type maxent -model en-custom-pos-maxent.bin \
                           -lang en -data en-custom-pos.train -encoding UTF-8]]>
		    </screen>
		</para>
		</section>
		<section id="tools.postagger.training.api">
		<title>Training API</title>
		<para>
		The Part-of-Speech Tagger training API supports the training of a new pos model.
		Basically three steps are necessary to train it:
		<itemizedlist>
			<listitem>
				<para>The application must open a sample data stream</para>
			</listitem>
			<listitem>
				<para>Call the 'POSTagger.train' method</para>
			</listitem>
			<listitem>
				<para>Save the POSModel to a file</para>
			</listitem>
		</itemizedlist>
		The following code illustrates that:
		<programlisting language="java">
				<![CDATA[
POSModel model = null;

try {
  ObjectStream<String> lineStream = new PlainTextByLineStream(
  	new MarkableFileInputStreamFactory(new File("en-custom-pos-maxent.bin")), StandardCharsets.UTF_8);

  ObjectStream<POSSample> sampleStream = new WordTagSampleStream(lineStream);

  model = POSTaggerME.train("eng", sampleStream, TrainingParameters.defaultParams(), new POSTaggerFactory());
} catch (IOException e) {
  e.printStackTrace();
}]]>
	</programlisting>
	The above code performs the first two steps, opening the data and training
	the model. The trained model must still be saved into an OutputStream, in
	the sample below it is written into a file.
	<programlisting language="java">
				<![CDATA[
try (OutputStream modelOut = new BufferedOutputStream(new FileOutputStream(modelFile))){
  model.serialize(modelOut);
}]]>
		</programlisting>
		</para>
		</section>
		<section id="tools.postagger.training.tagdict">
		<title>Tag Dictionary</title>
		<para>
		The tag dictionary is a word dictionary which specifies which tags a specific token can have. Using a tag
		dictionary has two advantages, inappropriate tags can not been assigned to tokens in the dictionary and the
		beam search algorithm has to consider fewer possibilities and can search faster.
		</para>
		<para>
		The dictionary is defined in a xml format and can be created and stored with the POSDictionary class.
		Below is an example to train a custom model using a tag dictionary.
		</para>
		<para>
		Sample POS Training material (file : en-custom-pos.train)
			<screen>
				<![CDATA[
It_PRON is_OTHER spring_PROPN season_NOUN. The_DET flowers_NOUN are_OTHER red_ADJ and_CCONJ yellow_ADJ ._PUNCT
Red_NOUN is_OTHER my_DET favourite_ADJ colour_NOUN ._PUNCT]]>
			</screen>
		</para>
		<para>
		Sample Tag Dictionary (file : dictionary.xml)
			<programlisting language="xml">
				<![CDATA[
<?xml version="1.0" encoding="UTF-8"?>
 <dictionary case_sensitive="false">
  <entry tags="PRON">
    <token>It</token>
  </entry>
  <entry tags="OTHER">
    <token>is</token>
  </entry>
  <entry tags="PROPN">
    <token>Spring</token>
  </entry>
  <entry tags="NOUN">
    <token>season</token>
  </entry>
  <entry tags="DET">
    <token>the</token>
  </entry>
  <entry tags="NOUN">
    <token>flowers</token>
  </entry>
  <entry tags="OTHER">
    <token>are</token>
  </entry>
  <entry tags="NOUN">
    <token>red</token>
  </entry>
  <entry tags="CCONJ">
    <token>and</token>
  </entry>
  <entry tags="NOUN">
    <token>yellow</token>
  </entry>
  <entry tags="PRON">
    <token>my</token>
  </entry>
  <entry tags="ADJ">
    <token>favourite</token>
  </entry>
  <entry tags="NOUN">
    <token>colour</token>
  </entry>
  <entry tags="PUNCT">
    <token>.</token>
  </entry>
</dictionary>]]>
			</programlisting>
		</para>
		<para>Sample code to train a model using above tag dictionary
			<programlisting language="java">
			<![CDATA[
POSModel model = null;
	try {
		ObjectStream<String> lineStream = new PlainTextByLineStream(
				new MarkableFileInputStreamFactory(new File("en-custom-pos.train")), StandardCharsets.UTF_8);

		ObjectStream<POSSample> sampleStream = new WordTagSampleStream(lineStream);

		TrainingParameters params = ModelUtil.createDefaultTrainingParameters();
		params.put(TrainingParameters.CUTOFF_PARAM, 0);

		POSTaggerFactory factory = new POSTaggerFactory();
		TagDictionary dict = factory.createTagDictionary(new File("dictionary.xml"));
		factory.setTagDictionary(dict);

		model = POSTaggerME.train("eng", sampleStream, params, factory);

		OutputStream modelOut = new BufferedOutputStream(new FileOutputStream("en-custom-pos-maxent.bin"));
		model.serialize(modelOut);

	} catch (IOException e) {
		e.printStackTrace();
	}]]>
			</programlisting>
		</para>
		<para>
		The custom model is then used to tag a sequence.
		<programlisting language="java">
			<![CDATA[
String[] sent = new String[]{"Spring", "is", "my", "favourite", "season", "."};
String[] tags = tagger.tag(sent);
Arrays.stream(tags).forEach(k -> System.out.print(k + " "));]]>
		</programlisting>
		</para>
		<para>
			<literallayout>
				Input
				    Sentence:	Spring is my favourite season.

				Output
				    POS Tags using the custom model (en-custom-pos-maxent.bin): PROPN OTHER PRON ADJ NOUN PUNCT

				Output with the default model
				    POS Tags using the default model (opennlp-en-ud-ewt-pos-1.3-2.5.4.bin):	NOUN AUX PRON ADJ NOUN PUNCT
			</literallayout>
		</para>
		</section>
		</section>
		
		<section id="tools.postagger.eval">
		<title>Evaluation</title>
		<para>
		The built-in evaluation can measure the accuracy of the pos tagger.
		The accuracy can be measured on a test data set or via cross validation.
		</para>
		<section id="tools.postagger.eval.tool">
		<title>Evaluation Tool</title>
		<para>
		There is a command line tool to evaluate a given model on a test data set.
		The following command shows how the tool can be run:
		<screen>
				<![CDATA[
$ opennlp POSTaggerEvaluator -model pt.postagger.bin -data pt.postagger.test -encoding utf-8]]>
			 </screen>
			 This will display the resulting accuracy score, e.g.:
			 <screen>
				<![CDATA[
Loading model ... done
Evaluating ... done

Accuracy: 0.9659110277825124]]>
			 </screen>
		</para>
            <para>
            There is a command line tool for cross-validation of the test data set.
            The following command shows how the tool can be run:
            <screen>
                    <![CDATA[
$ opennlp POSTaggerCrossValidator -lang pt -data pt.postagger.test -encoding utf-8]]>
                 </screen>
                 This will display the resulting accuracy score, e.g.:
                 <screen>
                    <![CDATA[
Accuracy: 0.9659110277825124]]>
                 </screen>
            </para>

		</section>
		</section>
</chapter>