Merge pull request #1826 from vespa-engine/kkraune/linguistics

add details to guide

Author: kkraune

examples/lucene-linguistics/non-java/README.md

@@ -1,5 +1,4 @@
-
-<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. -->
+<!-- Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. -->
 
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="https://assets.vespa.ai/logos/Vespa-logo-green-RGB.svg">
@@ -9,48 +8,75 @@
 
 # Vespa sample applications - Lucene Linguistics 
 
-This app demonstrates using [Lucene Linguistics](https://docs.vespa.ai/en/lucene-linguistics.html).
-
+This app demonstrates using [Lucene Linguistics](https://docs.vespa.ai/en/linguistics/lucene-linguistics.html).
 
 <p data-test="run-macro init-deploy examples/lucene-linguistics/non-java">
 Requires at least Vespa 8.315.19
 </p>
 
-## To try this application
-
-Follow [Vespa getting started](https://docs.vespa.ai/en/basics/deploy-an-application)
+Follow [app deploy guide](https://docs.vespa.ai/en/basics/deploy-an-application)
 through the <code>vespa deploy</code> step, cloning `examples/lucene-linguistics/non-java` instead of `album-recommendation`.
 
+
+## Feed test data
 Feed 3 sample documents in Norwegian, Swedish, and Finnish: 
 
 <pre data-test="exec">
 vespa feed ext/*.json
 </pre>
 
-Example queries:
+
+## Run queries
 
 <pre data-test="exec" data-test-assert-contains="id:no:doc::1">
-vespa query 'yql=select * from doc where userQuery()'\
+vespa query 'yql=select * from doc where userInput(@query)'\
  'language=no' 'summary=debug-text-tokens' \
  'query=tips til utendørsaktiviteter'
 </pre>
 
 <pre data-test="exec" data-test-assert-contains="id:sv:doc::1">
-vespa query 'yql=select * from doc where userQuery()'\
+vespa query 'yql=select * from doc where userInput(@query)'\
  'language=sv' 'summary=debug-text-tokens' \
  'query=tips til utomhusaktiviteter'
 </pre>
 
 <pre data-test="exec" data-test-assert-contains="id:fi:doc::1">
-vespa query 'yql=select * from doc where userQuery()'\
+vespa query 'yql=select * from doc where userInput(@query)'\
  'language=fi' 'summary=debug-text-tokens' \
  'query=vinkkejä ulkoilma-aktiviteetteihin'
 </pre>
 
-### Terminate container 
+Using the [tokens](https://docs.vespa.ai/en/reference/schemas/schemas.html#tokens) feature,
+you can easily inspect the tokens generated by the linguistics processing
+
+
+### Use query tracing for insights
+Observe the query term linguistic processing in the trace output:
+
+<pre data-test="exec" data-test-assert-contains="id:no:doc::1">
+vespa query 'yql=select * from doc where userInput(@query)'\
+ language=no summary=debug-text-tokens \
+ 'query=tips til utendørsaktiviteter' \
+ tracelevel=1
+</pre>
 
-Remove the container after use (Only relevant for local deployments)
+
+Increasing tracelevel is often useful:
+
+<pre data-test="exec" data-test-assert-contains="id:no:doc::1">
+vespa query 'yql=select * from doc where ({grammar: "linguistics"}userInput(@query))'\
+ language=no summary=debug-text-tokens \
+ 'query=tips til utendørsaktiviteter' \
+ tracelevel=3
+</pre>
+
+Above, you will find output like _"Stemming with language NORWEGIAN_BOKMAL using LuceneLinguistics"_ -
+This helps you validate that your application config is correct
+
+
+
+## Cleanup
+For local deployments: Remove the container after use:
 <pre data-test="exec">
 $ docker rm -f vespa
 </pre>
-