[DOCS] Add search and filtering tutorial/quickstart, edit filtering page #114353

leemthompo · 2024-10-08T18:45:57Z

Ꙫ URL PREVIEW

First draft was very WIPpy
Based on SME feedback 92e47af represents significant tightening and focus of tutorial
Includes minor update to the filtering overview 🚗

Todo

Change filename, ID

github-actions · 2024-10-08T18:46:08Z

Documentation preview:

✨ Changed pages

leemthompo · 2024-10-09T11:34:04Z

@elasticmachine update branch

kderusso

Nice start so far!

docs/reference/query-dsl/query_filter_context.asciidoc

kderusso · 2024-10-09T12:47:20Z

docs/reference/quickstart/full-text-tutorial.asciidoc

+[TIP]
+====
+Full-text search is powered by <<analysis,text analysis>>. 
+Text analysis normalizes and standardizes text data so it can be efficiently stored in an inverted index and searched in near real-time.


Link to docs defining inverted index here?

We don't have this today :sad_panda:.

I would love to have a page about data structures in the basics section that summarizes how Elasticsearch stores different data types in optimized structures: inverted index for text, BKD trees for geospatial data, HNSW for vectors etc.

A lot of this core stuff is buried in blogs.

Another disappointing learning doing some of this work is that at some point the Definitive Guide was deprecated but not all of the essential basics were ported over into the docs!

See https://github.com/elastic/elasticsearch-definitive-guide/blob/master/052_Mapping_Analysis/35_Inverted_index.asciidoc. Need to restore this to the docs ASAP in the right spot.

kderusso · 2024-10-09T12:47:43Z

docs/reference/quickstart/full-text-tutorial.asciidoc

+
+[source,console]
+----
+POST /cooking_blog/_bulk


Would this make more sense in a notebook?

You mean the whole tutorial? For sure, the plan is to have notebook versions too once have a few of these :)

docs/reference/quickstart/full-text-tutorial.asciidoc

Co-authored-by: Kathleen DeRusso <[email protected]>

carlosdelest · 2024-10-14T10:52:00Z

docs/reference/query-dsl/query_filter_context.asciidoc

+
+* `filter` or `must_not` parameters in <<query-dsl-bool-query,`bool`>>  queries
+* `filter` parameter in <<query-dsl-constant-score-query,`constant_score`>> queries
+* <<search-aggregations-bucket-filter-aggregation,`filter`>> aggregations


Should we add knn here as an example as well?

~~It might be a bit too much in this 101 searching and filtering intro! This is the first tentative toe in the search paddling pool. 🦶~~

Currently I'm thinking the next basics "quickstart" could be asemantic_text quickstart (maybe ELSER + hybrid via rrf). That would be a toy example, API-only, with no reindexing or anything, just the happiest path — and then link to Search your data for the full-blown semantic search examples.

~~I think there might be a little too much in this first draft already, let's see how we feel once I've incorporated the first round of feedback.~~

@carlosdelest sorry I forgot that I edited this page too, so I glanced over this comment as if it related to the tutorial!

Indeed we should add whatever else is missing from this query_filter_context.asciidoc page too 😄

docs/reference/quickstart/full-text-tutorial.asciidoc

carlosdelest · 2024-10-14T13:14:34Z

docs/reference/quickstart/full-text-tutorial.asciidoc

+----
+GET /cooking_blog/_search
+{
+  "size": 0,


A comment on why we're using size: 0 would be interesting I think - we don't want to return search results, but just the aggregated information on all of them.

Out of scope now that we've decided to trim aggregations from this tutorial, but relevant for follow-up aggregations tutorial

docs/reference/quickstart/full-text-tutorial.asciidoc

Co-authored-by: Carlos Delgado <[email protected]>

Mikep86

Great start, thank you for doing this! I think the tutorial could benefit from a tighter "story arc", so to speak. Since this is a targeted at a brand new user, I say we try to cover less material with more detail, including putting the search responses in the documentation so that the user can see matching docs and we can explain why they match. I think it could also help to create a business scenario for each section to ground the tutorial in the real world.

We can stretch this material across multiple pages too. For example, there's no need to cover aggregations in the "searching and filtering" page, we can make an aggregation-specific tutorial page instead.

docs/reference/quickstart/full-text-tutorial.asciidoc

Mikep86 · 2024-10-14T19:20:52Z

docs/reference/quickstart/full-text-tutorial.asciidoc

+[discrete]
+=== Perform basic aggregations
+
+Aggregations provide summary statistics and analytics on your search results.


Can we link to a page with more detailed info on aggregations here?

out of scope now, we'll have a follow up aggregations heavy tutorial methinks

Added detailed examples of how to configure the cooking_blog index with text and keyword fields for different properties, improving exact matching and filtering capabilities. Expanded explanations and added concrete examples and responses to illustrate how match and multi_match queries work. Provided detailed examples on how to use filter in queries, including exact matches for categories and date ranges. Added a detailed example of a complex bool query combining multiple search criteria, tailored to the cooking blog scenario. Added learn more section

leemthompo · 2024-10-15T13:35:25Z

Tried to generally sharpen this tutorial, make it more user-focused, removing non-essential steps and going into more detail, with more annotated responses in 92e47af

…-fields explanation.

kderusso

Looks great, nice progress!

docs/reference/query-dsl/query_filter_context.asciidoc

kderusso · 2024-10-18T17:21:56Z

docs/reference/quickstart/full-text-filtering-tutorial.asciidoc

+<titleabbrev>Basics: Full-text search and filtering</titleabbrev>
++++
+
+This is a hands-on introduction to the basics of full-text search with {es}, also known as _lexical search_, using the <<search-search,`_search` API>> and <<query-dsl,Query DSL>>.


Philosophical question here, but should we refactor new tutorials to use retrievers now that they will be GA soon?

Maybe future tutorials, honestly I'd like an aligned answer from product because the query languages decision for new users is getting complex between query DSL / retrievers / ES|QL

docs/reference/quickstart/full-text-filtering-tutorial.asciidoc

leemthompo · 2024-10-22T13:58:38Z

@kderusso random question: when or where in a new user's journey would it be helpful to introduce the explain API?

kderusso · 2024-10-22T19:11:44Z

@kderusso random question: when or where in a new user's journey would it be helpful to introduce the explain API?

@leemthompo TBH, I could see the explain API deserving its own page with an explanation of how to use it?

leemthompo · 2024-10-25T11:17:34Z

@elasticmachine update branch

… initialization

Mikep86

Great work! I left a few comments about wording and technical edge-cases.

docs/reference/quickstart/full-text-filtering-tutorial.asciidoc

leemthompo · 2024-10-28T09:46:11Z

For the record @kderusso saved the day by finding that the docstests CI checks were failing due to invalid JSON 🥇

elasticsearchmachine · 2024-10-28T10:06:09Z

💔 Backport failed

The backport operation could not be completed due to the following error:

An unexpected error occurred when attempting to backport this PR.

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 114353

…age (elastic#114353) (cherry picked from commit 0d8d8bd)

leemthompo · 2024-10-28T10:09:20Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x
✅	8.16
✅	8.15

Questions ?

Please refer to the Backport tool documentation

…age (elastic#114353) (cherry picked from commit 0d8d8bd)

…age (#114353) (#115738) (cherry picked from commit 0d8d8bd)

…age (#114353) (#115740) (cherry picked from commit 0d8d8bd)

…age (#114353) (#115739) (cherry picked from commit 0d8d8bd)

…age (elastic#114353)

Add search and filtering tutorial, edit filtering page

4438d2a

leemthompo self-assigned this Oct 8, 2024

elasticsearchmachine added the v9.0.0 label Oct 8, 2024

leemthompo added 2 commits October 9, 2024 09:52

Fix example syntax

f32b83a

Fix syntax

59d98cc

Merge branch 'main' into qs-full-text

acd0055

kderusso reviewed Oct 9, 2024

View reviewed changes

leemthompo and others added 2 commits October 11, 2024 11:09

Merge branch 'main' into qs-full-text

549f110

Typos

7a0ead7

Co-authored-by: Kathleen DeRusso <[email protected]>

carlosdelest reviewed Oct 14, 2024

View reviewed changes

leemthompo commented Oct 14, 2024

View reviewed changes

docs/reference/quickstart/full-text-tutorial.asciidoc Outdated Show resolved Hide resolved

leemthompo commented Oct 14, 2024

View reviewed changes

docs/reference/quickstart/full-text-tutorial.asciidoc Outdated Show resolved Hide resolved

leemthompo and others added 2 commits October 14, 2024 15:34

Apply Carlos' suggestions

3abc6c8

Co-authored-by: Carlos Delgado <[email protected]>

Fix syntax

23f292a

Mikep86 reviewed Oct 14, 2024

View reviewed changes

leemthompo added 8 commits October 16, 2024 13:12

Fix test?

a3c6c87

docscheck CI whackamole

262a15b

Add links to Query DSL, mapping types, and bool query; expanded multi…

75c52de

…-fields explanation.

Add link to search api

e00dbbd

Will docscheck work now?

7d92427

Use substitutions in response test

4ca8132

Comment out flakey test

b677609

Add step numbers

faa12a5

kderusso reviewed Oct 18, 2024

View reviewed changes

leemthompo added 3 commits October 25, 2024 12:05

Fix docscheck?

dacbe7c

Is explicit analyzer here messing things up?

1295518

Fix wording

b46725a

elasticmachine and others added 6 commits October 25, 2024 13:17

Merge branch 'main' into qs-full-text

0624b19

Desperado

ab1f11b

Change query

df4184a

Split index creation + mapping steps and add refresh param for better…

2488afd

… initialization

Update result text

9589b6a

Fix took time

9cce834

Mikep86 approved these changes Oct 25, 2024

View reviewed changes

docs/reference/quickstart/full-text-filtering-tutorial.asciidoc Outdated Show resolved Hide resolved

docs/reference/quickstart/full-text-filtering-tutorial.asciidoc Show resolved Hide resolved

docs/reference/quickstart/full-text-filtering-tutorial.asciidoc Outdated Show resolved Hide resolved

leemthompo added 2 commits October 28, 2024 10:18

Fix test syntax, json spacing🤞

3eafe1b

Fixes per Mike's final pass

a083d42

leemthompo enabled auto-merge (squash) October 28, 2024 09:46

leemthompo merged commit 0d8d8bd into elastic:main Oct 28, 2024
4 of 5 checks passed

elasticsearchmachine added the backport pending label Oct 28, 2024

This was referenced Oct 28, 2024

[8.x] [DOCS] Add search and filtering tutorial/quickstart, edit filtering page (#114353) #115738

Merged

[8.16] [DOCS] Add search and filtering tutorial/quickstart, edit filtering page (#114353) #115739

Merged

leemthompo added a commit to leemthompo/elasticsearch that referenced this pull request Oct 28, 2024

[DOCS] Add search and filtering tutorial/quickstart, edit filtering p…

b836c53

…age (elastic#114353) (cherry picked from commit 0d8d8bd)

leemthompo added a commit to leemthompo/elasticsearch that referenced this pull request Oct 28, 2024

[DOCS] Add search and filtering tutorial/quickstart, edit filtering p…

215775c

…age (elastic#114353) (cherry picked from commit 0d8d8bd)

leemthompo mentioned this pull request Oct 28, 2024

[8.15] [DOCS] Add search and filtering tutorial/quickstart, edit filtering page (#114353) #115740

Merged

leemthompo added a commit to leemthompo/elasticsearch that referenced this pull request Oct 28, 2024

[DOCS] Add search and filtering tutorial/quickstart, edit filtering p…

035e61f

…age (elastic#114353) (cherry picked from commit 0d8d8bd)

elasticsearchmachine pushed a commit that referenced this pull request Oct 28, 2024

[DOCS] Add search and filtering tutorial/quickstart, edit filtering p…

8135f95

…age (#114353) (#115738) (cherry picked from commit 0d8d8bd)

elasticsearchmachine pushed a commit that referenced this pull request Oct 28, 2024

[DOCS] Add search and filtering tutorial/quickstart, edit filtering p…

331cff3

…age (#114353) (#115740) (cherry picked from commit 0d8d8bd)

elasticsearchmachine pushed a commit that referenced this pull request Oct 28, 2024

[DOCS] Add search and filtering tutorial/quickstart, edit filtering p…

0f1ecd9

…age (#114353) (#115739) (cherry picked from commit 0d8d8bd)

ioanatia pushed a commit to ioanatia/elasticsearch that referenced this pull request Nov 4, 2024

[DOCS] Add search and filtering tutorial/quickstart, edit filtering p…

88341f6

…age (elastic#114353)

jfreden pushed a commit to jfreden/elasticsearch that referenced this pull request Nov 4, 2024

[DOCS] Add search and filtering tutorial/quickstart, edit filtering p…

043ee20

…age (elastic#114353)

[DOCS] Add search and filtering tutorial/quickstart, edit filtering page #114353

[DOCS] Add search and filtering tutorial/quickstart, edit filtering page #114353

Uh oh!

Conversation

leemthompo commented Oct 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ꙫ URL PREVIEW

Todo

Uh oh!

github-actions bot commented Oct 8, 2024

Uh oh!

leemthompo commented Oct 9, 2024

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leemthompo Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leemthompo Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mikep86 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leemthompo Oct 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leemthompo commented Oct 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

leemthompo commented Oct 8, 2024 •

edited

Loading

leemthompo Oct 14, 2024 •

edited

Loading

leemthompo Oct 14, 2024 •

edited

Loading

leemthompo Oct 15, 2024 •

edited

Loading

leemthompo commented Oct 15, 2024 •

edited

Loading