diff --git a/content/develop/ai/search-and-query/advanced-concepts/aggregations-syntax.md b/content/develop/ai/search-and-query/advanced-concepts/aggregations-syntax.md new file mode 100644 index 0000000000..6eeacd1e03 --- /dev/null +++ b/content/develop/ai/search-and-query/advanced-concepts/aggregations-syntax.md @@ -0,0 +1,376 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +description: Order of operations for the FT.AGGREGATE command +linkTitle: Aggregation syntax +title: FT.AGGREGATE order of operations +weight: 2 +--- + +## Overview + +[`FT.AGGREGATE`]({{< relref "/commands/ft.aggregate" >}}) is a powerful Redis Query Engine (RQE) command for performing advanced data aggregation, filtering, sorting, and transformations on indexed hash or JSON documents. This reference page provides a structured breakdown of syntax, ordering rules, and best practices. + +The [main aggregations page]({{< relref "/develop/ai/search-and-query/advanced-concepts/aggregations" >}}) has a simple diagram showing how `FT.AGGREGATE` pipelines are constructed, but it doesn't tell the whole story. For example, you can create more complex aggregation pipelines by applying multiple `REDUCE` functions under a single `GROUPBY` clause, or you can chain groupings and mix in additional mapping steps: + +`GROUPBY` ... `REDUCE` ... `APPLY` ... `GROUPBY` ... `REDUCE` + +{{< note >}} +The examples on this page are based on a hypothetical "products" data set, which you can [download here](./data/products.txt). +{{< /note >}} + +## Syntax and expression ordering + +The `FT.AGGREGATE` command processes multiple expressions in a pipeline. Below is the recommended order: + +1. `index` – the name of your index, which must be the first argument. +1. `query` – your query, which must be the second argument. +1. `FILTER` – filters raw documents before transformations or aggregation. +1. `LOAD` – loads document fields. +1. `APPLY` – applies transformations on fields. +1. `GROUPBY` – groups results by specific fields. +1. `REDUCE` – performs aggregations. For example, `SUM`, `COUNT`, and `AVG`. +1. `SORTBY` – orders the results based on specified fields. +1. `LIMIT` – restricts the number of results returned. +1. `DIALECT 2` - provides for more comprehensive query syntax, for example using parameters in `FILTER` expressions. + +Other keywords will be discussed toward the end of this page. + +## When to use `@` + +You must add `@` at the start of a field name in the following circumstances: + +- When referencing fields loaded from documents. In the following example, `price` is a document field and must be prefixed with `@`. + +```sh +FT.AGGREGATE products "*" + LOAD 1 @price + APPLY "@price * 1.1" AS adjusted_price + SORTBY 2 @adjusted_price DESC + LIMIT 0 10 + + 1) (integer) 200 + 2) 1) "price" + 2) "623" + 3) "adjusted_price" + 4) "685.3" + 3) 1) "price" + 2) "619.75" + 3) "adjusted_price" + 4) "681.725" + . + . + . +``` + +- When referencing fields inside a `FILTER` clause that were loaded from documents. + +```sh +FT.AGGREGATE products "*" + LOAD 1 @rating + FILTER "@rating >= 4.5" + LIMIT 0 10 + + 1) (integer) 5 + 2) 1) "rating" + 2) "4.5" + 3) 1) "rating" + 2) "4.8" + 4) 1) "rating" + 2) "4.5" + . + . + . +``` + +- When referencing fields inside `GROUPBY` or `REDUCE` clauses. + +```sh +FT.AGGREGATE products "*" + GROUPBY 1 @category + REDUCE SUM 1 @price AS total_price + LIMIT 0 10 + +1) (integer) 6 +2) 1) "category" + 2) "Toys" + 3) "total_price" + 4) "9799.25" +3) 1) "category" + 2) "Electronics" + 3) "total_price" + 4) "10683.5" +4) 1) "category" + 2) "Apparel" + 3) "total_price" + 4) "10273.5" + . + . + . +``` + +- When referencing fields created by `REDUCE` in `APPLY` or `FILTER` clauses. + +```sh +FT.AGGREGATE products "*" + GROUPBY 1 @category + REDUCE SUM 1 @price AS total_price + APPLY "@total_price * 1.2" AS boosted_price + FILTER "@total_price > 1000" + LIMIT 0 10 + +1) (integer) 6 +2) 1) "category" + 2) "Toys" + 3) "total_price" + 4) "9799.25" + 5) "boosted_price" + 6) "11759.1" +3) 1) "category" + 2) "Electronics" + 3) "total_price" + 4) "10683.5" + 5) "boosted_price" + 6) "12820.2" + . + . + . +``` + +- When referencing fields created by `APPLY` in another `APPLY` or `FILTER` clause. + +```sh +FT.AGGREGATE products "*" + LOAD 2 @price @discount + APPLY "@price - @discount" AS net_price + APPLY "@net_price * 1.1" AS marked_up + FILTER "@net_price > 200" + LIMIT 0 10 + +1) (integer) 60 +2) 1) "price" + 2) "220" + 3) "discount" + 4) "0" + 5) "net_price" + 6) "220" + 7) "marked_up" + 8) "242" +3) 1) "price" + 2) "223.25" + 3) "discount" + 4) "1.5" + 5) "net_price" + 6) "221.75" + 7) "marked_up" + 8) "243.925" + . + . + . +``` + +- When referencing fields created by `APPLY` in a `SORTBY` clause. + +```sh +FT.AGGREGATE products "*" + LOAD 2 @price @discount + APPLY "@price - @discount" AS net_price + SORTBY 2 @net_price DESC + LIMIT 0 10 + + 1) (integer) 200 + 2) 1) "price" + 2) "623" + 3) "discount" + 4) "6" + 5) "net_price" + 6) "617" + 3) 1) "price" + 2) "619.75" + 3) "discount" + 4) "4.5" + 5) "net_price" + 6) "615.25" + . + . + . +``` + +## GROUPBY with multiple REDUCE operations + +You can use multiple `REDUCE` operations after `GROUPBY` for different aggregations. + +```sh +FT.AGGREGATE products "*" + GROUPBY 1 @category + REDUCE COUNT 0 AS product_count + REDUCE SUM 1 @price AS total_price + REDUCE AVG 1 @rating AS avg_rating + SORTBY 2 @total_price DESC + LIMIT 0 10 + +1) (integer) 6 +2) 1) "category" + 2) "Groceries" + 3) "product_count" + 4) "44" + 5) "total_price" + 6) "13495.25" + 7) "avg_rating" + 8) "3.94090909091" +3) 1) "category" + 2) "Home" + 3) "product_count" + 4) "40" + 5) "total_price" + 6) "11026.75" + 7) "avg_rating" + 8) "3.78" + . + . + . +``` + +## Multiple APPLY operations followed by GROUPBY and REDUCE + +You can use `APPLY` in various ways before and after `GROUPBY` and `REDUCE`. + +```sh +FT.AGGREGATE products "*" + LOAD 3 @price @discount @quantity + APPLY "@price - @discount" AS final_price + APPLY "@final_price * @quantity" AS total_revenue + GROUPBY 1 @category + REDUCE SUM 1 @total_revenue AS total_category_revenue + SORTBY 2 @total_category_revenue DESC + LIMIT 0 10 + +1) (integer) 6 +2) 1) "category" + 2) "Groceries" + 3) "total_category_revenue" + 4) "81373" +3) 1) "category" + 2) "Home" + 3) "total_category_revenue" + 4) "55797.5" + . + . + . +``` + +## FILTER and PARAMS + +Use `FILTER` to remove unwanted records, and `PARAMS` to pass values to parameterized queries. + +```sh +FT.AGGREGATE products "*" + LOAD 3 @price @rating @quantity + FILTER "@price >= 500" + FILTER "@rating >= 4.0" + APPLY "@price * @quantity" AS total_value + SORTBY 2 @total_value DESC + LIMIT 0 10 + DIALECT 2 + +1) (integer) 200 + 2) 1) "price" + 2) "606.75" + 3) "rating" + 4) "4.2" + 5) "quantity" + 6) "10" + 7) "total_value" + 8) "6067.5" + 3) 1) "price" + 2) "541.75" + 3) "rating" + 4) "4.5" + 5) "quantity" + 6) "10" + 7) "total_value" + 8) "5417.5" + . + . + . +``` + +## Placement of FILTER before and after GROUPBY/APPLY + +- **Before GROUPBY:** Removes records before aggregation. +- **After GROUPBY:** Filters based on aggregated results. + +## LOAD after GROUPBY/REDUCE + +This is not allowed and you'll get a syntax error. + +## Placement rules for specific parameters + +| Parameter | Placement | +|----- |----- | +| `TIMEOUT` | Can be placed anywhere. | +| `LIMIT` | Must be at the end, before `DIALECT`. | +| `WITHCURSOR` | Must be at the end, before `DIALECT`. | +| `SCORER` | Must be placed between the query and pipeline operations. | +| `ADDSCORES` | Must be placed between the query and pipeline operations. | +| `DIALECT` | Must be at the end. | + +## LIMIT and WITHCURSOR used together + +While you wouldn't ordinarily use `LIMIT` and `WITHCURSOR` together in the same query, you can do so if necessary. +`LIMIT`, as the name suggests, will limit the total number of results returned for the given query. `WITHCURSOR` will paginate the results in chunks of size `COUNT`. You can use the [cursor API]({{< relref "/develop/ai/search-and-query/advanced-concepts/aggregations/#cursor-api" >}}) to retrieve more results, as shown below. + +```sh +FT.AGGREGATE products "*" + GROUPBY 1 @category + REDUCE COUNT 0 AS product_count + LIMIT 0 100 + WITHCURSOR COUNT 3 + +1) 1) (integer) 6 + 2) 1) "category" + 2) "Toys" + 3) "product_count" + 4) "28" + 3) 1) "category" + 2) "Electronics" + 3) "product_count" + 4) "31" + 4) 1) "category" + 2) "Apparel" + 3) "product_count" + 4) "36" +2) (integer) 89400486 +127.0.0.1:6379> FT.CURSOR READ products 89400486 COUNT 3 +1) 1) (integer) 0 + 2) 1) "category" + 2) "Home" + 3) "product_count" + 4) "40" + 3) 1) "category" + 2) "Groceries" + 3) "product_count" + 4) "44" + 4) 1) "category" + 2) "Books" + 3) "product_count" + 4) "21" +2) (integer) 89400486 + . + . + . +``` + +See the following resources for more information: + +- [Aggregations]({{< relref "/develop/ai/search-and-query/advanced-concepts/aggregations" >}}) discussion page. +- [`FT.AGGREGATE` command page](https://redis.io/docs/latest/commands/ft.aggregate/) +- [RQE source code](https://github.com/RediSearch/RediSearch/tree/master/src/aggregate) diff --git a/content/develop/ai/search-and-query/advanced-concepts/autocomplete.md b/content/develop/ai/search-and-query/advanced-concepts/autocomplete.md index 6a02182d64..2eaafca312 100644 --- a/content/develop/ai/search-and-query/advanced-concepts/autocomplete.md +++ b/content/develop/ai/search-and-query/advanced-concepts/autocomplete.md @@ -14,7 +14,7 @@ categories: description: Learn how to use the autocomplete feature of Redis for efficient prefix-based suggestion retrieval. linkTitle: Autocomplete title: Autocomplete with Redis -weight: 1 +weight: 3 --- ## Overview diff --git a/content/develop/ai/search-and-query/advanced-concepts/stopwords.md b/content/develop/ai/search-and-query/advanced-concepts/stopwords.md index 78e292c840..2f96cbf2bc 100644 --- a/content/develop/ai/search-and-query/advanced-concepts/stopwords.md +++ b/content/develop/ai/search-and-query/advanced-concepts/stopwords.md @@ -14,7 +14,7 @@ categories: description: Stop words support linkTitle: Stop words title: Stop words -weight: 1 +weight: 4 --- Redis Open Source has a default list of [stop words](https://en.wikipedia.org/wiki/Stop_words). These are words that are usually so common that they do not add much information to search, but take up a lot of space and CPU time in the index.