- [`max()`](./function-syntax-semantics/avg-count-max-min-sum/#max-min)
- [`min()`](./function-syntax-semantics/avg-count-max-min-sum/#max-min)
- [`sum()`](./function-syntax-semantics/avg-count-max-min-sum/#sum) - - [`array_agg()`](./function-syntax-semantics/array-string-jsonb-jsonb-object-agg/#array-agg)
- [`string_agg()`](./function-syntax-semantics/array-string-jsonb-jsonb-object-agg/#string-agg)
- [`jsonb_agg()`](./function-syntax-semantics/array-string-jsonb-jsonb-object-agg/#jsonb-agg)
- [`jsonb_object_agg()`](./function-syntax-semantics/array-string-jsonb-jsonb-object-agg/#jsonb-object-agg) - - [`bit_and()`](./function-syntax-semantics/bit-and-or-bool-and-or/#bit-and)
- [`bit_or()`](./function-syntax-semantics/bit-and-or-bool-and-or/#bit-or)
- [`bool_and()`](./function-syntax-semantics/bit-and-or-bool-and-or/#bool-and)
- [`bool_or()`](./function-syntax-semantics/bit-and-or-bool-and-or/#bool-or) - - [`variance()`](./function-syntax-semantics/variance-stddev/#variance)
- [`var_pop()`](./function-syntax-semantics/variance-stddev/#var-pop)
- [`var_samp()`](./function-syntax-semantics/variance-stddev/#var-samp)
- [`stddev()`](./function-syntax-semantics/variance-stddev/#stddev)
- [`stddev_pop()`](./function-syntax-semantics/variance-stddev/#stddev-pop)
- [`stddev_samp()`](./function-syntax-semantics/variance-stddev/#stddev-samp) - - [`covar_pop()`](./function-syntax-semantics/linear-regression/covar-corr/#covar-pop-covar-samp)
- [`covar_samp()`](./function-syntax-semantics/linear-regression/covar-corr/#covar-pop-covar-samp)
- [`corr()`](./function-syntax-semantics/linear-regression/covar-corr/#corr) - - [`regr_avgy()`](./function-syntax-semantics/linear-regression/regr/#regr-avgy-regr-avgx)
- [`regr_avgx()`](./function-syntax-semantics/linear-regression/regr/#regr-avgy-regr-avgx)
- [`regr_count()`](./function-syntax-semantics/linear-regression/regr/#regr-count)
- [`regr_slope()`](./function-syntax-semantics/linear-regression/regr/#regr-slope-regr-intercept)
- [`regr_intercept()`](./function-syntax-semantics/linear-regression/regr/#regr-slope-regr-intercept)
- [`regr_r2()`](./function-syntax-semantics/linear-regression/regr/#regr-r2)
- [`regr_syy()`](./function-syntax-semantics/linear-regression/regr/#regr-syy-regr-sxx-regr-sxy)
- [`regr_sxx()`](./function-syntax-semantics/linear-regression/regr/#regr-syy-regr-sxx-regr-sxy)
- [`regr_sxy()`](./function-syntax-semantics/linear-regression/regr/#regr-syy-regr-sxx-regr-sxy) - - [`mode()`](./function-syntax-semantics/mode-percentile-disc-percentile-cont/#mode)
- [`percentile_disc()`](./function-syntax-semantics/mode-percentile-disc-percentile-cont/#percentile-disc-percentile-cont)
- [`percentile_cont()`](./function-syntax-semantics/mode-percentile-disc-percentile-cont/#percentile-disc-percentile-cont) - - [`rank()`](./function-syntax-semantics/rank-dense-rank-percent-rank-cume-dist/#rank)
- [`dense_rank()`](./function-syntax-semantics/rank-dense-rank-percent-rank-cume-dist/#dense-rank)
- [`percent_rank()`](./function-syntax-semantics/rank-dense-rank-percent-rank-cume-dist/#percent-rank)
- [`cume_dist()`](./function-syntax-semantics/rank-dense-rank-percent-rank-cume-dist/#cume-dist) - -### Aggregate functions case study—the "68–95–99.7" rule - -**[Here](case-study-the-68-95-997-rule/)**. Regard this section as an optional extra. It shows the use of aggregate functions to demonstrate the so-called "68–95–99.7 rule"—described in [this Wikipedia article](https://en.wikipedia.org/wiki/68%e2%80%9395%e2%80%9399.7_rule). This case-study focuses on just one part of the rule: - -> 68.27% of the values in a normal distribution lie within one standard deviation each side of the mean. diff --git a/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/_index.md b/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/_index.md deleted file mode 100644 index 024ca186bb07..000000000000 --- a/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/_index.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -title: > - Case study: linear regression analysis of COVID data -linkTitle: > - Case study: linear regression on COVID data -headerTitle: > - Case study: linear regression analysis of COVID data from Carnegie Mellon's COVIDcast project -description: Case study—using the YSQL regr_r2(), regr_slope(), regr_intercept() to examine the correlation between COVID-like symptoms and mask-wearing using data from Carnegie Mellon's COVIDcast. -image: /images/section_icons/api/subsection.png -menu: - preview_api: - identifier: covid-data-case-study - parent: aggregate-functions - weight: 110 -type: indexpage -showRightNav: true ---- -## Overview of the data and the code - -[Carnegie Mellon’s COVIDcast](https://covidcast.cmu.edu/) is an academic project that tracks real-time coronavirus statistics. The team uses various data collection methods and exposes data for download in various formats. This case study uses data that were collected using daily Facebook surveys with the aim of examining the possible correlation between wearing a face-mask and showing symptoms like those of SARS-CoV-2—hereinafter COVID. Specifically, three so-called signals are recorded. - -- Does the respondent wear a face mask? - -- Does the respondent have COVID-like symptoms? -- Does the respondent know someone in their community who has COVID-like symptoms? - -Each signal is expressed as a percentage relative to the number of people who answered the particular question. - -The download format, for each signal, is a comma-separated values file—hereinafter `.csv` file. The download page says this: - -> We are happy for you to use this [sic] data in products and publications. Please acknowledge us as a source: Data from Delphi COVIDcast, [covidcast.cmu.edu](https://covidcast.cmu.edu/). - -This case study shows you how to use the `ysqlsh` `\COPY` meta-command to load each downloaded file into its own table, how to check that the values conform to rules that the COVIDcast team has documented, and how to join the rows in these staging tables into a single table with this format: - -``` -survey_date date not null } primary -state text not null } key -mask_wearing_pct numeric not null -mask_wearing_stderr numeric not null -mask_wearing_sample_size integer not null -symptoms_pct numeric not null -symptoms_stderr numeric not null -symptoms_sample_size integer not null -cmnty_symptoms_pct numeric not null -cmnty_symptoms_stderr numeric not null -cmnty_symptoms_sample_size integer not null -``` - -It then shows you how to use the linear-regression functions [`regr_r2()`](../function-syntax-semantics/linear-regression/regr/#regr-r2), [`regr_slope()`](../function-syntax-semantics/linear-regression/regr/#regr-slope-regr-intercept), and [`regr_intercept()`](../function-syntax-semantics/linear-regression/regr/#regr-slope-regr-intercept) to examine the correlation between mask-wearing and COVID-like symptoms. The section [Functions for linear regression analysis](../function-syntax-semantics/linear-regression/) explains the general background for these functions. - -The remaining account of this case-study is divided into three parts: - -- [How to find and download the COVIDcast data](./download-the-covidcast-data/) -- [How to ingest the data](./ingest-the-covidcast-data/), check that the values conform to the rules that the COVIDcast team has documented, and to transform these into the single _"covidcast_fb_survey_results"_ table. -- [How to use YSQL aggregate functions to examine the possible correlation](./analyze-the-covidcast-data/) between wearing a face-mask and showing COVID-like symptoms. - -{{< tip title="Download a zip of all the files that this case study uses" >}} - -All of the `.sql` scripts that this case-study presents for copy-and-paste at the `ysqlsh` prompt are included for download in a zip-file. The zip also includes the three `csv` files that you will download from the [Carnegie Mellon COVIDcast](https://delphi.cmu.edu/covidcast/) site. This will allow you, after you've studied the account of the case study and run the files one by one, then to run everything by starting a single master script that will ingest the data and spool the reports the this study explains to files. It will allow you easily to re-run the analysis on newer data as these become available. - -It is expected that the raw data will be available from the COVIDcast site into the indefinite future. But the downloadable self-contained zip-fie of the complete case study assures readers of the longevity of this study's pedagogy. - -[Download `covid-data-case-study.zip`](https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/covid-data-case-study/covid-data-case-study.zip). - -After unzipping it on a convenient new directory, you'll see a `README.txt`. It tells you to run `0.sql`. You'll see this in the top directory. It looks like this: - -```plpgsql -\i ingest-the-data.sql -\i analysis-queries.sql -\i synthetic-data.sql -``` - -Simply start it in `ysqlsh`. You can run it time and again. It always finishes silently. You can see the reports that it produces on the _"analysis-results"_ directory and confirm that the files that are spooled are identical to the corresponding reference copies that are delivered in the zip-file. -{{< /tip >}} - -## Conclusion - -The function [`regr_r2()`](../function-syntax-semantics/linear-regression/regr/#regr-r2) implements a measure that the literature refers to as "R-squared". When the "R-squared" value is _0.6_, it means that _60%_ of the relationship of the putative _dependent_ variable (incidence of COVID-like symptoms) to the putative _independent_ variable (mask-wearing) is explained by a simple _"y = m*x + c"_ linear dependence—and that the remaining _40%_ is unexplained. A value greater than about _60%_ is generally taken to indicate that the putative _dependent_ variable really does depend upon the putative _independent_ variable. - -The downloaded COVIDcast data spanned a fifty day period (from 13-Sep-2020 through 1-Nov-2020). The value of "R-squared" was computed, in turn, for each of these days. It was greater than or equal to _60%_ on about _80%_ of these days. - -This outcome means that empirical evidence supports the claim that wearing a mask does indeed inhibit the spread of COVID. diff --git a/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/analyze-the-covidcast-data/_index.md b/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/analyze-the-covidcast-data/_index.md deleted file mode 100644 index 24db586ed9cd..000000000000 --- a/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/analyze-the-covidcast-data/_index.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -title: Analyze the COVIDcast data -linkTitle: Analyze the COVIDcast data -headerTitle: Using the YSQL linear regression analysis functions on the COVIDcast data—introduction -description: Using regr_r2(), regr_slope(), regr_intercept() on the COVIDcast data—introduction -image: /images/section_icons/api/subsection.png -menu: - preview_api: - identifier: analyze-the-covidcast-data - parent: covid-data-case-study - weight: 30 -type: indexpage -showRightNav: true ---- - -## Introduction - -Try this query: - -```plpgsql -select max(symptoms_pct) from covidcast_fb_survey_results; -``` - -The result is about _2.7%_. This indicates that the signal "_symptoms_pct"_ (characterized on the [COVIDcast download page](../ingest-the-covidcast-data/inspect-the-csv-files/) by "Percentage of people with COVID-like symptoms, based on surveys of Facebook users") has little power of discrimination. - -Now try this: - -```plpgsql -select - (select min(cmnty_symptoms_pct) as "Min" from covidcast_fb_survey_results), - (select max(cmnty_symptoms_pct) as "Max" from covidcast_fb_survey_results); -``` - -The results here are about _7%_ for the minimum and about _55%_ for the maximum. This indicates that the signal "_cmnty_symptoms_pct"_ (characterized on the [COVIDcast download page](../ingest-the-covidcast-data/inspect-the-csv-files/) by "Percentage of people who know someone in their local community with COVID-like symptoms, based on surveys of Facebook users") will have a reasonable power of discrimination. - -None of the YSQL built-in aggregate functions can take account of the _"stderr"_ or _"sample_size"_ values that were carried forward into the final _"covidcast_fb_survey_results"_ table. But you might like to try some _ad hoc_ queries to get an idea of the variability and reliability of the data. - -For example, this: - -```plpgsql -select avg(cmnty_symptoms_stderr) from covidcast_fb_survey_results; -``` - -gives a result of about _0.8_ for the percentage values in the range _7%_ through _55%_. This suggests that the seven day moving averages are reasonably reliable. - -And this: - -```plpgsql -select - (select min(cmnty_symptoms_sample_size) as "Min" from covidcast_fb_survey_results), - (select max(cmnty_symptoms_sample_size) as "Max" from covidcast_fb_survey_results); -``` - -results in about _325_ for the minimum and about _24.6_ thousand for the maximum. This is a rather troublesomely wide range. The result of this query: - -```plpgsql -select - round(avg(cmnty_symptoms_sample_size)) as "Avg", - state -from covidcast_fb_survey_results -group by state -order by 1; -``` - -suggests that the sample size is probably correlated with the state's population. For example, the two biggest sample size values are from California and Texas. and the two smallest are from DC and Wyoming. It would be straightforward to find a list of recent values for state populations from the Internet and to join these, using state, into a table together with the average sample sizes from the query above. You could then use the [`regr_r2()`](../../function-syntax-semantics/linear-regression/regr/#regr-r2) function to see how well-correlated the size of a state's response to the COVIDcast Facebook survey is to its population. This is left as an exercise for the reader. - -Create a view to focus your attention on the values that the analysis presented in the remainder of this section uses: - -```plpgsql -create or replace view covidcast_fb_survey_results_v as -select - survey_date, - state, - mask_wearing_pct, - cmnty_symptoms_pct as symptoms_pct -from covidcast_fb_survey_results; -``` - -This is included in the [`analysis-queries.sql`](./analysis-scripts/analysis-queries-sql/) script that also implements all of the queries that the analysis presented in the remainder of this section uses. - -If you want to see how the results come out when you use the _"symptoms_pct"_ column instead of the _"cmnty_symptoms_pct"_ column, just redefine the view, thus: - -```plpgsql -create or replace view covidcast_fb_survey_results_v as -select - survey_date, - state, - mask_wearing_pct, - symptoms_pct as symptoms_pct -from covidcast_fb_survey_results; -``` - -## How the rest of this analysis section is organized - -- The section [Daily values for regr_r2(), regr_slope(), regr_intercept() for symptoms vs mask-wearing](./daily-regression-analysis/) describes the actual linear regression analysis code. - -- The section [Select the data for COVID-like symptoms vs mask-wearing by state scatter plot](./symptoms-vs-mask-wearing-by-state/) shows the SQL that lists out the _51_ individual _"(symptoms_pct, mask_wearing_pct)"_ tuples for the day that was arbitrarily chosen for drawing a scatter-plot on top of which the outcome of the regression analysis for that day is drawn. diff --git a/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/analyze-the-covidcast-data/analysis-scripts/_index.md b/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/analyze-the-covidcast-data/analysis-scripts/_index.md deleted file mode 100644 index e9d628e02f60..000000000000 --- a/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/analyze-the-covidcast-data/analysis-scripts/_index.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: SQL scripts for analyzing the COVIDcast data -linkTitle: SQL scripts -headerTitle: SQL scripts for performing linear regression analysis on the COVIDcast data -description: SQL scripts for performing linear regression analysis on COVIDcast data -image: /images/section_icons/api/subsection.png -menu: - preview_api: - identifier: analysis-scripts - parent: analyze-the-covidcast-data - weight: 100 -type: indexpage ---- - -Here are the `.sql` scripts that jointly implement analysis: - -- [`analysis-queries.sql`](./analysis-queries-sql) executes queries on the actual COVIDcast data. - -- [`synthetic-data.sql`](./synthetic-data-sql) generates the data that are used to create the plot descibed in the section [Scatter-plot for synthetic data](../scatter-plot-for-2020-10-21/#scatter-plot-for-synthetic-data). diff --git a/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/ingest-the-covidcast-data/_index.md b/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/ingest-the-covidcast-data/_index.md deleted file mode 100644 index 72bf39ba8159..000000000000 --- a/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/ingest-the-covidcast-data/_index.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -title: Ingest the COVIDcast data into YugabyteDB -linkTitle: Ingest the COVIDcast data -headerTitle: Ingesting, checking, and combining the COVIDcast data -description: Ingest the COVIDcast data, check it for format compliance, and to combine it all into the single "covidcast_fb_survey_results" table -image: /images/section_icons/api/subsection.png -menu: - preview_api: - identifier: ingest-the-covidcast-data - parent: covid-data-case-study - weight: 20 -type: indexpage ---- - -## Ingest the .csv files, check the assumptions, and combine the interesting values into a single table - -Here are the steps: - -- [Manually inspect the .csv files](./inspect-the-csv-files). - -- [Copy the data from each `.csv` file "as is" into a dedicated staging table, with effective primary key _("state, survey_date)"_](./stage-the-csv-files). (The qualifier "effective" recognizes the fact that, as yet, these columns will have different names that reflect how they're named in the `.csv` files.) - -- [Check that the values from the `.csv` files do indeed conform to the stated rules](./check-data-conforms-to-the-rules). - -- [Project the columns of interest from the staging tables and join these into a single table](./join-the-staged-data/), with primary key _("state, survey_date)"_ for analysis. - -All of these steps are implemented by the [`ingest-the-data.sql`](./ingest-scripts/ingest-the-data-sql/) script. It's designed so that you can run, and re-run, it time and again. It will always finish silently (provided that you say `set client_min_messages = warning;`) Each time you run it. It calls various other scripts. You will download these, along with [`ingest-the-data.sql`](./ingest-scripts/ingest-the-data-sql/), as you step through the sections in the order that the left-hand navigation menu presents. diff --git a/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/ingest-the-covidcast-data/ingest-scripts/_index.md b/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/ingest-the-covidcast-data/ingest-scripts/_index.md deleted file mode 100644 index f00fc0ccf9a9..000000000000 --- a/docs/content/preview/api/ysql/exprs/aggregate_functions/covid-data-case-study/ingest-the-covidcast-data/ingest-scripts/_index.md +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: SQL scripts for ingesting the COVIDcast data -linkTitle: SQL scripts -headerTitle: SQL scripts for ingesting the COVIDcast data -description: SQL scripts for ingesting COVIDcast data -image: /images/section_icons/api/subsection.png -menu: - preview_api: - identifier: ingest-scripts - parent: ingest-the-covidcast-data - weight: 100 -type: indexpage ---- -Here are the `.sql` scripts that jointly implement the whole ingestion flow: creating staging tables; copying in the data from the `.csv` files; checking that the data in the staging tables conforms to the expected rules; and projecting the relevant columns and joining them into a single _"covidcast_fb_survey_results"_ table. - -- [`ingest-the-data.sql`](./ingest-the-data-sql) is the master script for this purpose. It contains some explicit SQL statements and it invokes other files that, for example, create procedures that the master script calls. Save it to the _"covid-data-case-study"_ directory that you created for this case study. You can't run it yet because the files that it needs aren't yet in place. You will step through each manually and then save it. When you've done this once, and saved all the files, you will then be able to run, and re-run, the whole ingestion process with a single "button press". This will be useful if you decide, later, to download more recent COVIDcast data. - - Here are the scripts that the master script relies on: - -- [`cr-cr-staging-tables.sql`](./cr-cr-staging-tables-sql) - -- [`cr-cr-copy-from-csv-scripts.sql`](./cr-cr-copy-from-csv-scripts-sql) - -- [`cr-assert-assumptions-ok.sql`](./cr-assert-assumptions-ok-sql) - -- [`cr-xform-to-covidcast-fb-survey-results.sql`](./cr-xform-to-joined-table-sql) diff --git a/docs/content/preview/api/ysql/exprs/aggregate_functions/function-syntax-semantics/_index.md b/docs/content/preview/api/ysql/exprs/aggregate_functions/function-syntax-semantics/_index.md deleted file mode 100644 index 44838e43dd07..000000000000 --- a/docs/content/preview/api/ysql/exprs/aggregate_functions/function-syntax-semantics/_index.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -title: YSQL aggregate functions signature and purpose -linkTitle: Per function signature and purpose -headerTitle: Signature and purpose of each aggregate function -description: This section summarizes the signature and purpose of each of the YSQL aggregate functions and links to their individual accounts. -image: /images/section_icons/api/subsection.png -menu: - preview_api: - identifier: aggregate-function-syntax-semantics - parent: aggregate-functions - weight: 90 -aliases: - - /preview/api/ysql/exprs/aggregate_functions -type: indexpage -showRightNav: true ---- - -The aggregate functions are categorized into four classes: - -- [general-purpose aggregate functions](#general-purpose-aggregate-functions) - -- [statistical aggregate functions](#statistical-aggregate-functions) - -- [_within-group ordered-set_ aggregate functions](#within-group-ordered-set-aggregate-functions) - -- [_within-group hypothetical-set_ aggregate functions](#within-group-hypothetical-set-aggregate-functions) - -## General-purpose aggregate functions - -The aggregate functions in this class can be invoked in one of two ways: - -- _Either_ "ordinarily" on all the rows in a table or in connection with `GROUP BY`, when they return a single value for the set of rows. - - In this use, row ordering often doesn't matter. For example, [`avg()`](./avg-count-max-min-sum/#avg) has this property. Sometimes, row ordering _does_ matter. For example, the order of grouped values determines the mapping between array index and array value with [`array_agg()`](./array-string-jsonb-jsonb-object-agg/#array-agg). - -- _Or_ as a [window function](../../window_functions/) with `OVER`. - - In this use, where the aggregate function is evaluated for each row in the [window](../../window_functions/invocation-syntax-semantics/#the-window-definition-rule), ordering always matters. - -Arguably, `avg()` might be better classified as a statistical aggregate function. (It is often used together with `stddev()`.) But, because it is so very commonly used, and used as a canonical example of the _"aggregate function"_ notion, it is classified here as a general-purpose aggregate function. - -| Function | Description | -| ---- | ---- | -| [`array_agg()`](./array-string-jsonb-jsonb-object-agg/#array-agg) | Returns an array whose elements are the individual values that are aggregated. This is described in full detail the [`array_agg()`](../../../datatypes/type_array/functions-operators/array-agg-unnest/#array-agg) subsection in the main [Array](../../../datatypes/type_array/) section. | -| [`avg()`](./avg-count-max-min-sum/#avg) | Computes the arithmetic mean of a set of summable values by adding them all together and dividing by the number of values. If the set contains nulls, then these are simply ignored—both when computing the sum and when counting the number of values. | -| [`bit_and()`](./bit-and-or-bool-and-or/#bit-and) | Returns a value that represents the outcome of the applying the two-by-two matrix `AND` rule to each alligned set of bits for the set of `NOT NULL` input values. | -| [`bit_or()`](./bit-and-or-bool-and-or#bit-or) | Returns a value that represents the outcome of the applying the two-by-two matrix `OR` rule to each alligned set of bits for the set of `NOT NULL` input values. | -| [`bool_and()`](./bit-and-or-bool-and-or/#bool-and) | Returns a value that represents the outcome of the applying the two-by-two matrix `AND` rule to the set of `NOT NULL` input boolean values. | -| [`bool_or()`](./bit-and-or-bool-and-or/#bool-or) | Returns a value that represents the outcome of the applying the two-by-two matrix `OR` rule to the set of `NOT NULL` input boolean values. | -| [`count()`](./avg-count-max-min-sum/#count) | Counts the number of non null values in a set. The data type of the values is of no consequence. | -| `every()` | `every()` is a synonym for [`bool_and()`](./bit-and-or-bool-and-or/#bool-and) | -| [`jsonb_agg()`](./array-string-jsonb-jsonb-object-agg/#jsonb-agg) | This, and `json_agg()` are described in detail the [`jsonb_agg()`](../../../datatypes/type_json/functions-operators/jsonb-agg/) section in the main [JSON](../../../datatypes/type_json/) section. | -| [`jsonb_object_agg()`](./array-string-jsonb-jsonb-object-agg/#jsonb-object-agg) | This and `json_object_agg()` are described in detail the [`jsonb_object_agg()`](../../../datatypes/type_json/functions-operators/jsonb-object-agg/) section in the main [JSON](../../../datatypes/type_json/) section. | -| [`max()`](./avg-count-max-min-sum/#max-min) | Computes the greatest value among the values in the set using the rule that is used for the particular data type in the ORDER BY clause. nulls are removed before sorting the values. | -| [`min()`](./avg-count-max-min-sum/#max-min) | Computes the least value among the values in the set using the rule that is used for the particular data type in the ORDER BY clause. nulls are removed before sorting the values. | -| [`string_agg()`](./array-string-jsonb-jsonb-object-agg/#string-agg) | Returns a single value produced by concatenating the aggregated values (first argument) separated by a mandatory separator (second argument). The first overload has `text` inputs and returns `text`. The second overload has `bytea` inputs and returns `bytea`. | -| [`sum()`](./avg-count-max-min-sum/#sum) | Computes the sum of a set of summable values by adding them all together. If the set contains nulls, then these are simply ignored. | -| [`xmlagg()`](https://www.postgresql.org/docs/15/functions-aggregate.html) | This is not supported through Version YB 2.2. See [GitHub Issue #1043](https://github.com/yugabyte/yugabyte-db/issues/1043) | - -## Statistical aggregate functions - -The aggregate functions in this class can be invoked in one of two ways: - -- _Either_ "ordinarily" on all the rows in a table or in connection with `GROUP BY`, when they return a single value for the set of rows. In this use, row ordering doesn't matter. - -- _Or_ as a [window function](../../window_functions/) with `OVER`. - - In this use, where the aggregate function is evaluated for each row in the [window](../../window_functions/invocation-syntax-semantics/#the-window-definition-rule), ordering always matters. - -| Function | Description | -| ---- | ---- | -| [`covar_pop()`](./linear-regression/covar-corr/#covar-pop-covar-samp) | Returns the so-called covariance, taking the available values to be the entire population. | -| [`covar_samp()`](./linear-regression/covar-corr/#covar-pop-covar-samp) | Returns the so-called covariance, taking the available values to be a sample of the population. | -| [`corr()`](./linear-regression/covar-corr/#corr) | Returns the so-called correlation coefficient. This measures the extent to which the _"y"_ values are linearly related to the _"x"_ values. A return value of 1.0 indicates perfect correlation. | -| [`regr_avgy()`](./linear-regression/regr/#regr-avgy-regr-avgx) | Returns the average of the first argument for those rows where both arguments are `NOT NULL`. | -| [`regr_avgx()`](./linear-regression/regr/#regr-avgy) | Returns the average of the second argument for those rows where both arguments are `NOT NULL`. | -| [`regr_count()`](./linear-regression/regr/#regr-count) | Returns the number of rows where both arguments are `NOT NULL`. | -| [`regr_slope()`](./linear-regression/regr/#regr-slope-regr-intercept) | Returns the slope of the straight line that linear regression analysis has determined best fits the "(y, x)" pairs. | -| [`regr_intercept()`](./linear-regression/regr/#regr-slope-regr-intercept) | Returns the point at which the straight line that linear regression analysis has determined best fits the "(y, x)" pairs intercepts the "y"-axis. | -| [`regr_r2()`](./linear-regression/regr/#regr-r2) | Returns the square of the correlation coefficient, [`corr()`](./linear-regression/covar-corr/#corr). | -| [`regr_syy()`](./linear-regression/regr/#regr-syy-regr-sxx-regr-sxy) | Returns `regr_count(y, x)*var_pop(y)` for `NOT NULL` pairs. | -| [`regr_sxx()`](./linear-regression/regr/#regr-syy-regr-sxx-regr-sxy) | Returns `regr_count(y, x)*var_pop(x)` for `NOT NULL pairs`. | -| [`regr_sxy()`](./linear-regression/regr/#regr-syy-regr-sxx-regr-sxy) | Returns `regr_count(y, x)*covar_pop(y, x)` for `NOT NULL` pairs. | -| [`variance()`](./variance-stddev/#variance) | The semantics of `variance()` and [`var_samp()`](./variance-stddev/#var-samp) are identical. | -| [`var_pop()`](./variance-stddev/#var-pop) | Returns the variance of a set of values using the "population" variant of the formula that divides by _N_, the number of values. | -| [`var_samp()`](./variance-stddev/#var-samp) | Returns the variance of a set of values using the "sample" variant of the formula that divides by _(N - 1)_ where _N_ is the number of values. | -| [`stddev()`](./variance-stddev/#stddev) | The semantics of `stddev()` and [`stddev_samp()`](./variance-stddev/#stddev-samp) are identical. | -| [`stddev_pop()`](./variance-stddev/#stddev-pop) | Returns the standard deviation of a set of values using the naïve formula (i.e. the "population" variant) that divides by the number of values, _N_. | -| [`stddev_samp()`](./variance-stddev/#stddev-samp) | Returns the standard deviation of a set of values using the "sample" variant of the formula that divides by _(N - 1)_ where _N_ is the number of values. | - -## Within-group ordered-set aggregate functions - -These functions are sometimes referred to as “inverse distribution” functions. They can be invoked only with the dedicated `WITHIN GROUP (ORDER BY ...)` syntax. They cannot be invoked as a [window function](../../window_functions/) with `OVER`. - -| Function | Description | -| ---- | ---- | -| [`mode()`](./mode-percentile-disc-percentile-cont/#mode) | Return the most frequent value of "sort expression". If there's more than one equally-frequent value, then one of these is silently chosen arbitrarily. | -| [`percentile_disc()`](./mode-percentile-disc-percentile-cont/#percentile-disc-percentile-cont) | Discrete variant. The scalar overload of `percentile_disc()` takes a percentile rank value as input and returns the value, within the specified window, that would produce this. The array overload takes an array of percentile rank values as input and returns the array of values that would produce these. | -| [`percentile_cont()`](./mode-percentile-disc-percentile-cont/#percentile-disc-percentile-cont) | Continuous variant. The scalar overload of `percentile_cont()` takes a percentile rank value as input and returns the value, within the specified window, that would produce this. The array overload takes an array of percentile rank values as input and returns the array of values that would produce these. | - -## Within-group hypothetical-set aggregate functions - -These functions as invoked, as within-group hypothetical-set aggregate functions, with the dedicated `WITHIN GROUP (ORDER BY ...)` syntax. Further, each of the functions listed in this class is associated with a [window function](../../window_functions/) of the same name. (But not every window function can be invoked in this way.) For each, the result is the value that the associated window function would have returned for the “hypothetical” row constructed, as the invocation syntax specifies, as if such a row had been added to the [window](../../window_functions/invocation-syntax-semantics/#the-window-definition-rule). - -| Function | Description | -| ---- | ---- | -| [`rank()`](./rank-dense-rank-percent-rank-cume-dist/#rank) | Returns the integer ordinal rank of each row according to the emergent order that the window `ORDER BY` clause specifies. The series of values starts with 1 but, when the window contains ties, the series is not dense. See the account of [rank()](../../window_functions/function-syntax-semantics/row-number-rank-dense-rank/#rank) in the [Window functions](../../window_functions/) section for more information. | -| [`dense_rank()`](./rank-dense-rank-percent-rank-cume-dist/#dense-rank) | Returns the integer ordinal rank of the distinct value of each row according to what the window `ORDER` BY clause specifies. The series of values starts with 1 and, even when the window contains ties, the series is dense. See the account of [dense_rank()](../../window_functions/function-syntax-semantics/row-number-rank-dense-rank/#dense-rank) in the [Window functions](../../window_functions/) section for more information.| -| [`percent_rank()`](./rank-dense-rank-percent-rank-cume-dist/#percent-rank) | Returns the percentile rank of each row within the window, with respect to the argument of the window_definition's window `ORDER BY` clause. See the account of [percent_rank()](../../window_functions/function-syntax-semantics/percent-rank-cume-dist-ntile/#percent-rank) in the [Window functions](../../window_functions/) section for more information. | -| [`cume_dist()`](./rank-dense-rank-percent-rank-cume-dist/#cume-dist) | Returns a value that represents the number of rows with values less than or equal to the current row’s value divided by the total number of rows—in other words, the relative position of a value in a set of values. See the account of [cume_dist()](../../window_functions/function-syntax-semantics/percent-rank-cume-dist-ntile/#cume-dist) in the [Window functions](../../window_functions/) section for more information. | diff --git a/docs/content/preview/api/ysql/exprs/aggregate_functions/function-syntax-semantics/linear-regression/_index.md b/docs/content/preview/api/ysql/exprs/aggregate_functions/function-syntax-semantics/linear-regression/_index.md deleted file mode 100644 index e0e6b7eef1dc..000000000000 --- a/docs/content/preview/api/ysql/exprs/aggregate_functions/function-syntax-semantics/linear-regression/_index.md +++ /dev/null @@ -1,176 +0,0 @@ ---- -title: covar_pop(), covar_samp(), corr(), regr_%() -linkTitle: linear regression -headerTitle: Functions for linear regression analysis -description: Describes the functionality of the covar_pop(), covar_samp(), corr(), and regr_%() family of YSQL aggregate functions for linear regression analysis -image: /images/section_icons/api/subsection.png -menu: - preview_api: - identifier: linear-regression - parent: aggregate-function-syntax-semantics - weight: 50 -type: indexpage -showRightNav: true ---- - -This parent section and its two child sections describe these aggregate functions for linear regression analysis: - -- [`covar_pop()`](./covar-corr/#covar-pop-covar-samp), [`covar_samp()`](./covar-corr/#covar-pop-covar-samp), [`corr()`](./covar-corr/#corr) -- [`regr_avgy()`](./regr/#regr-avgy-regr-avgx), [`regr_avgx()`](./regr/#regr-avgy-regr-avgx), [`regr_count()`](./regr/#regr-count), [`regr_slope()`](./regr/#regr-slope-regr-intercept), [`regr_intercept()`](./regr/#regr-slope-regr-intercept), [`regr_r2()`](./regr/#regr-r2), [`regr_syy()`](./regr/#regr-syy-regr-sxx-regr-sxy)[`regr_sxx()`](./regr/#regr-syy-regr-sxx-regr-sxy), [`regr_sxy()`](./regr/#regr-syy-regr-sxx-regr-sxy). - -## Overview - -See, for example, this Wikipedia article on [Regression analysis](https://en.wikipedia.org/wiki/Regression_analysis). Briefly, linear regression analysis estimates the relationship between a dependent variable and an independent variable, aiming to find the line that most closely fits the data. This is why each of the functions described has two input formal parameters. The _dependent variable_, the first formal parameter, is conventionally designated by _"y"_; and the _independent variable_, the second formal parameter, is conventionally designated by _"x"_. - -See, for example, the article ["How To Interpret R-squared in Regression Analysis"](https://statisticsbyjim.com/regression/interpret-r-squared-regression/). It says this: - -> Linear regression identifies the equation that produces the smallest difference between all of the observed values and their [fitted values](https://statisticsbyjim.com/glossary/fitted-values/). To be precise, linear regression finds the smallest sum of squared [residuals](https://statisticsbyjim.com/glossary/residuals/) that is possible for the dataset. - -In terms of the high school equation for a straight line: - -``` -y = m*x + c -``` - -the function [`regr_slope(y, x)`](./regr/#regr-slope-regr-intercept) estimates the gradient, _"m"_, of the straight line that best fits the set of coordinate pairs over which the aggregation is done; and the function [`regr_intercept(y, x)`](./regr/#regr-slope-regr-intercept) estimates its intercept with the y-axis, _"c"_. The so-called "R-squared " measure, implemented by [`regr_r2(y, x)`](./regr/#regr-r2), indicates the goodness-of-fit. It measures the percentage of the variance in the dependent variable that the independent variables explain collectively—in other words, the strength of the relationship between your model and the dependent variable on a 0 – 100% scale. For example, if `regr_r2()` returns a value of _0.7_, it means that seventy percent of the relationship between the putative dependent variable and the independent variable can be explained by a straight line with the gradient and intercept returned, respectively, by `regr_slope()` and `regr_intercept()`. The remaining thirty percent can be attributed to stochastic variation. - -The purpose of each of the functions is rather specialized; but the domain is also very familiar to people who need to do linear regression. For this reason, the aim here is simply to explain enough for specialists to be able to understand exactly what is available, and how to invoke what they decide that they need. Each function is illustrated with a simple example. - -Each of these aggregate functions is invoked by using the same syntax: - -- _either_ the simple syntax, `select aggregate_fn(expr, expr) from t` -- _or_ the `GROUP BY` syntax -- _or_ the `OVER` syntax - -Only the simple invocation is illustrated. See, for example, the sections [`GROUP BY` syntax](../avg-count-max-min-sum/#group-by-syntax) and [`OVER` syntax](../avg-count-max-min-sum/#over-syntax) in the section [`avg(), count(), max(), min(), sum()`](../avg-count-max-min-sum/) for how to use these syntax patterns. - -**Signature:** - -Each one of the aggregate functions for linear regression analysis, except for `regr_count()`, has the same signature: - -``` -input value: double precision, double precision - -return value: double precision -``` - -Because it returns a count, `regr_count()` returns a `bigint`, thus: - -``` -input value: double precision, double precision - -return value: bigint -``` -In all cases, the first input parameter represents the values that you want to be taken as the _dependent variable_ (conventionally denoted by _"y"_) and the second input parameter represents the values that you want to be taken as the _independent variable_ (conventionally denoted by _"x"_). - -{{< note title="About nullness" >}} -If, for a particular input row, _either_ the expression for _"y"_, _or_ the expression for _"x"_, evaluates to `null`, then that row is implicitly filtered out. -{{< /note >}} - -## Create the test table - -The same test table recipe serves for illustrating all of the functions for linear regression analysis. The design is straightforward. Noise is added to a pure linear function, thus: - -```output - y = slope*x + intercept + delta -``` - -where _"delta"_ is picked, for each _"x"_ value from a pseudorandom normal distribution with specified mean and standard deviation. - -The procedure _"populate_t()"_ lets you try different values for _"slope"_, _"intercept"_, and for the size and variability of _"delta"_. It uses the function `normal_rand()`, brought by the [tablefunc](../../../../../../explore/ysql-language-features/pg-extensions/extension-tablefunc) extension. - -```plpgsql -drop procedure if exists populate_t( - int, double precision, double precision, double precision, double precision) - cascade; -drop table if exists t cascade; - -create table t( - k int primary key, - x double precision, - y double precision, - delta double precision); - -create procedure populate_t( - no_of_rows in int, - slope in double precision, - intercept in double precision, - mean in double precision, - stddev in double precision) - language plpgsql -as $body$ -begin - delete from t; - - with - a1 as ( - select - s.v as k, - s.v as x, - (s.v * slope) + intercept as y - from generate_series(1, no_of_rows) as s(v)), - - a2 as ( - select ( - row_number() over()) as k, - r.v as delta - from normal_rand(no_of_rows, mean, stddev) as r(v)) - - insert into t(k, x, y, delta) - select - k, x, a1.y, a2.delta - from a1 inner join a2 using(k); - - insert into t(k, x, y, delta) values - (no_of_rows + 1, 0, null, null), - (no_of_rows + 2, null, 0, null); -end; -$body$; - -\set no_of_rows 100 -call populate_t( - no_of_rows => :no_of_rows, - - mean => 0.0, - stddev => 20.0, - slope => 5.0, - intercept => 3.0); - -\pset null
- [`rank()`](./function-syntax-semantics/row-number-rank-dense-rank/#rank)
- [`dense_rank()`](./function-syntax-semantics/row-number-rank-dense-rank/#dense-rank) - - [`percent_rank()`](./function-syntax-semantics/percent-rank-cume-dist-ntile/#percent-rank)
- [`cume_dist()`](./function-syntax-semantics/percent-rank-cume-dist-ntile/#cume-dist)
- [`ntile()`](./function-syntax-semantics/percent-rank-cume-dist-ntile/#ntile) - - [`first_value()`](./function-syntax-semantics/first-value-nth-value-last-value/#first-value)
- [`nth_value()`](./function-syntax-semantics/first-value-nth-value-last-value/#nth-value)
- [`last_value()`](./function-syntax-semantics/first-value-nth-value-last-value/#last-value) - - [`lag()`](./function-syntax-semantics/lag-lead/#lag)
- [`lead()`](./function-syntax-semantics/lag-lead/#lead) - - [The data sets used by the code examples](./function-syntax-semantics/data-sets/) - -### Analyzing a normal distribution with percent_rank(), cume_dist() and ntile() - -**[Here](./analyzing-a-normal-distribution/)**. Regard this section as an optional extra. It answers an interesting question: - -- If you want to allocate a row set into buckets where each contains the same number of rows, based on your ordering rule, is there any difference between the result produced by using, in turn, each of the three functions `percent_rank()`, `cume_dist()`, or `ntile()`? - -The answer is, of course, "Yes". But it's a qualified "Yes" because when certain conditions hold, there is _no_ difference. The value that the study brings is due to the fact that it aims to meet a high-level goal rather than to demonstrate the basic use of low-level primitives. This means that it necessarily combines the basic use of the window functions under study with all sorts of other generic SQL techniques (and even stored procedure techniques) because these are needed to meet the goal. This kind of bigger-picture use typically goes hand-in-hand with any serious use of window functions. diff --git a/docs/content/preview/api/ysql/exprs/window_functions/analyzing-a-normal-distribution/_index.md b/docs/content/preview/api/ysql/exprs/window_functions/analyzing-a-normal-distribution/_index.md deleted file mode 100644 index 0a79e5363f91..000000000000 --- a/docs/content/preview/api/ysql/exprs/window_functions/analyzing-a-normal-distribution/_index.md +++ /dev/null @@ -1,303 +0,0 @@ ---- -title: > - Case study: compare percent_rank(), cume_dist(), and ntile() on a normal distribution -linkTitle: > - Case study: analyzing a normal distribution -headerTitle: > - Case study: analyzing a normal distribution with percent_rank(), cume_dist(), and ntile() -description: Case study to compare and contrast the window functions percent_rank(), cume_dist(), and ntile() on large sets of normally distributed values. -image: /images/section_icons/api/subsection.png -menu: - preview_api: - identifier: analyzing-a-normal-distribution - parent: window-functions - weight: 40 -type: indexpage -showRightNav: true ---- - -## Introduction - -This section describes an empirical approach to answering the following question: - -- If you want to allocate a row set into buckets where each contains the same number of rows, based on your ordering rule, is there any difference between the result produced by using, in turn, each of the three functions [`percent_rank()`](../function-syntax-semantics/percent-rank-cume-dist-ntile/#percent-rank), [`cume_dist()`](../function-syntax-semantics/percent-rank-cume-dist-ntile/#cume-dist), or [`ntile()`](../function-syntax-semantics/percent-rank-cume-dist-ntile/#ntile)? - -The answer is, of course, "Yes"—why else would the three functions all be supported? The [`ntile()`](../function-syntax-semantics/percent-rank-cume-dist-ntile/#ntile) function is specified exactly to meet the stated goal. From the dedicated [`ntile()`](../function-syntax-semantics/percent-rank-cume-dist-ntile/#ntile) section: - -> **Purpose:** Return an integer value for each row that maps it to a corresponding percentile. For example, if you wanted to mark the boundaries between the highest-ranking 20% of rows, the next-ranking 20% of rows, and so on, then you would use `ntile(5)`. The top 20% of rows would be marked with _1_, the next-to-top 20% of rows would be marked with _2_, and so on so that the bottom 20% of rows would be marked with _5_. - -The other two functions implement more fine grained measures. Here's what the [`percent_rank()`](../function-syntax-semantics/percent-rank-cume-dist-ntile/#percent-rank) documentation says: - -> **Purpose:** Return the percentile rank of each row within the [_window_](../invocation-syntax-semantics/#the-window-definition-rule), with respect to the argument of the [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition)'s window `ORDER BY` clause. The value _p_ returned by `percent_rank()` is a number in the range _0 <= p <= 1_. It is calculated like this: -``` -percentile_rank = (rank - 1) / ("no. of rows in window" - 1) -``` -And here's what the [`cume_dist()`](../function-syntax-semantics/percent-rank-cume-dist-ntile/#cume-dist) documentation says: - -> **Purpose:** Return a value that represents the number of rows with values less than or equal to the current row’s value divided by the total number of rows—in other words, the relative position of a value in a set of values. The graph of all values of `cume_dist()` within the [_window_](../invocation-syntax-semantics/#the-window-definition-rule) is known as the cumulative distribution of the argument of the [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition)'s window `ORDER BY` clause. The value _c_ returned by `cume_dist()` is a number in the range _0 < c <= 1_. It is calculated like this: -``` -cume_dist() = - "no of rows with a value <= the current row's value" / - "no. of rows in window" -``` -The algorithm that the [`percent_rank()`](../function-syntax-semantics/percent-rank-cume-dist-ntile/#percent-rank) uses is different from the one that [`cume_dist()`](../function-syntax-semantics/percent-rank-cume-dist-ntile/#cume-dist) uses. And the algorithm that `ntile()` uses to produce its bucket allocations directly is unspecified. - -However, the answer "Yes" to the question "is there any difference between the result produced by each of the three functions?" is a qualified "Yes" because when certain conditions hold, there is no difference. - -The pedagogic value of this empirical study is brought by the fact that it aims to meet the stated high-level goal rather than to demonstrate the basic use of low-level primitives. This means that it necessarily combines the basic use of the window functions under study with all sorts of other generic SQL techniques (and even stored procedure techniques) because these are needed to meet the goal. This kind of bigger-picture use typically goes hand-in-hand with any serious use of window functions. - -Here is the problem statement at the next level of detail: - -- A row set with _N_ rows has only unique values of the column list that the OVER clause specifies for ordering the rows in the [_window_](../invocation-syntax-semantics/#the-window-definition-rule). (In other words, there are no ties.) -- The aim is to allocate the rows into _n_ buckets that each have the same number of rows. -- _N_ is an integral multiple of _n_. - -The study shows that if these special qualifications are met, then the bucket allocations are identical. It shows, too, that if either of those special qualifications isn't met (in other words, either if there are ties, or if _N_ is not an integral multiple of _n_), then the bucket allocations produced by each approach differ. - -## How to make best use of the code examples -Before starting, create the data set that the code relies on using the `ysqlsh` script that [table t4](../function-syntax-semantics/data-sets/table-t4/) presents. It takes only a few seconds to run, and then you can simply leave _"t4"_ untouched as you run, and re-run, the code examples. Notice that the table is populated by values that are created by a generator that picks, pseudorandomly, from an ideal normal distribution. This means that the values in table _"t4"_ will be different each time that you drop and re-create it. The code examples will therefore always show the same patterns. And the invariants that it illustrates will hold reliably. But the details will change with each new re-creation of _"t4"_. Your experience will be best if, as you step through the code examples, you run them all against fixed content in table _"t4"_. - -The steps that are described in the next section should all be run at the `ysqlsh` prompt. They do a mixture of things: - -- They use `.sql` scripts to drop and re-create schema objects of these kinds: - - views through which to access the `double precision` column, and then the `int` column, in table _"t4"_ through a uniform interface, so that the same-spelled queries can run in turn against each of these two columns - - tables to hold results from different queries so that later queries can compare these - - views through which to populate the separate results tables for the `double precision` column, and then the `int` column, through a uniform interface, so that the same-spelled reports can run in turn against each of these two columns - - PL/pgSQL procedures to populate the results tables and PL/pgSQL functions to encapsulate prepared queries. -- They use `.sql` scripts to run various queries, and execute various anonymous PL/pgSQL blocks (also known as `DO` blocks) that are too small to warrant encapsulation in functions and are best encapsulated in dedicated `.sql` scripts, -- And they invoke the functions and the procedures, and execute the `.sql` scripts, using small anonymous scripts that are simply presented in line in the account. - -It's best to create a dedicated user for the purpose of running this code that owns no other objects. - -Each of the `.sql` scripts is presented on a dedicated page. (You can see all of these in the order in which they are to be run in the navigation bar.) Each page starts with a sentence like this: - -> Save this script as `
-
- [Output from running `histogram()` on _"t4.dp_score"_](./histogram-report/)
- [Output from running `do_ntile()`, `do_percent_rank()`, and `do_cume_dist()` on _"t4.dp_score"_](./dp-results/)
- [Output from running `do_compare_dp_results.sql` on _"dp_results_"](./compare-dp-results/)
- [Output from running `do_ntile()`, `do_percent_rank()`, and `do_cume_dist()` on _"t4.int_score"_](./int-results/)
diff --git a/docs/content/preview/api/ysql/exprs/window_functions/function-syntax-semantics/_index.md b/docs/content/preview/api/ysql/exprs/window_functions/function-syntax-semantics/_index.md deleted file mode 100644 index 4153e06e715f..000000000000 --- a/docs/content/preview/api/ysql/exprs/window_functions/function-syntax-semantics/_index.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: YSQL window functions signature and purpose -linkTitle: Per function signature and purpose -headerTitle: Signature and purpose of each window function -description: This section summarizes the signature and purpose of each of the YSQL window functions and links to their individual accounts. -image: /images/section_icons/api/subsection.png -menu: - preview_api: - identifier: window-function-syntax-semantics - parent: window-functions - weight: 30 -aliases: - - /preview/api/ysql/exprs/window_functions -type: indexpage -showRightNav: true ---- - -The two tables at the end classify the eleven built-in window functions into two groups according to their general common characteristics. - -**Note:** The navigation bar lists these window functions in four functional groups. The members in each group bear a strong family resemblance to each other. The first two groups list functions from the [first table](./#window-functions-that-return-an-int-or-double-precision-value-as-a-classifier-of-the-rank-of-the-row-within-its-window) below. And the second two groups list functions from the [second table](./#window-functions-that-return-column-s-of-another-row-within-the-window). - -### Aggregate function variants - -A few of these also have an aggregate function variant. This can be seen with the `\df` meta-command. For example, `df lag` shows this: - -``` - Result data type | Argument data types | Type -------------------+----------------------------------------+-------- - bigint | | window - bigint | VARIADIC "any" ORDER BY VARIADIC "any" | agg -``` -This property is marked by _"Y"_ in the column _"agg?"_ in the following tables; a blank in this column means that the entry has only a window function variant. - -{{< note title="Functions with both a 'window' and an 'aggregate' variant" >}} - -The definitive description of the use, as an aggregate function, of a window function that has such a variant, is described within the [Aggregate functions](../../aggregate_functions/) major section in the section [Within-group hypothetical-set aggregate functions -](../../aggregate_functions/function-syntax-semantics/rank-dense-rank-percent-rank-cume-dist/). - -{{< /note >}} - -### Frame_clause sensitivity - -The results of all of the window functions depend upon what the window `ORDER BY` clause and the optional `PARTITION BY` clause say. Though you don't get an error if you omit the window `ORDER BY` clause, its omission brings unpredictable and therefore meaningless results. - -The results of a few of the window functions _are_ sensitive to what the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause) says. This is marked by _"Y"_ in the column _"frame?"_ in the following tables; a blank in this column means that the entry is _not_ sensitive to what the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause) says. - -#### Frame_clause-insensitive window functions - -All of the window functions listed in the first table, and `lag()` and `lead()` from the second table, are insensitive to whatever the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause) might say. You can easily show this by trying a few variants like, for example, this: - -``` --- Counter example. Don't use this for window functions --- that aren't frame_clause-sensitive. -range between 1 preceding and 1 following exclude current row -``` - -It says "consider only the row immediately before and the row immediately after the current row". You'll see that including this, or any other variant, in the [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) brings the identical result to what including only the window `ORDER BY` clause brings for each of the window functions that aren't sensitive to the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause). - -Yugabyte recommends that you never include a [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause) in the [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) that you use when you invoke a window function that isn't sensitive to the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause). - -#### Frame_clause-sensitive window functions - -The names of the other window functions that the second table lists, `first_value()`, `nth_value()` and `last_value()`, tell you that the output of each makes obvious sense when the scope within which the specified row is found is the entire [_window_](../invocation-syntax-semantics/#the-window-definition-rule). The results of these three functions certainly _are_ sensitive to the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause). This is the default for the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause): - -``` - -- This specification is used if you omit the frame_clause. - -- You probably don't want the results that this specifies. - between unbounded preceding and current row -``` - -See the section [Window function invocation—SQL syntax and semantics](../invocation-syntax-semantics). But the default does _not_ specify the entire [_window_](../invocation-syntax-semantics/#the-window-definition-rule). To do this, use this variant: - -```plpgsql --- You must specify this explicitly unless you are sure --- that you want a different specification. -range between unbounded preceding and unbounded following -``` - -Yugabyte recommends, therefore, that you include [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause) in the [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) that you use when you invoke a window function that is sensitive to the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause), unless you have one of the very rare use cases where the output that you want is produced by a different [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause). - -**Note:** The [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause)'s many variants are useful when an aggregate function is invoked using the `OVER` clause. The section [Using the aggregate function `avg()` to compute a moving average](./#using-the-aggregate-function-avg-to-compute-a-moving-average) shows an example. - -### Window functions that return an "int" or "double precision" value as a "classifier" of the rank of the row within its window - -The only information, in the input row set, that the functions in this group depend upon is the emergent order as determined by the window `ORDER BY` clause. The actual order depends in the usual way on what this clause says. With the one exception of `ntile()`, these functions have no formal parameters. The `ntile()` function has a single, mandatory `int` formal parameter. It specifies the number of subsets into which the input row set should be classified. This means that it reflects only the invoker's intention and does not reflect the shape of the input. - -| Function | agg? | frame? | Description | -| ---- | ---- | ---- | ---- | -| [`row_number()`](./row-number-rank-dense-rank/#row-number) | | | Returns a unique integer for each row in a [_window_](../invocation-syntax-semantics/#the-window-definition-rule), from a dense series that starts with _1_, according to the emergent order that the window `ORDER BY` clause specifies. For the two or more rows in a tie group, the unique values are assigned randomly. | -| [`rank()`](./row-number-rank-dense-rank/#rank) | Y | | Returns the integer ordinal rank of each row according to the emergent order that the window `ORDER BY` clause specifies. The series of values starts with 1 but, when the [_window_](../invocation-syntax-semantics/#the-window-definition-rule) contains ties, the series is not dense. The "ordinal rank" notion is familiar from sporting events. If three runners reach the finish line at the same time, then they are all deemed to have tied for first place. The runner who finishes next after these is deemed to have come in fourth place because three runners came in before this finisher. | -| [`dense_rank()`](./row-number-rank-dense-rank/#dense-rank) | Y | | Returns the integer ordinal rank of the distinct value of each row according to what the window `ORDER BY` clause specifies. The series of values starts with _1_ and, even when the [_window_](../invocation-syntax-semantics/#the-window-definition-rule) contains ties, the series is dense. The "dense rank" notion reflects the ordering of distinct values of the list of expressions that the window `ORDER BY` clause specifies. In the running race example, the three runners who tied for first place would get a dense rank of 1. And the runner who finished next after these would get a dense rank of 2, because this finisher got the second fastest distinct finish time. | -| [`percent_rank()`](./percent-rank-cume-dist-ntile/#percent-rank) | Y | | Returns the percentile rank of each row within the [_window_](../invocation-syntax-semantics/#the-window-definition-rule), with respect to the argument of the [`window_definition`](./#the-window-definition-rule)'s window `ORDER BY` clause, as a number in the range _0.0_ through _1.0_. The lowest value of `percent_rank()` within the [_window_](../invocation-syntax-semantics/#the-window-definition-rule) will always be _0.0_, even when there is a tie between the lowest-ranking rows. The highest possible value of `percent_rank()` within the [_window_](../invocation-syntax-semantics/#the-window-definition-rule) is _1.0_, but this value is seen only when the highest-ranking row has no ties. If the highest-ranking row does have ties, then the highest value of `percent_rank()` within the [_window_](../invocation-syntax-semantics/#the-window-definition-rule) will be correspondingly less than _1.0_ according to how many rows tie for the top rank. The notion is well-known from statistics. More details are given in this function's dedicated account. | -| [`cume_dist()`](./percent-rank-cume-dist-ntile/#cume-dist) | Y | | Returns a value that represents the number of rows with values less than or equal to the current row's value divided by the total number of rows—in other words, the relative position of a value in a set of values. The graph of all values of `cume_dist()` within the [_window_](../invocation-syntax-semantics/#the-window-definition-rule) is known as the cumulative distribution of the argument of the [`window_definition`](./#the-window-definition-rule)'s window `ORDER BY` clause. The value _c_ returned by `cume_dist()` is a number in the range _0 < c <= 1_. You can use `cume_dist()` to answer questions like this: Show me the rows whose score is within the top "x%" of the [_window_](../invocation-syntax-semantics/#the-window-definition-rule)'s population, ranked by score. The notion is well-known from statistics. More details are given in this function's dedicated account. | -| [`ntile()`](./percent-rank-cume-dist-ntile/#ntile) | | | Returns an integer value for each row that maps it to a corresponding percentile. For example, if you wanted to mark the boundaries between the highest-ranking 20% of rows, the next-ranking 20% of rows, and so on, then you would use `ntile(5)`. The top 20% of rows would be marked with _1_, the next-to-top 20% of rows would be marked with _2_, and so on, so that the bottom 20% of rows would be marked with _5_. If the number of rows in the [_window_](../invocation-syntax-semantics/#the-window-definition-rule), _N_, is a multiple of the actual value with which you invoke `ntile()`, `n`, then each percentile set would have exactly _N/n_ rows. This is achieved, if there are ties right at the boundary between two percentile sets, by randomly assigning some to one set and some to the other. If _N_ is not a multiple of _n_, then `ntile()` assigns the rows to the percentile sets so that the numbers assigned to each are as close as possible to being the same. | - -### Window functions that return columns of another row within the window - -The functions in this group depend, in the same way as those in the first group do, upon the emergent order as determined by the window `ORDER BY` clause. Each has, at least, a single, mandatory, `anyelement` formal parameter that specifies which value to fetch from the designated other row in the [_window_](../invocation-syntax-semantics/#the-window-definition-rule). If you want to fetch values from more than one column, you must combine them into a scalar value. The most obvious way to do this is to list the columns in the constructor for a user-defined _"row"_ type. The `nth_value()` function has a mandatory second `int` formal parameter that specifies the value of _"N"_ to give _"Nth"_ its meaning. The other functions in this group have just one formal parameter. - -| Function | agg? | frame? | Description | -| ---- | ---- | ---- | ---- | -| [`first_value()`](./first-value-nth-value-last-value/#first-value) | | Y | Returns the specified value from the first row, in the specified sort order, in the current [_window frame_](../invocation-syntax-semantics/#frame-clause-semantics-for-window-functions). If you specify the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause) to start at a fixed offset before the current row, then `first_value()` would produce the same result as would the correspondingly parameterized `lag()`. If this is your aim, then you should use `lag()` for clarity. | -| [`nth_value()`](./first-value-nth-value-last-value/#nth-value) | | Y | Returns the specified value from the "_Nth"_ row, in the specified sort order, in the current [_window frame_](../invocation-syntax-semantics/#frame-clause-semantics-for-window-functions). The second, mandatory, parameter specifies _"N"_ in _"Nth"_. | -| [`last_value()`](./first-value-nth-value-last-value/#last-value) | | Y | Returns the specified value from the last row, in the specified sort order, in the current [_window frame_](../invocation-syntax-semantics/#frame-clause-semantics-for-window-functions). | -| [`lag()`](./lag-lead/#lag) | | | Returns, for the current row, the designated value from the row in the ordered input set that is _"lag_distance "_ rows before it. The data type of the _return value_ matches that of the _input value_. `NULL` is returned when the value of _"lag_distance"_ places the earlier row before the start of the [_window_](../invocation-syntax-semantics/#the-window-definition-rule). Use the optional last parameter to specify the value to be returned, instead of `NULL`, when the looked-up row falls outside of the current [_window_](../invocation-syntax-semantics/#the-window-definition-rule). | -| [`lead()`](./lag-lead/#lead) | | | Returns, for the current row, the designated value from the row in the ordered input set that is _"lead_distance"_ rows after it. The data type of the _return value_ matches that of the _input value_. `NULL` is returned when the value of _"lead_distance"_ places the later row after the start of the [_window_](../invocation-syntax-semantics/#the-window-definition-rule). Use the optional last parameter to specify the value to be returned, instead of `NULL`, when the looked-up row falls outside of the current [_window_](../invocation-syntax-semantics/#the-window-definition-rule). | diff --git a/docs/content/preview/api/ysql/exprs/window_functions/function-syntax-semantics/data-sets/_index.md b/docs/content/preview/api/ysql/exprs/window_functions/function-syntax-semantics/data-sets/_index.md deleted file mode 100644 index cdb48e0d4750..000000000000 --- a/docs/content/preview/api/ysql/exprs/window_functions/function-syntax-semantics/data-sets/_index.md +++ /dev/null @@ -1,107 +0,0 @@ ---- -title: Example data sets for window-functions -linkTitle: Tables for the code examples -headerTitle: The data sets used by the code examples -description: This section describes and presents the code to create, selection of four data sets for the code examples that illustrate the use of window functions. -image: /images/section_icons/api/subsection.png -menu: - preview_api: - identifier: data-sets - parent: window-function-syntax-semantics - weight: 50 -type: indexpage ---- - -These four pages: - -- [table t1](./table-t1/) -- [table t2](./table-t2/) -- [table t3](./table-t3/) -- [table t4](./table-t4/) - -contain scripts to create and populate tables with data sets that are useful for demonstrating window function semantics. - -Each table uses a surrogate `uuid` primary key whose values are provided by the function `gen_random_uuid()`, brought by the `pgcrypto` extension. The procedure to populate table _"t4"_ also uses the function `normal_rand()`, brought by the `tablefunc` extension. These extensions are described in the sections [pgcrypto](../../../../../../explore/ysql-language-features/pg-extensions/extension-pgcrypto) and [tablefunc](../../../../../../explore/ysql-language-features/pg-extensions/extension-tablefunc). Each is a pre-bundled extension. This means that the installation for each will work without any preparatory steps, as long as you install them as a `superuser` like this: - -```plpgsql -create extension pgcrypto; -create extension tablefunc; -``` - -If you plan to run the code samples in this main "_Window functions"_ section on your laptop using a YugabyteDB cluster that you've created for your own personal use, then you probably have already adopted the habit of running any and all _ad hoc_ tests as a `superuser`. If so, then simply install the [pgcrypto](../../../../../../explore/ysql-language-features/pg-extensions/extension-pgcrypto) and [tablefunc](../../../../../../explore/ysql-language-features/pg-extensions/extension-tablefunc) extensions just as you'd do anything else and then create the tables _"t1"_, _"t2"_, _"t3"_, and _"t4"_. - -{{< note title="Note 1: about the installation of extensions" >}} - -If you've established the practice of creating different databases within your cluster for different purposes, and within a database using a regime of different users that own different schemas to model how a real-world application would be organized, then you'll doubtless want to create and install the extensions in a dedicated central schema and ensure that this is in the _search path_ for all ordinary users. - -{{< /note >}} - -{{< note title="Note 2: about the use of the gen_random_uuid() function" >}} - -Yugabyte recommends that, when you want a self-populating surrogate primary key column, you should use the approach shown here for the test tables, like this: - -```sql -create table my_table( - k uuid default gen_random_uuid() primary key, ... -``` - -This is preferred to using, for example `serial` or `bigserial`, like this: - -```sql -create table t4( - k serial primary key, ... -``` - -(This approach that is common, and that works well, in PostgreSQL—a monolithic SQL database.) This is because `serial` and `bigserial` use a `SEQUENCE` to generate unique values, and this involves expensive coordination between the nodes in a YugabyteDB cluster. In contrast, any invocation of `gen_random_uuid()` on any node, will reliably produce a new globally unique value entirely algorithmically. This brings a noticeable performance benefit. - -The tables _"t1"_, _"t2"_, and _"t3"_ have only a handful of rows and so this performance benefit is well below the noise level. But [table _"t4"_](./table-t4/) is populated using a purpose-written procedure parameterized with the number of rows to create. You get the most convincing demonstration effect with a large number, like _100,000_, rows. - -You can expect to see that populating the table "t4" if you use `gen_random_uuid()` is about _20x_ faster than if you use a sequence. - -{{< /note >}} - -Each of the tables _"t1"_, _"t2"_, _"t3"_, and _"t4"_ is populated so that the values of interest for the demonstrations come back in random order (as you are taught to expect) when a query has no `ORDER BY` clause. This takes just a little programming effort for the tables _"t1"_, _"t2"_, and _"t3"_. Effort is needed because, following a bulk insert into a newly-created table, queries with no `ORDER BY` clause tend to see the rows come back in the order in which they are inserted. And the `INSERT` statements for the tables _"t1"_, _"t2"_, and _"t3"_ explicitly list the to-be-inserted values in an intuitive order where they increase monotonically. No effort is needed for table _"t4"_ because the values of interest are generated by `normal_rand()`—which generates its values in a random order. - -Deliberately subverting this tendency (that when rows are inserted naïvely, as they usually are for functionality demonstrations, they come back in a "natural" order, even with no `ORDER BY`) allows a vivid demonstration of the fact that if the [`window_definition`](../../../../syntax_resources/grammar_diagrams/#window-definition) that's used to invoke _any_ window function has no window`ORDER BY` clause (even if there is such a clause at overall query level), then the results are unpredictable and therefore meaningless. This is demonstrated in the section [Showing the importance of the window ORDER BY clause](../../functionality-overview/#showing-the-importance-of-the-window-order-by-clause). (There are cases where the order doesn't matter—for example, when the set of rows is the input to a conventionally invoked aggregate function.) - -{{< note title="Save a script to (re)create all four test tables." >}} - -It's a good idea to save a script that you can use quickly and effortlessly to create the test tables should you lose, or change, them. This can happen easily with a YugabyteDB cluster on your laptop that you use for all sorts of _ad hoc_ tests. For example, it takes only moments to destroy and re-create the cluster (or even to upgrade to a new version of YugabyteDB)—and this is common practice for certain kinds of test. - - Of course, you must first visit each of these pages: - -- [table t1](./table-t1/) -- [table t2](./table-t2/) -- [table t3](./table-t3/) -- [table t4](./table-t4/) - -and save each of the scripts that these present onto the same directory where you save the "Master" installation script. - -Save this script as, for example, `install_all_tables.sql`: - -```plpgsql --- You can run this script time and again. It will always finish silently. - -\i t1.sql -\echo 't1 done' - -\i t2.sql -\echo 't2 done' - -\i t3.sql -\echo 't3 done' - -\i t4_1.sql -\i t4_2.sql -\echo 't4 done' -``` - -Then you can simply do this whenever you need to re-establish the state that the code examples rely on: - -```plpgsql -\i install_all_tables.sql -``` - -It takes only a few seconds to finish. Each of the scripts `t1.sql`, `t2.sql`, `t3.sql`, `t4_1.sql`, and `t4_2.sql` that it runs is designed to be able to be repeated. Each finishes silently on its first and all subsequent runs. - -{{< /note >}} diff --git a/docs/content/preview/api/ysql/exprs/window_functions/functionality-overview.md b/docs/content/preview/api/ysql/exprs/window_functions/functionality-overview.md deleted file mode 100644 index 8a0c1cc5366b..000000000000 --- a/docs/content/preview/api/ysql/exprs/window_functions/functionality-overview.md +++ /dev/null @@ -1,406 +0,0 @@ ---- -title: Window function invocation using the OVER clause -linkTitle: Informal functionality overview -headerTitle: Informal overview of window function invocation using the OVER clause -description: This section provides an informal introduction to the invocation of window functions and aggregate functions using the OVER clause. -menu: - preview_api: - identifier: window-functions-functionality-overview - parent: window-functions - weight: 10 -type: docs ---- - -A good sense of the general functionality of window functions is given by examples that use [`row_number()`](../function-syntax-semantics/row-number-rank-dense-rank/#row-number), [`nth_value()`](../function-syntax-semantics/first-value-nth-value-last-value/#nth-value), [`last_value()`](../function-syntax-semantics/first-value-nth-value-last-value/#last-value), [`lag()`](../function-syntax-semantics/lag-lead/#lag), and [`lead()`](../function-syntax-semantics/lag-lead/#lead). - -Aggregate functions can be invoked with the `OVER` clause. Examples are given using `avg()` and `sum()`. - -These examples are sufficient to give a general sense of the following notions: - -- how window functions are invoked, and their general semantics -- the three clauses of the [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) : the `PARTITION BY` clause, the window `ORDER BY` clause, and the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause) -- how an aggregate function gains different useful functionality when it's invoked using an `OVER` clause rather than (as is probably more common) in conjunction with the regular `GROUP BY` clause. - -{{< note title=" " >}} - -If you haven't yet installed the tables that the code examples use, then go to the section [The data sets used by the code examples](../function-syntax-semantics/data-sets/). - -{{< /note >}} - -## Using row_number() in the simplest way - -The [`row_number()`](../function-syntax-semantics/row-number-rank-dense-rank/#row-number) window function is the simplest among the set of eleven such functions that YSQL supports. Briefly, this function assigns an ordinal number, starting at _1_, to the rows within the specified [_window_](../invocation-syntax-semantics/#the-window-definition-rule) according to the specified ordering rule. Here is the most basic example. - -```plpgsql -select - k, - row_number() over(order by k desc) as r -from t1 -order by k asc; -``` - -The syntax and semantics of the `ORDER BY` clause, within the parentheses of the `OVER` clause, are identical to what you're used to when an `ORDER BY` clause is used after the `FROM` clause in a subquery. The `DESC` keyword is used in this example to emphasize this point. It says that the values returned by [`row_number()`](../function-syntax-semantics/row-number-rank-dense-rank/#row-number) are to be assigned in the order corresponding to sorting the values of _"k"_ in descending order—and it specifies nothing else. Here is the result: - -``` - k | r -----+---- - 1 | 25 - 2 | 24 - 3 | 23 - 4 | 22 - 5 | 21 - ... - 21 | 5 - 22 | 4 - 23 | 3 - 24 | 2 - 25 | 1 -``` - -The output lines for values of _"r"_ between _6_ and _20_ were manually removed to reduce the clutter. - -Because the `OVER` clause doesn't specify a `PARTITION BY` clause, the so-called [_window_](../invocation-syntax-semantics/#the-window-definition-rule) that [`row_number()`](../function-syntax-semantics/row-number-rank-dense-rank/#row-number) operates on coincides with all of the rows in table _"t1"_. - -The next example emphasizes the point that a window function is often used in a subquery which, like any other subquery, is used to define a `WITH` clause view to allow further logic to be applied—in this case, a `WHERE` cause restriction on the values returned by [`row_number()`](../function-syntax-semantics/row-number-rank-dense-rank/#row-number) (and, of course, a final query-level `ORDER BY` rule). - -```plpgsql -with v as ( - select - k, - row_number() over(order by k desc) as r - from t1) -select - k, - r -from v -where (r between 1 and 5) or (r between 21 and 25) -order by r asc; -``` - -This is the result: - -``` - k | r -----+---- - 25 | 1 - 24 | 2 - 23 | 3 - 22 | 4 - 21 | 5 - 5 | 21 - 4 | 22 - 3 | 23 - 2 | 24 - 1 | 25 -``` - -## Showing the importance of the window ORDER BY clause - -Here is a counter example. Notice that the [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) doesn't specify a window `ORDER BY` clause. - -```plpgsql -with a as ( - select - -- The use of the bare OVER() here brings meaningless results. - row_number() over () as r, - class, - k - from t1) -select - r, - class, - k, - case k=r - when true then 'true' - else '' - end as chk -from a -order by r; -``` - -To see the most dramatic effect of the unpredictability of the result set, save the code from [table t1](../function-syntax-semantics/data-sets/table-t1/) into a file called, say, _"unpredictable.sql"_. Then copy the SQL statement, above, at the end of this file and invoke it time and again in `ysqlsh`. Here is a typical result: - -``` - r | class | k | chk -----+-------+----+------ - 1 | 5 | 23 | - 2 | 5 | 25 | - 3 | 2 | 9 | - 4 | 1 | 4 | true - 5 | 3 | 11 | - 6 | 1 | 1 | - 7 | 3 | 13 | - 8 | 4 | 16 | - 9 | 1 | 2 | - 10 | 2 | 7 | - 11 | 1 | 3 | - 12 | 4 | 18 | - 13 | 3 | 15 | - 14 | 5 | 21 | - 15 | 3 | 14 | - 16 | 3 | 12 | - 17 | 4 | 17 | true - 18 | 1 | 5 | - 19 | 2 | 10 | - 20 | 4 | 20 | true - 21 | 5 | 24 | - 22 | 5 | 22 | true - 23 | 2 | 6 | - 24 | 2 | 8 | - 25 | 4 | 19 | -``` - -Sometimes, you'll see that, by chance, not a single output row is marked _"true"_. Sometimes, you'll see that a few are so marked. - -## Using row_number() with "PARTITION BY" - -This example adds a `PARTITION BY` clause to the window `ORDER BY` clause in the [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) . It selects and orders by _"v"_ rather than _"k"_ because this has `NULL`s and demonstrates the within-[_window_](../invocation-syntax-semantics/#the-window-definition-rule) effect of `NULLS FIRST`. The [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) is moved to a dedicated `WINDOW` clause that names it so that the `OVER` clause can simply reference the definition that it needs. This might seem only to add verbosity in this example. But using a dedicated `WINDOW` clause reduces verbosity when invocations of several different window functions in the same subquery use the same [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition). - -```plpgsql -\pset null '??' - -with a as ( - select - class, - v, - row_number() over w as r - from t1 - window w as (partition by class order by v desc nulls first)) -select - class, - v, - r -from a -where class in (2, 4) -order by class, r; -``` - -This is the result: - -``` - class | v | r --------+----+--- - 2 | ?? | 1 - 2 | 9 | 2 - 2 | 8 | 3 - 2 | 7 | 4 - 2 | 6 | 5 - 4 | ?? | 1 - 4 | 19 | 2 - 4 | 18 | 3 - 4 | 17 | 4 - 4 | 16 | 5 -``` - -## Using nth_value() and last_value() to return the whole row - -If you want the output value for any of `first_value()`, `last_value()`, `nth_value()`, `lag()`, or `lead()` to include more than one column, then you must list them in a _"row"_ type constructor. This example uses [`nth_value()`](../function-syntax-semantics/first-value-nth-value-last-value/#nth-value). This accesses the _Nth_ row within the ordered set that each [_window_](../invocation-syntax-semantics/#the-window-definition-rule) defines. It picks out the third row. The restriction _"class in (3, 5)"_ cuts down the result set to make it easier to read. - -```plpgsql -drop type if exists rt cascade; -create type rt as (class int, k int, v int); - -select - class, - nth_value((class, k, v)::rt, 3) over w as nv -from t1 -where class in (3, 5) -window w as ( - partition by class - order by k - range between unbounded preceding and unbounded following - ) -order by class; -``` - -It produces this result: - -``` - class | nv --------+----------- - 3 | (3,13,13) - 3 | (3,13,13) - 3 | (3,13,13) - 3 | (3,13,13) - 3 | (3,13,13) - 5 | (5,23,23) - 5 | (5,23,23) - 5 | (5,23,23) - 5 | (5,23,23) - 5 | (5,23,23) -``` - -Each of `first_value()`, `last_value()`, and `nth_value()`, as their names suggest, produces the same output for each row of a [_window_](../invocation-syntax-semantics/#the-window-definition-rule). It would be natural, therefore, to use the query above in a `WITH` clause whose final `SELECT` picks out the individual columns from the record and adds a `GROUP BY` clause, thus: - -```plpgsql -drop type if exists rt cascade; -create type rt as (class int, k int, v int); - -\pset null '??' -with a as ( - select - last_value((class, k, v)::rt) over w as lv - from t1 - window w as ( - partition by class - order by k - range between unbounded preceding and unbounded following)) -select - (lv).class, - (lv).k, - (lv).v -from a -group by class, k, v -order by class; -``` - -This example uses [`last_value()`](../function-syntax-semantics/first-value-nth-value-last-value/#last-value) because the data set has different values for _"k"_ and _"v"_ for the last row in each [_window_](../invocation-syntax-semantics/#the-window-definition-rule). This is the result: - -``` - class | k | v --------+----+---- - 1 | 5 | ?? - 2 | 10 | ?? - 3 | 15 | ?? - 4 | 20 | ?? - 5 | 25 | ?? -``` - -## Using lag() and lead() to compute a moving average - -The aim is to compute the moving average for each day within the window, where this is feasible, over the last-but one day, the last day, the current day, the next day, and the next-but-one day. - -Notice that the following section uses the aggregate function `avg()` to produce the same result, and it shows the advantages of that approach over using the window functions [`lag()`](../function-syntax-semantics/lag-lead/#lag) and [`lead()`](../function-syntax-semantics/lag-lead/#lead). There are many other cases where `lag()` and/or `lead()`are needed and where `avg()` is of no use. The present use case was chosen here because it shows very clearly what `lag()` and `lead()` do and, especially, because it allows the demonstration of invoking an aggregate function with an `OVER` clause. - -The query is specifically written to meet the exact requirements. It would need to be manually re-written to base the moving average on a bigger, or smaller, range of days. Notice that the same [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) , _"w"_, is used as the argument for each of the four uses of the `OVER` clause. This is where using a separate `WINDOW` clause delivers its intended benefit. - -The statement of requirement implies that the computation is not feasible for the first two and the last two days in the window. Under these circumstances, `lag()` and `lead()`, return `NULL`—or, it you prefer, a default value that you supply using an optional third parameter. See the dedicated section on [`lag()` and `lead()`](../function-syntax-semantics/lag-lead/) for details. - -```plpgsql -with v as ( - select - day, - lag (price::numeric, 2) over w as lag_2, - lag (price::numeric, 1) over w as lag_1, - price::numeric, - lead(price::numeric, 1) over w as lead_1, - lead(price::numeric, 2) over w as lead_2 - from t3 - window w as (order by day)) -select - to_char(day, 'Dy DD-Mon') as "Day", - ((lag_2 + lag_1 + price + lead_1 + lead_2)/5.0)::money as moving_avg -from v -where (lag_2 is not null) and (lead_2 is not null) -order by day; -``` - -This is the result: - -``` - Day | moving_avg -------------+------------ - Wed 17-Sep | $18.98 - Thu 18-Sep | $19.13 - Fri 19-Sep | $19.27 - Mon 22-Sep | $19.64 - Tue 23-Sep | $19.99 - Wed 24-Sep | $20.10 - Thu 25-Sep | $19.90 - Fri 26-Sep | $19.62 - Mon 29-Sep | $19.60 - Tue 30-Sep | $19.41 - Wed 01-Oct | $19.18 - Thu 02-Oct | $19.08 - Fri 03-Oct | $18.78 - Mon 06-Oct | $18.19 - Tue 07-Oct | $17.53 - Wed 08-Oct | $16.97 - Thu 09-Oct | $17.08 - Fri 10-Oct | $17.26 - Mon 13-Oct | $17.08 - Tue 14-Oct | $17.23 - Wed 15-Oct | $17.30 -``` - -## Using the aggregate function avg() to compute a moving average - -This solution takes advantage of this [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) to determine the rows that `avg()` uses: - -``` -order by day groups between $1 preceding and $1 following -``` - -Here, the statement is first prepared and then executed to emphasize the fact that a single formulation of the statement text works for any arbitrary range of days around the current row. The section [Window function invocation—SQL syntax and semantics](../invocation-syntax-semantics/) explains the full power of expression brought by the `OVER` clause. - -Notice that this approach uses the value returned by [`row_number()`](../function-syntax-semantics/row-number-rank-dense-rank/#row-number), using an `OVER` clause that does no more than order the rows, to exclude the meaningless first _N_ and last _N_ averages, where _N_ is the same parameterized value that _"groups between N preceding and N following"_ uses. These rows, if not excluded, would simply show the averages over the rows that allow access. You probably don't want to see those answers. - -```plpgsql -prepare stmt(int) as -with v as ( - select - day, - avg(price::numeric) over w1 as a, - row_number() over w2 as r - from t3 - window - w1 as (order by day groups between $1 preceding and $1 following), - w2 as (order by day)) -select - to_char(day, 'Dy DD-Mon') as "Day", - a::money as moving_avg -from v -where r between ($1 + 1) and (select (count(*) - $1) from v) -order by day; - -execute stmt(2); -``` - -The result is identical to that produced by the `lag()`/`lead()` approach. Try repeating the `EXECUTE` statement with a few different actual arguments. The bigger it gets, the fewer result rows you see, and the closer the values of the moving average get to each other. - -## Using the aggregate function sum() with the OVER clause - -This example shows a different spelling of the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause): - -``` -range between unbounded preceding and current row -``` - -so that the average includes, for each row, the row itself and only the rows that precede it in the sort order. - -```plpgsql -with v as ( - select - class, - k, - sum(k) over w as s - from t1 - window w as ( - partition by class - order by k - range between unbounded preceding and current row)) -select - class, - k, - s -from v -where class in (2, 4) -order by class, k; -``` - -This is the result: - -``` - class | k | s --------+----+---- - 2 | 6 | 6 - 2 | 7 | 13 - 2 | 8 | 21 - 2 | 9 | 30 - 2 | 10 | 40 - 4 | 16 | 16 - 4 | 17 | 33 - 4 | 18 | 51 - 4 | 19 | 70 - 4 | 20 | 90 -``` diff --git a/docs/content/preview/api/ysql/exprs/window_functions/invocation-syntax-semantics.md b/docs/content/preview/api/ysql/exprs/window_functions/invocation-syntax-semantics.md deleted file mode 100644 index 26bd1d64d123..000000000000 --- a/docs/content/preview/api/ysql/exprs/window_functions/invocation-syntax-semantics.md +++ /dev/null @@ -1,294 +0,0 @@ ---- -title: Window function syntax and semantics -linkTitle: Invocation syntax and semantics -headerTitle: Window function invocation—SQL syntax and semantics -description: This section specifies the syntax and semantics of the OVER clause and the WINDOW clause. You may also invoke aggregate functions t_is way. -menu: - preview_api: - identifier: window-functions-aggregate-functions-syntax-semantics - parent: window-functions - weight: 20 -type: docs ---- - -{{< note title="The rules described in this section also govern the invocation of aggregate functions." >}} - -The dedicated [Aggregate functions](../../aggregate_functions/) section explains that one kind of aggregate function—so-called ordinary aggregate functions, exemplified by [`avg()`](../../aggregate_functions/function-syntax-semantics/avg-count-max-min-sum/#avg) and [`count()`](../../aggregate_functions/function-syntax-semantics/avg-count-max-min-sum/#count)—can optionally be invoked using the identical syntax that you use to invoke window functions. That dedicated section has many examples. See also the sections [Using the aggregate function avg() to compute a moving average](../functionality-overview/#using-the-aggregate-function-avg-to-compute-a-moving-average) and [Using the aggregate function sum() with the OVER clause](../functionality-overview/#using-the-aggregate-function-sum-with-the-over-clause) in the present Window functions main section. -{{< /note >}} - -{{< note title="A note on orthography" >}} - -Notice these three different orthography styles: - -- `OVER` is a keyword that names a clause. You write such a keyword in a SQL statement. - -- [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) is the name of a rule within the overall SQL grammar. You never type such a name in a SQL statement. It is written in bold lower case with underscores, as appropriate, between the English words. Because such a rule is always shown as a link, you can jump directly to the rule in the [Grammar Diagrams](../../../syntax_resources/grammar_diagrams/#abort) page. This page shows every single one of the SQL rules. It so happens that the [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) rule starts with the keyword `WINDOW` and might therefore, according to the context of use, be referred to alternatively as the `WINDOW` clause. - -- [_window frame_](./#frame-clause-semantics-for-window-functions) is a pure term of art. It is written in italic lower case with spaces, as appropriate, between the English words. You neither write it in a SQL statement nor use it to look up anything in the [Grammar Diagrams](../../../syntax_resources/grammar_diagrams/#abort) page. Because such a term of art is always shown as a link, you can jump directly to its definition within this _"Window function invocation—SQL syntax and semantics"_ page. - -{{< /note >}} - -## Syntax - -### Reproduced from the SELECT statement section - -The following three diagrams, [`select_start`](../../../syntax_resources/grammar_diagrams/#select-start), [`WINDOW` clause](../../../syntax_resources/grammar_diagrams/#window-clause), and [`fn_over_window`](../../../syntax_resources/grammar_diagrams/#fn-over-window) rule, are reproduced from the section that describes the [`SELECT` statement](../../../the-sql-language/statements/dml_select/). - -{{%ebnf localrefs="window_definition"%}} -select_start, -window_clause, -fn_over_window -{{%/ebnf%}} - -### Definition of the window_definition rule - -As promised in the `SELECT` statement section, this section explains the [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) rule and its use as the argument of either the `OVER` keyword or the `WINDOW` keyword. - -A [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) can be used only at these two syntax spots, within the enclosing syntax of a subquery. - -{{%ebnf localrefs="frame_clause" %}} -window_definition -{{%/ ebnf %}} - -### The frame_clause - -{{% ebnf %}} -frame_clause, -frame_bounds, -frame_start, -frame_end, -frame_bound, -frame_exclusion -{{%/ ebnf %}} - -## Semantics - -### The fn_over_window rule - -A window function can be invoked only at the syntax spot in a subquery that the diagram for the [`select_start`](../../../syntax_resources/grammar_diagrams/#select-start) rule shows. An [aggregate function](../../aggregate_functions/) _may_ be invoked in this way as an alternative to its more familiar invocation as a regular `SELECT` list item in conjunction with the `GROUP BY` clause. (The invocation of an aggregate function in conjunction with the `GROUP BY` clause is governed by the `ordinary_aggregate_fn_invocation` rule or the `within_group_aggregate_fn_invocation` rule.) - -The number, data types, and meanings of a window function's formal parameters are function-specific. The eleven window functions are classified into functional groups, and summarized, in the two tables at the end of the section [Signature and purpose of each window function](../function-syntax-semantics/). Each entry links to the formal account of the function which also provides runnable code examples. - -Notice that, among the dedicated window functions (as opposed to aggregate functions that may be invoked as window functions), only [`ntile()`](../function-syntax-semantics/percent-rank-cume-dist-ntile/#ntile) takes an argument. Every other dedicated window function is invoked with an empty parentheses pair. Some aggregate functions (like, for example, [`jsonb_object_agg()`](../../aggregate_functions/function-syntax-semantics/array-string-jsonb-jsonb-object-agg/#jsonb-object-agg)) take more than one argument. When an aggregate function is invoke as a window function, the keyword `DISTINCT` is not allowed within the parenthesized list of arguments. The attempt causes this error: - -``` -0A000: DISTINCT is not implemented for window functions -``` - -### The window_definition rule - -The syntax diagram for the [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) shows that it uses three complementary specifications: - -- The `PARTITION BY` clause defines the maximal subsets, of what the subquery-level `WHERE` clause defines, that are operated upon, in turn, by a window function (or by an aggregate function in window mode). Tautologically, this maximal subset is referred to as the [_window_](./#the-window-definition-rule). In the limit, when the `PARTITION BY` clause is omitted, the maximal subset is identical with what the `WHERE` clause defines. -- The window `ORDER BY` clause defines how the rows are to be ordered within the [_window_](./#the-window-definition-rule). -- The [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause) defines a further narrowing of the [_window_](./#the-window-definition-rule), referred to as the [_window frame_](./#frame-clause-semantics-for-window-functions). The [_window frame_](./#frame-clause-semantics-for-window-functions) is anchored to the current row within the [_window_](./#the-window-definition-rule). In the degenerate case, the [_window frame_](./#frame-clause-semantics-for-window-functions) coincides with the [_window_](./#the-window-definition-rule) and is therefore insensitive to the position of the current row. - -In summary, the [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) defines the [_window_](./#the-window-definition-rule) as the scope within which a function's meaning (window function or aggregate function in window mode) is defined. The [_window_](./#the-window-definition-rule) is then further characterized by the ordering of its rows, the extent of the [_window frame_](./#frame-clause-semantics-for-window-functions), and how this moves with the current row. - -### The FILTER clause - -The `FILTER` clause's `WHERE` clause has the same syntax and semantics as it does at the regular `WHERE` clause syntax spot immediately after a subquery's `FROM` list. Notice that the `FILTER` clause is legal only for the invocation of an aggregate function. Here is an example: - -```plpgsql -select - class, - k, - count(*) - filter(where k%2 = 0) - over (partition by class) - as n -from t1 -order by class, k; -``` - -If you want to run this, then create a data set using the `ysqlsh` script that [table t1](../function-syntax-semantics/data-sets/table-t1/) presents. - -Using the `FILTER` clause in the invocation of a window function causes this compilation error: - -``` -0A000: FILTER is not implemented for non-aggregate window functions -``` - -### The PARTITION BY clause - -The `PARTITION BY` clause groups the rows that the subquery defines into [_windows_](./#the-window-definition-rule), which are processed separately by the window function. (This holds, too, when an aggregate function is invoked in this way.) It works similarly to a query-level `GROUP BY` clause, except that its expressions are always just expressions and cannot be output-column names or numbers. If the `PARTITION BY` clause is omitted, then all rows are treated as a single [_window_](./#the-window-definition-rule). - -### The window ORDER BY clause - -The window `ORDER BY` clause determines the order in which the rows of a [_window_](./#the-window-definition-rule) are processed by the window function. It works similarly to a query-level `ORDER BY` clause; but it cannot use output-column names or numbers. If the window `ORDER BY` clause is omitted, then rows are processed in an unspecified order so that the results of any window function invoked in this way would be unpredictable and therefore meaningless. Aggregation functions invoked in this way might be sensitive to what the window `ORDER BY` clause says. This will be the case when, for example, the [_window frame_](./#frame-clause-semantics-for-window-functions) is smaller than the whole [_window_](./#the-window-definition-rule) and moves with the current row. The section [Using the aggregate function avg() to compute a moving average](../functionality-overview/#using-the-aggregate-function-avg-to-compute-a-moving-average) provides an example. - -### The frame_clause - -The [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause) has many variants. Only one basic variant is needed in the `OVER` clause that you use to invoke a window function. The other variants are useful in the `OVER` clause that you use to invoke an aggregate function. For completeness, those variants are described on this page. - -#### frame_clause semantics for window functions - -The [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause) specifies the set of rows constituting the so-called [_window frame_](./#frame-clause-semantics-for-window-functions). In general, this will be a subset of the rows in the current [_window_](./#the-window-definition-rule). Look at the two tables at the end of the section [Signature and purpose of each window function](../function-syntax-semantics/). - -- The functions in the first group, [Window functions that return an "int" or "double precision" value as a "classifier" of the rank of the row within its window](../function-syntax-semantics/#window-functions-that-return-an-int-or-double-precision-value-as-a-classifier-of-the-rank-of-the-row-within-its-window), are not sensitive to what the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause) specifies and always use all of the rows in the current [_window_](./#the-window-definition-rule). Yugabyte recommends that you therefore omit the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause) in the `OVER` clause that you use to invoke these functions. - -- The functions in the second group, [Window functions that return columns of another row within the window](../function-syntax-semantics/#window-functions-that-return-column-s-of-another-row-within-the-window), make obvious sense when the scope within which the specified row is found is the entire [_window_](./#the-window-definition-rule). If you have one of the very rare use cases where the output that you want is produced by a different [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause), then specify what you want explicitly. Otherwise, because it isn't the default, you must specify that the [_window frame_](./#frame-clause-semantics-for-window-functions) includes the entire current [_window_](./#the-window-definition-rule) like this: - - ``` - range between unbounded preceding and unbounded following - ``` - -Use cases where the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause)'s many other variants are useful arise when an aggregate function is invoked using the `OVER` clause. One example is given in the section [Using the aggregate function `avg()` to compute a moving average](../#using-the-aggregate-function-avg-to-compute-a-moving-average). Another example, that uses `count(*)`, is given in the code that explains the meaning of the [`percent_rank()`](../function-syntax-semantics/percent-rank-cume-dist-ntile/#percent-rank) function. Otherwise, see the main [Aggregate functions](../../aggregate_functions/) section. - -#### frame_clause semantics for aggregate functions - -The [_window frame_](./#frame-clause-semantics-for-window-functions) can be specified in `RANGE`, `ROWS` or `GROUPS` mode; in each case, it runs from the [`frame_start`](../../../syntax_resources/grammar_diagrams/#frame-start) to the [`frame_end`](../../../syntax_resources/grammar_diagrams/#frame-end). If [`frame_end`](../../../syntax_resources/grammar_diagrams/#frame-end) is omitted, then the end defaults to `CURRENT ROW`. - -A [`frame_start`](../../../syntax_resources/grammar_diagrams/#frame-start) of `UNBOUNDED PRECEDING` means that the [_window frame_](./#frame-clause-semantics-for-window-functions) starts with the first row of the [_window_](./#the-window-definition-rule). Similarly, a [`frame_end`](../../../syntax_resources/grammar_diagrams/#frame-end) of `UNBOUNDED FOLLOWING` means that the [_window frame_](./#frame-clause-semantics-for-window-functions) ends with the last row of the [_window_](./#the-window-definition-rule). - -In `RANGE` or `GROUPS` mode, a [`frame_start`](../../../syntax_resources/grammar_diagrams/#frame-start) of `CURRENT ROW` means that the [_window frame_](./#frame-clause-semantics-for-window-functions) starts with the first member of the current row's _peer group_. A _peer group_ is a set of rows that the window `ORDER BY` clause sorts with the same rank as the current row. And a [`frame_end`](../../../syntax_resources/grammar_diagrams/#frame-end) of `CURRENT ROW` means that the [_window frame_](./#frame-clause-semantics-for-window-functions) ends with the last row in the current row's _peer group_. In `ROWS` mode, `CURRENT ROW` simply means the current row. - -For the [`offset`](../../../syntax_resources/grammar_diagrams/#offset) `PRECEDING` and [`offset`](../../../syntax_resources/grammar_diagrams/#offset) `FOLLOWING` modes of the [`frame_start`](../../../syntax_resources/grammar_diagrams/#frame-start) and [`frame_end`](../../../syntax_resources/grammar_diagrams/#frame-end) clauses, the [`offset`](../../../syntax_resources/grammar_diagrams/#offset) argument must be an expression that doesn't include any variables, aggregate functions, or window functions. The meaning of the [`offset`](../../../syntax_resources/grammar_diagrams/#offset) value depends on the `RANGE | ROWS | GROUPS` mode: - -- In `ROWS` mode, the [`offset`](../../../syntax_resources/grammar_diagrams/#offset) value must be a `NOT NULL`, non-negative integer. This brings the meaning that the [_window frame_](./#frame-clause-semantics-for-window-functions) starts or ends the specified number of rows before or after the current row. - -- In `GROUPS` mode, the [`offset`](../../../syntax_resources/grammar_diagrams/#offset) value must again be a `NOT NULL`, non-negative integer. Here, this brings the meaning that the [_window frame_](./#frame-clause-semantics-for-window-functions) starts or ends the specified number of _peer groups_ before or after the current row's _peer group_. Recall that there's always a logical requirement to include a window `ORDER BY` clause in the window definition that is used to invoke a window function. In `GROUPS` mode, whatever is your intended use of the window definition, you get this error if it doesn't include a window `ORDER BY` clause: - - ``` - 42P20: GROUPS mode requires an ORDER BY clause - ``` - -- In `RANGE` mode, these options require that the window `ORDER BY` clause specify exactly one column. The [`offset`](../../../syntax_resources/grammar_diagrams/#offset) value specifies the maximum difference between the value of that column in the current row and its value in the preceding or following rows of the [_window frame_](./#frame-clause-semantics-for-window-functions). The [`offset`](../../../syntax_resources/grammar_diagrams/#offset) expression must yield a value whose data type depends upon that of the ordering column. For numeric ordering columns (like `int`, `double precision`, and so on), it is typically of the same data type as the ordering column; but for date-time ordering columns it is an `interval`. For example, if the ordering column is `date` or `timestamp`, you could specify `RANGE BETWEEN '1 day' PRECEDING AND '10 days' FOLLOWING`. Here too, the [`offset`](../../../syntax_resources/grammar_diagrams/#offset) value must be `NOT NULL` and non-negative. The meaning of “non-negative” depends on the data type. - -In all cases, the distance to the start and end of the [_window frame_](./#frame-clause-semantics-for-window-functions) is limited by the distance to the start and end of the [_window_](./#the-window-definition-rule), so that for rows near the [_window_](./#the-window-definition-rule) boundaries, the [_window frame_](./#frame-clause-semantics-for-window-functions) might contain fewer rows than elsewhere. - -Notice that in both `ROWS` and `GROUPS` mode, `0 PRECEDING` and `0 FOLLOWING` is equivalent to `CURRENT ROW`. This normally holds in `RANGE` mode too, for an appropriate meaning of “zero” specific to the data type. - -The [`frame_exclusion`](../../../syntax_resources/grammar_diagrams/#frame-exclusion) clause allows rows around the current row to be excluded from the [_window frame_](./#frame-clause-semantics-for-window-functions), even if they would be included according to what the [`frame_start`](../../../syntax_resources/grammar_diagrams/#frame-start) and [`frame_end`](../../../syntax_resources/grammar_diagrams/#frame-end) clauses say. - -- `EXCLUDE CURRENT ROW` excludes the current row from the [_window frame_](./#frame-clause-semantics-for-window-functions). -- `EXCLUDE GROUP` excludes all the rows in the current row's _peer group_. -- `EXCLUDE TIES` excludes any peers of the current row, but not the current row itself. -- `EXCLUDE NO OTHERS` simply specifies explicitly the default behavior of not excluding the current row or its peers. - -Omitting the [`frame_clause`](../../../syntax_resources/grammar_diagrams/#frame-clause) is the same as specifying - -``` -RANGE UNBOUNDED PRECEDING -``` - -and this means the same as - -``` -RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW -``` - -If the window `ORDER BY` clause is specified, then this default meaning sets the [_window frame_](./#frame-clause-semantics-for-window-functions) to be all rows from the [_window_](./#the-window-definition-rule) start up through the last row in the current row's _peer group_. And if the window `ORDER BY` clause is omitted this means that all rows of the [_window_](./#the-window-definition-rule) are included in the [_window frame_](./#frame-clause-semantics-for-window-functions), because all rows become peers of the current row. - -**Notes:** - -- The [`frame_start`](../../../syntax_resources/grammar_diagrams/#frame-start) clause cannot be `UNBOUNDED FOLLOWING` -- The [`frame_end`](../../../syntax_resources/grammar_diagrams/#frame-end) clause cannot be `UNBOUNDED PRECEDING`, and cannot appear before the [`frame_start`](../../../syntax_resources/grammar_diagrams/#frame-start) clause. - -For example `RANGE BETWEEN CURRENT ROW AND offset PRECEDING` causes this error: - -``` -42P20: frame starting from current row cannot have preceding rows -``` - -However, `ROWS BETWEEN 7 PRECEDING AND 8 PRECEDING` _is_ allowed, even though it would never select any rows. - -If the `FILTER` clause is specified, then only the input rows for which it evaluates to true are fed to the window function; other rows are discarded. As noted above, only window aggregate functions invoked using the `OVER` clause accept a `FILTER` clause. - -## Examples - -### First example - -This shows the use of a window function with an `OVER` clause that directly specifies the [`window_definition`](../../../syntax_resources/grammar_diagrams/#window-definition) : - -``` -select - ... - some_window_function(...) over (partition by
**begin** _\
**exception** _\
**end;** | -| [case statement](./case-statement/) | [plpgsql_case_stmt](../../../../../syntax_resources/grammar_diagrams/#plpgsql-case-stmt) | _Simple form:_
**case** _\
**when** _\
**else** _\
**end case;**
_Searched form:_
**case**
**when** _\
**else** _\
**end case;** | -| [if statement](./if-statement/) | [plpgsql_if_stmt](../../../../../syntax_resources/grammar_diagrams/#plpgsql-if-stmt) | **if** _\
**elsif** _\
**else** _\
**end if;** | -| [loop statement](./loop-exit-continue/) | [plpgsql_loop_stmt](../../../../../syntax_resources/grammar_diagrams/#plpgsql-loop-stmt) | [unbounded loop](./loop-exit-continue/infinite-and-while-loops/) _or_ [bounded loop](./loop-exit-continue/#bounded-loop) | diff --git a/docs/content/preview/api/ysql/user-defined-subprograms-and-anon-blocks/language-plpgsql-subprograms/plpgsql-syntax-and-semantics/executable-section/compound-statements/loop-exit-continue/_index.md b/docs/content/preview/api/ysql/user-defined-subprograms-and-anon-blocks/language-plpgsql-subprograms/plpgsql-syntax-and-semantics/executable-section/compound-statements/loop-exit-continue/_index.md deleted file mode 100644 index 261bd0be2a7b..000000000000 --- a/docs/content/preview/api/ysql/user-defined-subprograms-and-anon-blocks/language-plpgsql-subprograms/plpgsql-syntax-and-semantics/executable-section/compound-statements/loop-exit-continue/_index.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: Loop, exit, and continue statements [YSQL] -headerTitle: Loop, exit, and continue statements -linkTitle: - The "loop", "exit", and "continue" statements -description: Describes the syntax and semantics of Loop, exit, and continue statements. [YSQL] -image: /images/section_icons/api/subsection.png -menu: - preview_api: - identifier: loop-exit-continue - parent: compound-statements - weight: 40 -type: indexpage -showRightNav: true ---- - -## Syntax - -### The "loop" statement - -{{%ebnf%}} - plpgsql_loop_stmt, - plpgsql_unbounded_loop_defn, - plpgsql_bounded_loop_defn, - plpgsql_integer_for_loop_defn, - plpgsql_array_foreach_loop_defn, - plpgsql_query_for_loop_defn, - plpgsql_dynamic_subquery -{{%/ebnf%}} - -### The "exit" and "continue" statements - -{{%ebnf%}} - plpgsql_exit_stmt, - plpgsql_continue_stmt -{{%/ebnf%}} - -## Semantics - -There are two kinds of PL/pgSQL loop: the _unbounded loop_; and the (bounded) _for loop_. - -- The number of iterations that an _unbounded loop_ performs isn't given by a simple recipe. Rather, iteration continues until it is interrupted, at any statement within its statement list, by invoking _exit_. -- In contrast, the (maximum) number of iterations that a _for loop_ performs is determined, before the first iteration starts. The recipe might, for example, be just the consecutive integers between a lower bound and an upper bound. Or it might be "for every element in the specified array", or "for every row in the result set of a specified query. - -The functionality of all kinds of loops is complemented by the _exit_ statement and the _continue_ statement. The _exit_ statement aborts the iteration altogether. And the _continue_ statement aborts just the current iteration and then starts the next one. - -See the section [Two case studies: Using various kinds of "loop" statement, the "exit" statement, and the "continue" statement](../loop-exit-continue/two-case-studies/) for realistic uses of all of the statements (except for the _while loop_) that this page describes. - -### Unbounded loop - -The name _unbounded_ denotes the fact that the number of iterations that the loop will complete is not announced at the start. Rather, iteration ends when facts that emerge while it executes determine that it's time to stop iterating. There are two kinds of _unbounded loop_: the _infinite loop_; and the _while loop_. See the dedicated page [The "infinite loop" and the "while loop"](./infinite-and-while-loops/). - -### "Exit" statement - -Most usually, the _exit_ statement is written within a _loop_ statement—but see the note immediately below. - -The _exit_ statement aborts the execution of the loop. Notice the optional _label_. It must match an _end loop_ statement (or the _end_ statement of a block statement) with the same _label_ within the current top-level block statement. - -- When the _label_ is omitted, the point of execution moves to the statement that immediately follows the present loop's _end loop_ statement. -- When _exit some_label_ is used, the point of execution moves to the statement that immediately follows the _end loop_ statement (or the bare _end_ statement of the block statement) that has the same label. - -{{< note title="An 'exit' statement's 'label' must match that of an 'end loop' statement or that of the 'end' statement of a block statement." >}} -See the dedicated section [Using the "exit" statement to jump out of a block statement](./exit-from-block-statememt/). -{{< /note >}} - -### "Continue" statement - -The _continue_ statement is legal only within a _loop_ statement. - -- When _label_ is omitted, the _continue_ statement causes the current iteration to be abandoned and the next one to start immediately. - -- When _label_ is specified, it causes the current iteration of the most tightly enclosing loop, and any loops in which it is nested through the one whose _end loop_ matches the _label_, to be abandoned. Then the next iteration of the loop whose _end loop_ matches the _label_ starts immediately. - -- If used, the _label_ must match that of an _end loop_ statement. - -It's possible to write the _exit_ or the _continue_ statement in one of the legs of an _if_ statement or a _case_ statement in the executable section, or even in the exception section, of a block statement at any nesting depth. Probably, in such a context, you'd omit the optional _when_ clause. - -### Bounded loop - -The name _bounded_ denotes the fact that the (maximum) number of iterations that the loop will complete is computed, just before the first iteration, on entry into the loop. The qualifier "maximum" is used because the _exit_ statement can be used to cause premature exit. - -There are three kinds of _bounded loop_ loop: - -- the _[integer for loop](./integer-for-loop/)_ — defined by the _plpgsql_integer_for_loop_defn_ syntax rule -- the _[array foreach loop](./array-foreach-loop/)_ — defined by the _plpgsql_array_foreach_loop_defn_ syntax rule -- The _[query for loop](./query-for-loop/)_ — defined by the _plpgsql_query_for_loop_defn_ syntax rule. diff --git a/docs/content/preview/api/ysql/user-defined-subprograms-and-anon-blocks/subprogram-attributes/_index.md b/docs/content/preview/api/ysql/user-defined-subprograms-and-anon-blocks/subprogram-attributes/_index.md deleted file mode 100644 index b639c1942867..000000000000 --- a/docs/content/preview/api/ysql/user-defined-subprograms-and-anon-blocks/subprogram-attributes/_index.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: Subprogram attributes [YSQL] -headerTitle: Subprogram attributes -linkTitle: Subprogram attributes -description: Describes and categorizes the various attributes that characterize user-defined functions and procedures [YSQL]. -image: /images/section_icons/api/subsection.png -menu: - preview_api: - identifier: subprogram-attributes - parent: user-defined-subprograms-and-anon-blocks - weight: 10 -aliases: -type: indexpage -showRightNav: true ---- - -The overall behavior of a user-defined function or procedure is determined by a set of characteristics that are defined with the _create [or replace]_ and _alter_ statements for each subprogram kind. - -- The following characteristics are singletons in the categorization scheme in that they may be set _only_ with _create [or replace]_ and the rules that specify how they are set are not spelled with "attribute": - - - The argument list (i.e. the name, the mode, the data type, and optionally a default expression for each argument). See the [arg_decl_with_dflt](../../../ysql/syntax_resources/grammar_diagrams/#arg-decl-with-dflt) rule. - - - And, just for a function, the return data type. - -- All of the other determining characteristics are known by the term of art _attribute_. This term is used in the names of rules in the [YSQL Grammar](../../syntax_resources/grammar_diagrams/) for distinct subcategories of attribute, grouped according to _when_ they may be set (only with _create [or replace]_, only with _alter_, or with both) and to _which kind_ of subprogram they apply (only to functions, or to both functions and procedures). - -You can see the names of all of these rules in the grammars for _create [or replace] function_, _create [or replace] procedure_, _alter function_, and _alter procedure_, below: - -{{%ebnf localrefs="alterable_fn_only_attribute,alterable_fn_and_proc_attribute,special_fn_and_proc_attribute,unalterable_fn_attribute,unalterable_proc_attribute" %}} - create_function, - create_procedure, - alter_function, - alter_procedure -{{%/ebnf%}} - -Here are the different attribute rules. - -## Unalterable subprogram attributes - -The _unalterable subprogram attributes_ can be set _only_ with the _create [or replace]_ statement. Each of _function_ and _procedure_ has its own _unalterable attributes_ rule. They share _language_ and _subprogram_implementation_. But the status _regular function_ or _window function_ is meaningless for a procedure. - -{{%ebnf%}} - unalterable_fn_attribute, - unalterable_proc_attribute -{{%/ebnf%}} - -{{< note title="This major section, so far, describes only user-defined subprograms and anonymous blocks that are implemented in SQL or PL/pgSQL." >}} -Further, it does not yet describe how to create user-defined window functions. -{{< /note >}} - - -## Special subprogram attributes - -The special subprogram attributes are set using a general syntax style with the _alter_ statements. - -{{%ebnf%}} - special_fn_and_proc_attribute -{{%/ebnf%}} - -The syntax diagram shows that if you want to change any of these attributes, then you must change them one at a time by issuing _alter_ repeatedly. - -The _schema_ and the _name_ of a subprogram are set using dedicated explicit syntax with _create [or replace]_. But the _owner_ cannot be explicitly set with _create [or replace]_; rather, a new subprogram's _owner_ is implicitly set to what the _[current_role](https://www.postgresql.org/docs/15/functions-info.html#FUNCTIONS-INFO-SESSION-TABLE)_ built-in function returns. (This will be what the _[session_user](https://www.postgresql.org/docs/15/functions-info.html#FUNCTIONS-INFO-SESSION-TABLE)_ built-in function returns if _create [or replace]_ is issued as a top-level SQL statement; and it will be the _owner_ of a _security definer_ subprogram that issues the SQL statement.) - -As it happens, and just for PostgreSQL-historical reasons, if you want to specify the _extension_ on which a new subprogram depends you can do this only by _first_ creating the subprogram and _then_ specifying the name of the _extension_ using the subprogram-specific _alter_ statement. - -See the section [The semantics of the "depends on extension" subprogram attribute](depends-on-extension-semantics/) for more information about this attribute. - -## Alterable subprogram attributes - -These attributes are common for both functions and procedures: - -{{%ebnf%}} - alterable_fn_and_proc_attribute -{{%/ebnf%}} - -See the subsection [Alterable subprogram attributes](./alterable-subprogram-attributes/) for the explanations of the _configuration parameter_ and _security_ attributes. - -## Alterable function-only attributes - -Notice that there are no procedure-specific alterable attributes. These attributes are specific to just functions: - -{{%ebnf%}} - alterable_fn_only_attribute -{{%/ebnf%}} - -See the subsection [Alterable function-only attributes](./alterable-function-only-attributes/) for the explanations of the _volatility_, _On NULL input_, _parallel_, _leakproof_, _cost_ and _rows_ attributes. diff --git a/docs/content/preview/api/ysql/user-defined-subprograms-and-anon-blocks/subprogram-attributes/alterable-function-only-attributes/_index.md b/docs/content/preview/api/ysql/user-defined-subprograms-and-anon-blocks/subprogram-attributes/alterable-function-only-attributes/_index.md deleted file mode 100644 index 37e22f2c075e..000000000000 --- a/docs/content/preview/api/ysql/user-defined-subprograms-and-anon-blocks/subprogram-attributes/alterable-function-only-attributes/_index.md +++ /dev/null @@ -1,384 +0,0 @@ ---- -title: Alterable function-only attributes [YSQL] -headerTitle: Alterable function-only attributes -linkTitle: Alterable function-only attributes -description: Describes and categorizes the various attributes that characterize just user-defined functions [YSQL]. -image: /images/section_icons/api/subsection.png -menu: - preview_api: - identifier: alterable-function-only-attributes - parent: subprogram-attributes - weight: 30 -aliases: -type: indexpage -showRightNav: true ---- - -## Volatility - -The _[volatility](../../../syntax_resources/grammar_diagrams/#volatility)_ attribute has these allowed values: - -- _volatile_ -- _stable_ -- _immutable_ - -The default is _volatile_. - -This attribute allows the function's author to state a promise about the timespan over which a given (set of) actual arguments uniquely determines the function's return value. According to what is promised, PostgreSQL (and therefore YSQL) is allowed to cache some number of _"return-value-for-actual-arguments"_ pairs—in pursuit of improving performance. If caching is done, the scope is a single session and the duration is limited to the session's lifetime. - -Notice that _"is allowed to cache"_ does not mean _"will cache"_. The mere possibility is critical to the definition of the semantics of volatility. The conditions that make caching more, or less, likely are a separable concern. - -Tautologically, PostgreSQL (and therefore YSQL) is unable to detect if the promise is good—so an author who gives a false promise is asking for wrong results. - - -{{< tip title="See the PostgreSQL documentation for more detail." >}} -The section [38.7. Function Volatility Categories](https://www.postgresql.org/docs/15/xfunc-volatility.html) explains some of the more subtle aspects of function _volatility_. In particular, it makes this recommendation: - -> For best optimization results, you should label your functions with the strictest volatility category that is valid for them. - -The section [43.11.2. Plan Caching](https://www.postgresql.org/docs/15/plpgsql-implementation.html#PLPGSQL-PLAN-CACHING) explains the caching mechanism and how it is tied to the notion of _prepare_ and the possibility that a prepared statement might (or might not) cache its execution plan. -{{< /tip >}} - -### volatile - -This denotes no promise at all. Here's a compelling demonstration of a function where _volatile_ is the only possible honest choice: - -```plpgsql -drop function if exists volatile_result() cascade; - -create function volatile_result() - returns text - volatile - language sql -as $body$ - select gen_random_uuid()::text; -$body$; - -select volatile_result() as v1, volatile_result() as v2; -``` - -This is a typical result: - -```output - v1 | v2 ---------------------------------------+-------------------------------------- - 0869edfa-af63-4308-8b80-d9d5a58491b8 | 55589cf4-5441-4582-9cf1-688481b37f55 -``` - -The function _volatile_result()_ returns different results upon successive evaluations even during the execution of a single SQL statement. - -### stable - -This denotes a promise that holds good just for the duration of a SQL statement's execution. The human analyst can readily see that this following function can honestly be marked as _stable_: - -```plpgsql -drop function if exists stable_result(text) cascade; - -create function stable_result(which in text) - returns text - stable - language sql -as $body$ - select - case - when which = 'a' then current_setting('x.a') - when which = 'b' then current_setting('x.b') - end; -$body$; -``` - -The scope of a user-defined session parameter like _"x.a"_ is just the single session that sets it. And, in the example, the human can readily see that the code of _stable_result()_ doesn't set _"a.x"_ or _"x.b"_ — and that no other route exists to setting it from elsewhere while _stable_result()_ is executing. Test it like this: - -```plpgsql -set x.a = 'dog'; -set x.b = 'cat'; - -select stable_result('a'), stable_result('b'); -``` - -Clearly, the return values from _stable_result()_ can be different when it's invoked in a new SQL statement thus: - -```plpgsql -set x.a = 'frog'; -set x.b = 'bird'; - -select stable_result('a'), stable_result('b'); -``` - -### immutable - -When you mark a function as _immutable_, you give permission for PostgreSQL (and therefore YSQL) to build a session-duration cache where the key to a cache-entry is the vector of actual arguments with which the function is invoked and the key's payload is the function's return value. - -(The caching mechanism is the prepared statement and the possibility to cache an execution plan with it. But you needn't understand the mechanism in order to understand the semantic proposition.) - -Further, if you want to create an expression-based index that references a user-defined function, then it _must_ be marked _immutable_. Without this volatility setting, you get this error: - -```output -42P17: functions in index expression must be marked IMMUTABLE -``` - -Marking a function as _immutable_ expresses a promise that must hold good for the lifetime of the function's existence (in other words, from the moment it's created to the moment that it's dropped) thus: - -- The function has no side effects. -- The function is mathematically deterministic—that is, the vector of actual arguments uniquely determines the function's return value. - -Nothing prevents you from lying. But doing so will, sooner or later, bring wrong results. - -See the section [Immutable function examples](immutable-function-examples/). - -## On_null_input - -The _[on_null_input](../../../syntax_resources/grammar_diagrams/#on-null-input)_ attribute has these allowed values: - -- _called on null input_ -- _strict_ -- _returns null on null input_ - -The default is _called on null input_. Notice that _strict_ is simply a synonym for _returns null on null input_. - -### called on null input - -This allows the function to be executed just as its source code specifies when at least one of its actual arguments is _null_. Function authors must then take the responsibility for handling the case that any actual is _null_ appropriately. - -### strict - -This instructs YSQL simply to skip executing the function's source code when at least one of its actual arguments is _null_ and simply to return _null_ immediately. - -Try this: - -```plpgsql -\pset null '
| - | Snapshot isolation write | -Serializable write | -Serializable read | -
|---|---|---|---|
| Snapshot isolation write | -✘ Conflict | -✘ Conflict | -✘ Conflict | -
| Serializable write | -✘ Conflict | -✔ No conflict | -✘ Conflict | -
| Serializable read | -✘ Conflict | -✘ Conflict | -✔ No conflict | -
| - | Strong Snapshot isolation write | -Weak Snapshot isolation write | -Strong Serializable write | -Weak Serializable write | -Strong Serializable read | -Weak Serializable read | -
|---|---|---|---|---|---|---|
| Strong Snapshot isolation write | -✘ Conflict | -✘ Conflict | -✘ Conflict | -✘ Conflict | -✘ Conflict | -✘ Conflict | -
| Weak Snapshot isolation write | -✘ Conflict | -✔ No conflict | -✘ Conflict | -✔ No conflict | -✘ Conflict | -✔ No conflict | -
| Strong Serializable write | -✘ Conflict | -✘ Conflict | -✔ No conflict | -✔ No conflict | -✘ Conflict | -✘ Conflict | -
| Weak Serializable write | -✘ Conflict | -✔ No conflict | -✔ No conflict | -✔ No conflict | -✘ Conflict | -✔ No conflict | -
| Strong Serializable read | -✘ Conflict | -✘ Conflict | -✘ Conflict | -✘ Conflict | -✔ No conflict | -✔ No conflict | -
| Weak Serializable read | -✘ Conflict | -✔ No conflict | -✘ Conflict | -✔ No conflict | -✔ No conflict | -✔ No conflict | -
- | Feature | -Local SSD storage | -Remote SSD storage | -
|---|---|---|
| Provision large disk capacity per node | -Depends on cloud-provider | -Yes | -
| Disk storage resilient to failures or pod movement | -No | -Yes | -
| Performance - latency | -Lower | -Higher | -
| Performance - throughput | -Higher | -Lower | -
| Typical cost characteristics | -Lower | -Higher | -
| Kubernetes provisioning scheme | -Pre-provisioned | -Dynamic provisioning | -