Add dense_rank

Related to #710

Author: Yomguithereal

CHANGELOG.md

@@ -18,6 +18,7 @@
 * Adding `aarch64-apple-darwin` and `aarch64-unknown-linux-gnu` to CI builds.
 * Adding `to_fixed` moonblade function.
 * Adding decimal places optional argument to `ratio` & `percentage` aggregation functions.
+* Adding `frac` & `dense_rank` aggregation functions to `xan window`.
 
 *Fixes*
 

docs/cmd/window.md

@@ -3,12 +3,16 @@
 
 ```txt
 Compute window aggregations such as cumulative sums, rolling means, leading and
-lagging values etc.
+lagging values, rankings etc.
 
 This command is able to compute multiple aggregations in a single pass over the
 file, and never uses more memory that required to fit the largest desired window
 for rolling stats and leads/lags.
 
+Ranking aggregations however (such as `frac` or `dense_rank`), still require to
+buffer the whole file in memory (or at least whole groups when using -g/--groupby),
+since they cannot be computed otherwise.
+
 Computing a cumulative sum:
 
     $ xan window 'cumsum(n)' file.csv
@@ -21,6 +25,14 @@ Adding a lagged column:
 
     $ xan window 'lag(n) as "n-1"' file.csv
 
+Ranking numerical values:
+
+    $ xan window 'dense_rank(n) as rank' file.csv
+
+Computing fraction of cell wrt total sum of target column:
+
+    $ xan window 'frac(n) as frac' file.csv
+
 This command is also able to reset the statistics each time a new contiguous group
 of rows is encountered using the -g/--groupby flag. This means, however, that
 the file must be sorted by columns representing group identities beforehand:

docs/moonblade/aggs.md

@@ -48,12 +48,12 @@ the number of nodes in a graph represented by a CSV edge list.
 - **mode**(*\<expr\>*) -> `string`: Value appearing the most, breaking ties arbitrarily in favor of the first value in lexicographical order.
 - **most_common**(*k*, *\<expr\>*, *separator?*) -> `string`: List of top k most common values returned by expression joined by a pipe character ('|') or by the provided separator. Ties will be broken by lexicographical order.
 - **most_common_counts**(*k*, *\<expr\>*, *separator?*) -> `string`: List of top k most common counts returned by expression joined by a pipe character ('|') or by the provided separator. Ties will be broken by lexicographical order.
-- **percentage**(*\<expr\>*) -> `string`: Return the percentage of truthy values returned by expression.
+- **percentage**(*\<expr\>*, *decimals?*) -> `string`: Return the percentage of truthy values returned by expression, up to an optional number of decimal places.
 - **quantile**(*\<expr\>*, *q*) -> `number`: Return the desired quantile of numerical values.
 - **q1**(*\<expr\>*) -> `number`: Return the first quartile of numerical values.
 - **q2**(*\<expr\>*) -> `number`: Return the second quartile of numerical values.
 - **q3**(*\<expr\>*) -> `number`: Return the third quartile of numerical values.
-- **ratio**(*\<expr\>*) -> `number`: Return the ratio of truthy values returned by expression.
+- **ratio**(*\<expr\>*, *decimals?*) -> `number`: Return the ratio of truthy values returned by expression, up to an optional number of decimal places.
 - **rms**(*\<expr\>*) -> `number`: Return the Root Mean Square of numerical values.
 - **stddev**(*\<expr\>*) -> `number`: Population standard deviation. Same as `stddev_pop`.
 - **stddev_pop**(*\<expr\>*) -> `number`: Population standard deviation. Same as `stddev`.

docs/moonblade/window.md

@@ -3,6 +3,8 @@
 - **cummax**(*\<expr\>*) -> `number`: Returns the cumulative maximum of the numbers yielded by given expression.
 - **cummin**(*\<expr\>*) -> `number`: Returns the cumulative minimum of the numbers yielded by given expression.
 - **cumsum**(*\<expr\>*) -> `number`: Returns the cumulative sum of the numbers yielded by given expression.
+- **dense_rank**(*\<expr\>*) -> `number`: Returns the dense rank (there will be no gaps, but ties remain possible for a same rank) of numbers yielded by given expression. Beware, as this requires buffering whole file or group.
+- **frac**(*\<expr\>*, *decimals?*) -> `number`: Returns the fraction represented by numbers yielded by given expression over the total sum of them. Beware, as this requires buffering whole file or group.
 - **lag**(*\<expr\>*, *steps?*, *\<expr\>?*) -> `any`: Returns a value yielded by given expression, lagged by n steps or 1 step by default. Can take a second expression after the number of steps to return a default value for rows that come before first lagged value.
 - **lead**(*\<expr\>*, *steps?*, *\<expr\>?*) -> `any`: Returns a value yielded by given expression, leading by n steps or 1 step by default. Can take a second expression after the number of steps to return a default value for rows that come after last lead value.
 - **rolling_avg**(*window_size*, *\<expr\>*) -> `number`: Returns the rolling average in given window size of numbers yielded by given expression. Same as `rolling_mean`.

src/cmd/window.rs

@@ -6,12 +6,16 @@ use crate::CliResult;
 
 static USAGE: &str = "
 Compute window aggregations such as cumulative sums, rolling means, leading and
-lagging values etc.
+lagging values, rankings etc.
 
 This command is able to compute multiple aggregations in a single pass over the
 file, and never uses more memory that required to fit the largest desired window
 for rolling stats and leads/lags.
 
+Ranking aggregations however (such as `frac` or `dense_rank`), still require to
+buffer the whole file in memory (or at least whole groups when using -g/--groupby),
+since they cannot be computed otherwise.
+
 Computing a cumulative sum:
 
     $ xan window 'cumsum(n)' file.csv
@@ -24,6 +28,14 @@ Adding a lagged column:
 
     $ xan window 'lag(n) as \"n-1\"' file.csv
 
+Ranking numerical values:
+
+    $ xan window 'dense_rank(n) as rank' file.csv
+
+Computing fraction of cell wrt total sum of target column:
+
+    $ xan window 'frac(n) as frac' file.csv
+
 This command is also able to reset the statistics each time a new contiguous group
 of rows is encountered using the -g/--groupby flag. This means, however, that
 the file must be sorted by columns representing group identities beforehand:

src/moonblade/agg/window.rs

@@ -121,6 +121,7 @@ enum ConcreteWindowAggregation {
     RollingSum(ConcreteExpr, RollingSum),
     RollingWelford(ConcreteExpr, WelfordStat, RollingWelford),
     Frac(ConcreteExpr, Sum, Option<usize>),
+    DenseRank(ConcreteExpr, Vec<(DynamicNumber, usize)>, VecDeque<usize>),
 }
 
 fn eval_expression_to_number(
@@ -144,7 +145,7 @@ impl ConcreteWindowAggregation {
     }
 
     fn requires_total_buffer(&self) -> bool {
-        matches!(self, Self::Frac(_, _, _))
+        matches!(self, Self::Frac(_, _, _) | Self::DenseRank(_, _, _))
     }
 
     fn aggregate_total(
@@ -161,12 +162,46 @@ impl ConcreteWindowAggregation {
                     sum.add(value.try_as_number().map_err(|err| err.specify("frac"))?);
                 }
             }
+            Self::DenseRank(expr, numbers, _) => {
+                let value = eval_expression(expr, Some(index), record, headers_index)?;
+                let number = value
+                    .try_as_number()
+                    .map_err(|err| err.specify("dense_rank"))?;
+
+                numbers.push((number, numbers.len()));
+            }
             _ => (),
         };
 
         Ok(())
     }
 
+    fn finalize_total(&mut self) {
+        if let Self::DenseRank(_, numbers, ranks) = self {
+            numbers.sort();
+            ranks.resize(numbers.len(), 0);
+
+            let mut rank: usize = 0;
+            let mut last_number: Option<DynamicNumber> = None;
+
+            for (n, i) in numbers.iter() {
+                match last_number {
+                    None => {
+                        last_number = Some(*n);
+                        rank += 1;
+                    }
+                    Some(l) if l != *n => {
+                        last_number = Some(*n);
+                        rank += 1;
+                    }
+                    _ => {}
+                };
+
+                ranks[*i] = rank;
+            }
+        }
+    }
+
     fn run(
         &mut self,
         index: usize,
@@ -277,6 +312,7 @@ impl ConcreteWindowAggregation {
                     Some(d) => DynamicValue::from(frac.map(|f| format!("{:.p$}", f, p = d))),
                 })
             }
+            Self::DenseRank(_, _, ranks) => Ok(DynamicValue::from(ranks.pop_front().unwrap())),
         }
     }
 
@@ -304,6 +340,10 @@ impl ConcreteWindowAggregation {
             Self::Frac(_, sum, _) => {
                 sum.clear();
             }
+            Self::DenseRank(_, numbers, ranks) => {
+                numbers.clear();
+                ranks.clear();
+            }
         };
     }
 }
@@ -313,7 +353,7 @@ fn get_function(name: &str) -> Option<FunctionArguments> {
         "row_number" | "row_index" => FunctionArguments::nullary(),
         "frac" => FunctionArguments::with_range(1..=2),
         "lag" | "lead" => FunctionArguments::with_range(1..=3),
-        "cumsum" | "cummin" | "cummax" => FunctionArguments::unary(),
+        "cumsum" | "cummin" | "cummax" | "dense_rank" => FunctionArguments::unary(),
         "rolling_sum" | "rolling_mean" | "rolling_avg" | "rolling_var" | "rolling_stddev" => {
             FunctionArguments::binary()
         }
@@ -461,6 +501,14 @@ fn concretize_window_aggregations(
                     ConcreteWindowAggregation::Frac(expr, Sum::new(), decimals),
                 ));
             }
+            "dense_rank" => {
+                let expr = concretize_expression(agg.args.pop().unwrap(), headers, None)?;
+
+                concrete_aggs.push((
+                    agg.agg_name,
+                    ConcreteWindowAggregation::DenseRank(expr, Vec::new(), VecDeque::new()),
+                ));
+            }
             _ => unreachable!(),
         };
     }
@@ -607,8 +655,6 @@ impl WindowAggregationProgram {
         self.run_with_record_impl(index, record, false)
     }
 
-    // TODO: beware alignement issues, frac should work with holes, for instance
-    // TODO: avoid double expression evaluation (beware of unalignement when a value cannot hold)
     pub fn flush<F, E>(&mut self, mut from_index: usize, mut callback: F) -> Result<(), E>
     where
         F: FnMut(ByteRecord) -> Result<(), E>,
@@ -621,6 +667,10 @@ impl WindowAggregationProgram {
                 }
             }
 
+            for (_, agg) in self.aggs.iter_mut() {
+                agg.finalize_total();
+            }
+
             for (index, record) in total_buffer.iter() {
                 if let Some(output_record) = self.run_with_record(*index, record)? {
                     callback(output_record)?;
@@ -650,7 +700,7 @@ impl WindowAggregationProgram {
         F: FnMut(ByteRecord) -> Result<(), E>,
         E: From<SpecifiedEvaluationError>,
     {
-        let records = self.flush(from_index, callback)?;
+        self.flush(from_index, callback)?;
 
         if self.aggs.iter().any(|(_, agg)| agg.requires_total_buffer()) {
             self.total_buffer = Some(Vec::new());
@@ -668,6 +718,6 @@ impl WindowAggregationProgram {
             agg.clear();
         }
 
-        Ok(records)
+        Ok(())
     }
 }

src/moonblade/doc/window.json

@@ -17,6 +17,18 @@
     "returns": "number",
     "help": "Returns the cumulative sum of the numbers yielded by given expression."
   },
+  {
+    "name": "dense_rank",
+    "arguments": ["<expr>"],
+    "returns": "number",
+    "help": "Returns the dense rank (there will be no gaps, but ties remain possible for a same rank) of numbers yielded by given expression. Beware, as this requires buffering whole file or group."
+  },
+  {
+    "name": "frac",
+    "arguments": ["<expr>", "decimals?"],
+    "returns": "number",
+    "help": "Returns the fraction represented by numbers yielded by given expression over the total sum of them. Beware, as this requires buffering whole file or group."
+  },
   {
     "name": "lag",
     "arguments": ["<expr>", "steps?", "<expr>?"],

tests/test_window.rs

@@ -419,3 +419,37 @@ fn window_frac() {
 
     assert_eq!(got, expected);
 }
+
+#[test]
+fn window_dense_rank() {
+    let wrk = Workdir::new("window_dense_rank");
+    wrk.create(
+        "numbers.csv",
+        vec![
+            svec!["n"],
+            svec!["20"],
+            svec!["10"],
+            svec!["30"],
+            svec!["10"],
+            svec!["20"],
+            svec!["20"],
+            svec!["20"],
+        ],
+    );
+    let mut cmd = wrk.command("window");
+    cmd.arg("dense_rank(n) as rank").arg("numbers.csv");
+
+    let got: Vec<Vec<String>> = wrk.read_stdout(&mut cmd);
+    let expected = vec![
+        svec!["n", "rank"],
+        svec!["20", "2"],
+        svec!["10", "1"],
+        svec!["30", "3"],
+        svec!["10", "1"],
+        svec!["20", "2"],
+        svec!["20", "2"],
+        svec!["20", "2"],
+    ];
+
+    assert_eq!(got, expected);
+}