PARQUET-2249: Introduce IEEE 754 total order

This commit adds a new column order `IEEE754TotalOrder`, which can be
used for floating point types (`FLOAT`, `DOUBLE`, `FLOAT16`).

The advantage of the new order is a well-defined ordering between -0,+0
and the various possible bit patterns of NaNs. Thus, every single
possible bit pattern of a floating point value has a well-defined order
now, so there are no possibilities where two implementations might
apply different orders when the new column order is used.

With the default column order, there were many problems w.r.t. NaN
values which lead to reading engines not being able to use statistics
of floating point columns for scan pruning even in the case where no
NaNs were in the data set. The problems are discussed in detail in the
next section.

This solution to the problem is the result of the extended discussion
in PR apache#196 [1], which ended with the consensus that IEEE 754 total
ordering is the best approach to solve the problem in a simple manner
without introducing special fields for floating point columns (such as
`nan_counts`, which was proposed in that PR). Please refer to the
discussion in that PR for all the details why this solution was chosen
over various design alternatives.

Note that this solution is fully backward compatible and should not
break neither old readers nor writers, as a new column order is added.
Legacy writers can continue not writing this new order and instead
writing the default type defined order. Legacy readers should avoid
using any statistics on columns that have a column order they do not
understand and therefore should just not use the statistics for columns
ordered using the new order.

The remainder of this message explains in detail what the problems are
and how the proposed solution fixes them.

Problem Description
===================

Currently, the way NaN values are to be handled in statistics inhibits
most scan pruning once NaN values are present in `DOUBLE` or `FLOAT`
columns. Concretely the following problems exist:

Statistics don't tell whether NaNs are present
----------------------------------------------

As NaN values are not to be incorporated in min/max bounds, a reader
cannot know whether NaN values are present. This might seem to be not
too problematic, as most queries will not filter for NaNs. However, NaN
is ordered in most database systems. For example, Postgres, DB2, and
Oracle treat NaN as greater than any other value, while MSSQL and MySQL
treat it as less than any other value. An overview over what different
systems are doing can be found here [2]. The gist of it is that
different systems with different semantics exist w.r.t. NaNs and most
of the systems do order NaNs; either less than or greater than all
other values.

For example, if the semantics of the reading query engine mandate that
NaN is to be treated greater than all other values, the predicate
`x > 1.0` should include NaN values. If a page has max = 0.0 now, the
engine would not be able to skip the page, as the page might contain
NaNs which would need to be included in the query result.

Likewise, the predicate `x < 1.0` should include NaN if NaN is treated
to be less than all other values by the reading engine. Again, a page
with min = 2.0 couldn't be skipped in this case by the reader.

Thus, even if a user doesn't query for NaN explicitly, they might use
other predictes that need to filter or retain NaNs in the semantics of
the reading engine, so the fact that we currently can't know whether a
page or row group contains NaN is a bigger problem than it might seem
on first sight.

Currently, any predicate that needs to retain NaNs cannot use min and
max bounds in Parquet and therefore cannot be used for scan pruning at
all. And as state, that can be many seemingly innocuous greater than or
less than predicates in most databases systems. Conversely, it would be
nice if Parquet would enable scan pruning in these cases, regardless of
whether the reader and writer agree upon whether NaN is smaller,
greater, or incomparable to all other values.

Note that the problem exists especially if the Parquet file doesn't
include any NaNs, so this is not only a problem in the edge case where
NaNs are present; it is a problem in the way more common case of NaNs
not being present.

Handling NaNs in a ColumnIndex
------------------------------
There is currently no well-defined way to write a spec-conforming
`ColumnIndex` once a page has only NaN (and possibly null) values. NaN
values should not be included in min/max bounds, but if a page contains
only NaN values, then there is no other value to put into the min/max
bounds. However, bounds in a ColumnIndex are non-optional, so we have
to put something in here. The spec does not describe what engines
should do in this case. Parquet-mr takes the safe route and does not
write a column index once NaNs are present. But this is a huge
pessimization, as a single page containing NaNs will prevent writing a
column index for the column chunk containing that page, so even pages
in that chunk that don't contain NaNs will not be indexed.

It would be nice if there was a defined way of writing the ColumnIndex
when NaNs (and especially only-NaN pages) are present.

Handling only-NaN pages & column chunks
---------------------------------------

Note: Hereinafter, whenever the term only-NaN is used, it refers to a
page or column chunk, whose only non-null values are NaNs. E.g., an
only-NaN page is allowed to have a mixture of null values and NaNs or
only NaNs, but no non-NaN non-null values.

The `Statistics` objects stored in page headers and in the file footer
have a similar, albeit smaller problem: `min_value` and `max_value` are
optional here, so it is easier to not include NaNs in the min/max in
case of an only-NaN page or column chunk: Simply omit these optional
fields. However, this brings a semantic ambiguity with it, as it is
now unclear whether the min/max value wasn't written because there were
only NaNs, or simply because the writing engine did decide to omit them
for whatever other reason, which is allowed by the spec as the field is
optional.

Consequently, a reader cannot know whether missing `min_value` and
`max_value` means "only NaNs, I can skip this page if I am looking for
only non-NaN values" or "no stats written, you have to read this page
as it is undefined what values it contains".

It would be nice if we could handle NaNs in a way that would allow scan
pruning for these only-NaN pages.

Solution
========

IEEE 754 total order solves all the mentioned problems. As NaNs now
have a defined place in the ordering, they can be incorporated into min
and max bounds. In fact, in contrast to the default ordering, they do
not need any special casing anymore, so all the remarks how readers and
writers should special-handle NaNs and -0/+0 no longer apply to the new
ordering.

As NaNs are incorporated into min and max, a reader can now see whether
NaNs are contained through the statistics. Thus, a reading engine just
has to map its NaN semantics to the NaN semantics of total ordering.
For example, if the semantics of the reading engine treat all NaNs
(also -NaNs) as greater than all other values, a reading engine having
a predicate `x > 5.0` (which should include NaNs) may not filter any
pages / row groups if either min or max are (+/-)NaN.

Only-NaN pages can now also be included in the column index, as they
are no longer a special case.

[1]
apache#196
[2]
apache/arrow-rs#264 (comment)

Author: JFinis

LogicalTypes.md

@@ -254,7 +254,10 @@ Used in contexts where precision is traded off for smaller footprint and potenti
 
 The primitive type is a 2-byte `FIXED_LEN_BYTE_ARRAY`.
 
-The sort order for `FLOAT16` is signed (with special handling of NANs and signed zeros); it uses the same [logic](https://github.com/apache/parquet-format#sort-order) as `FLOAT` and `DOUBLE`.
+The type-defined sort order for `FLOAT16` is signed (with special handling of NaNs and signed zeros),
+as for `FLOAT` and `DOUBLE`. It is recommended that writers use IEEE754TotalOrder when writing columns
+of this type for a well-defined handling of NaNs and signed zeros. See the `ColumnOrder` union in the
+[Thrift definition](src/main/thrift/parquet.thrift) for details.
 
 ## Temporal Types
 

README.md

@@ -158,7 +158,9 @@ documented in [LogicalTypes.md][logical-types].
 Parquet stores min/max statistics at several levels (such as Column Chunk,
 Column Index, and Data Page). These statistics are according to a sort order,
 which is defined for each column in the file footer. Parquet supports common
-sort orders for logical and primitve types. The details are documented in the
+sort orders for logical and primitve types and also special orders for types
+where the common sort order is not unambiguously defined (e.g., NaN ordering
+for floating point types). The details are documented in the
 [Thrift definition](src/main/thrift/parquet.thrift) in the `ColumnOrder` union.
 
 ## Nested Encoding

src/main/thrift/parquet.thrift

@@ -1030,6 +1030,9 @@ struct RowGroup {
 /** Empty struct to signal the order defined by the physical or logical type */
 struct TypeDefinedOrder {}
 
+/** Empty struct to signal IEEE 754 total order for floating point types */
+struct IEEE754TotalOrder {}
+
 /**
  * Union to specify the order used for the min_value and max_value fields for a
  * column. This union takes the role of an enhanced enum that allows rich
@@ -1038,6 +1041,7 @@ struct TypeDefinedOrder {}
  * Possible values are:
  * * TypeDefinedOrder - the column uses the order defined by its logical or
  *                      physical type (if there is no logical type).
+ * * IEEE754TotalOrder - the floating point column uses IEEE 754 total order.
  *
  * If the reader does not support the value of this union, min and max stats
  * for this column should be ignored.
@@ -1082,8 +1086,12 @@ union ColumnOrder {
    *   BYTE_ARRAY - unsigned byte-wise comparison
    *   FIXED_LEN_BYTE_ARRAY - unsigned byte-wise comparison
    *
-   * (*) Because the sorting order is not specified properly for floating
-   *     point values (relations vs. total ordering) the following
+   * (*) Because the precise sorting order is ambiguous for floating
+   *     point types due to underspecified handling of NaN and -0/+0,
+   *     it is recommended that writers use IEEE_754_TOTAL_ORDER
+   *     for these types.
+   *
+   *     If this ordering is used for floating point types, then the following
    *     compatibility rules should be applied when reading statistics:
    *     - If the min is a NaN, it should be ignored.
    *     - If the max is a NaN, it should be ignored.
@@ -1099,6 +1107,53 @@ union ColumnOrder {
    *       `-0.0` should be written into the min statistics field.
    */
   1: TypeDefinedOrder TYPE_ORDER;
+
+  /*
+   * The floating point type is ordered according to the totalOrder predicate,
+   * as defined in section 5.10 of IEEE-754 (2008 revision). Only columns of
+   * physical type FLOAT or DOUBLE, or logical type FLOAT16 may use this ordering.
+   *
+   * Intuitively, this orders floats mathematically, but defines -0 to be less
+   * than +0, -NaN to be less than anything else, and +NaN to be greater than
+   * anything else. It also defines an order between different bit representations
+   * of the same value.
+   *
+   * The formal definition is as follows:
+   *   a) If x<y, totalOrder(x, y) is true.
+   *   b) If x>y, totalOrder(x, y) is false.
+   *   c) If x=y:
+   *     1) totalOrder(−0, +0) is true.
+   *     2) totalOrder(+0, −0) is false.
+   *     3) If x and y represent the same floating-point datum:
+   *        i) If x and y have negative sign, totalOrder(x, y) is true if and
+   *           only if the exponent of x ≥ the exponent of y
+   *       ii) otherwise totalOrder(x, y) is true if and only if the exponent
+   *           of x ≤ the exponent of y.
+   *   d) If x and y are unordered numerically because x or y is NaN:
+   *     1) totalOrder(−NaN, y) is true where −NaN represents a NaN with
+   *        negative sign bit and y is a non-NaN floating-point number.
+   *     2) totalOrder(x, +NaN) is true where +NaN represents a NaN with
+   *        positive sign bit and x is a non-NaN floating-point number.
+   *     3) If x and y are both NaNs, then totalOrder reflects a total ordering
+   *        based on:
+   *         i) negative sign orders below positive sign
+   *        ii) signaling orders below quiet for +NaN, reverse for −NaN
+   *       iii) lesser payload, when regarded as an integer, orders below
+   *            greater payload for +NaN, reverse for −NaN.
+   *
+   * Note that this ordering can be implemented efficiently in software by bit-wise
+   * operations on the integer representation of the floating point values.
+   * E.g., this is a possible implementation for DOUBLE in Rust:
+   *
+   *   pub fn totalOrder(x: f64, y: f64) -> bool {
+   *     let mut x_int = x.to_bits() as i64;
+   *     let mut y_int = y.to_bits() as i64;
+   *     x_int ^= (((x_int >> 63) as u64) >> 1) as i64;
+   *     y_int ^= (((y_int >> 63) as u64) >> 1) as i64;
+   *     return x_int <= y_int;
+   *   }
+   */
+  2: IEEE754TotalOrder IEEE_754_TOTAL_ORDER;
 }
 
 struct PageLocation {