549 implement additional deterministic and probabilistic metrics (#571)

* stash changes

* stashing changes again

* Implement conditional deterministic metrics

* addresses PR feedback, adds documentation for cat. metrics

Author: samland1116

docs/sphinx/user_guide/notebooks/08_adding_calculated_fields.ipynb

@@ -320,7 +320,7 @@
     "- DayOfYear\n",
     "\n",
     "Timeseries Aware Calculated Fields.\n",
-    "- PercentileEventDetection\n",
+    "- AbovePercentileEventDetection\n",
     "\n",
     "There will be more added over time.  If there is one you are particularly interested in, please reach out and let us know."
    ]
@@ -541,7 +541,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": null,
    "metadata": {},
    "outputs": [
     {
@@ -554,7 +554,7 @@
    ],
    "source": [
     "ev.joined_timeseries.add_calculated_fields([\n",
-    "    tcf.PercentileEventDetection()\n",
+    "    tcf.AbovePercentileEventDetection()\n",
     "]).write()"
    ]
   },
@@ -749,23 +749,23 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "pdf = ev.joined_timeseries.filter([\n",
     "    \"primary_location_id = 'usgs-14138800'\",\n",
-    "    \"event = true\",\n",
+    "    \"event_above = true\",\n",
     "]).to_pandas()"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "primary_plot = pdf.hvplot.points(x=\"value_time\", y=\"primary_value\", color=\"event_id\") #.opts(width=1200, height=400)"
+    "primary_plot = pdf.hvplot.points(x=\"value_time\", y=\"primary_value\", color=\"event_above_id\") #.opts(width=1200, height=400)"
    ]
   },
   {
@@ -947,10 +947,10 @@
     "(\n",
     "    ev.metrics\n",
     "    .query(\n",
-    "        group_by=[\"configuration_name\", \"primary_location_id\", \"event_id\"],\n",
+    "        group_by=[\"configuration_name\", \"primary_location_id\", \"event_above_id\"],\n",
     "        filters=[\n",
     "            \"primary_location_id = 'usgs-14138800'\",\n",
-    "            \"event = true\",\n",
+    "            \"event_above = true\",\n",
     "        ],\n",
     "        include_metrics=[\n",
     "            teehr.Signatures.Maximum(\n",
@@ -1004,10 +1004,10 @@
     "(\n",
     "    ev.metrics\n",
     "    .query(\n",
-    "        group_by=[\"configuration_name\", \"primary_location_id\", \"event_id\"],\n",
+    "        group_by=[\"configuration_name\", \"primary_location_id\", \"event_above_id\"],\n",
     "        filters=[\n",
     "            \"primary_location_id = 'usgs-14138800'\",\n",
-    "            \"event = true\",\n",
+    "            \"event_above = true\",\n",
     "        ],\n",
     "        include_metrics=[\n",
     "            teehr.Signatures.Maximum(\n",
@@ -1037,7 +1037,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "One last thing to cover here.  So far we have added the calculated fields on the `joined_timeseries` table, written them to disk, and then queried the `joined_timeseries` table to calculate metrics.  This works well and allows the calculated fields to be calculated once and used in many subsequent metrics, plots, etc.  However, you may wish to add temporary fields to the `joined_timeseries` table as part of the metrics calculation.  This can be done too.  Building on the pervious example, where we calculated the \"event_max_relative_bias\", lets now assume we want to calculate the same metric but for the 90th percentile instead of the default 85th percentile that we used when we added the added the `event` and `event_id` fields to the `joined_timeseries` table.  We could add the new \"90th percentile event\" to the `joined_timeseries` table and save to disk and then proceed as we did before, or we can add new `event90` and `event90_id` fields to the data frame temporarily before calculating the maximum event values and ultimately the \"event_90th_max_relative_bias\"."
+    "One last thing to cover here.  So far we have added the calculated fields on the `joined_timeseries` table, written them to disk, and then queried the `joined_timeseries` table to calculate metrics.  This works well and allows the calculated fields to be calculated once and used in many subsequent metrics, plots, etc.  However, you may wish to add temporary fields to the `joined_timeseries` table as part of the metrics calculation.  This can be done too.  Building on the previous example, where we calculated the \"event_max_relative_bias\", lets now assume we want to calculate the same metric but for the 90th percentile instead of the default 85th percentile that we used when we added the added the `event` and `event_id` fields to the `joined_timeseries` table.  We could add the new \"90th percentile event\" to the `joined_timeseries` table and save to disk and then proceed as we did before, or we can add new `event90` and `event90_id` fields to the data frame temporarily before calculating the maximum event values and ultimately the \"event_90th_max_relative_bias\"."
    ]
   },
   {
@@ -1075,10 +1075,10 @@
    "source": [
     "(\n",
     "    ev.metrics\n",
-    "    # Add the PercentileEventDetection calculated field to identify events greater than the 90th percentile.\n",
+    "    # Add the AbovePercentileEventDetection calculated field to identify events greater than the 90th percentile.\n",
     "    # Note the output_event_field_name and output_event_id_field_name are set to \"event90\" and \"event90_id\" respectively.\n",
     "    .add_calculated_fields([\n",
-    "        tcf.PercentileEventDetection(\n",
+    "        tcf.AbovePercentileEventDetection(\n",
     "            quantile=0.90,\n",
     "            output_event_field_name=\"event90\",\n",
     "            output_event_id_field_name=\"event90_id\"\n",
@@ -1118,6 +1118,96 @@
     ")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Categorical Deterministic Metrics\n",
+    "---------------------------------\n",
+    "\n",
+    "Now that we understand routines for grouping/filtering data and added calculated fields in TEEHR to obtain performance metrics, we can introduce the concept of categorical deterministic metrics. \n",
+    "\n",
+    "Categorical deterministic metrics are utilized to measure qualitative attributes and are used to evaluate binary outcomes given some condition or categorical classification. Put simply, these methods help compare predictions made by the model with observed data to indicate where the model was right or wrong.\n",
+    "\n",
+    "Currently, the following categorical deterministic methods are available in TEEHR:\n",
+    "\n",
+    "- [ConfusionMatrix](https://rtiinternational.github.io/teehr/api/generated/teehr.DeterministicMetrics.html#teehr.DeterministicMetrics.ConfusionMatrix)\n",
+    "- [FalseAlarmRatio](https://rtiinternational.github.io/teehr/api/generated/teehr.DeterministicMetrics.html#teehr.DeterministicMetrics.FalseAlarmRatio)\n",
+    "- [ProbabilityOfDetection](https://rtiinternational.github.io/teehr/api/generated/teehr.DeterministicMetrics.html#teehr.DeterministicMetrics.ProbabilityOfDetection)\n",
+    "- [ProbabilityOfFalseDetection](https://rtiinternational.github.io/teehr/api/generated/teehr.DeterministicMetrics.html#teehr.DeterministicMetrics.ProbabilityOfFalseDetection)\n",
+    "- [CriticalSuccessIndex](https://rtiinternational.github.io/teehr/api/generated/teehr.DeterministicMetrics.html#teehr.DeterministicMetrics.CriticalSuccessIndex)\n",
+    "\n",
+    "Unlike other metrics in TEEHR which operate on the default fields present in the joined timeseries table, categorical deterministic metrics require an additional 'threshold field' column to categorize model predictive performance based on a user defined flow threshold. \n",
+    "\n",
+    "For example, let's utilize our categorical metrics to evaluate model performance when predicting streamflow that exceeds the 10% return interval. Lets start from the default `joined_timeseries` table and complete the chained operation in-memory as opposed to writing to disk or using the fields we calculated in the previous steps."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# overwrite the existing joined_timeseries table to demonstrate chained query\n",
+    "ev.joined_timeseries.create(add_attrs=False, execute_scripts=False)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "metrics_df = ev.metrics.add_calculated_fields([\n",
+    "    # adds 'event', 'event_id', and 'quantile_value' fields to the joined timeseries table\n",
+    "    teehr.TimeseriesAwareCalculatedFields.AbovePercentileEventDetection(\n",
+    "        quantile=0.90,\n",
+    "        add_quantile_field=True\n",
+    "    )\n",
+    "]).query(\n",
+    "    # calculate all available categorical deterministic metrics using the 'quantile_value' field as the threshold\n",
+    "    group_by=['primary_location_id', 'configuration_name'],\n",
+    "    include_metrics=[\n",
+    "        teehr.DeterministicMetrics.ConfusionMatrix(\n",
+    "            threshold_field_name='quantile_value'\n",
+    "        ),\n",
+    "        teehr.DeterministicMetrics.FalseAlarmRatio(\n",
+    "            threshold_field_name='quantile_value'\n",
+    "        ),\n",
+    "        teehr.DeterministicMetrics.ProbabilityOfDetection(\n",
+    "            threshold_field_name='quantile_value'\n",
+    "        ),\n",
+    "        teehr.DeterministicMetrics.ProbabilityOfFalseDetection(\n",
+    "            threshold_field_name='quantile_value'\n",
+    "        ),\n",
+    "        teehr.DeterministicMetrics.CriticalSuccessIndex(\n",
+    "            threshold_field_name='quantile_value'\n",
+    "        )\n",
+    "    ]\n",
+    ").to_sdf().show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In the above example, we utilize a chained query to add a 'threshold field' to our unaltered `joined_timeseries` table and calculate all available categorical metrics in one operation. \n",
+    "\n",
+    "The chained operation executes in two steps:\n",
+    "\n",
+    "- The first portion of the operation uses the `AbovePercentileEventDetection` method with `add_quantile_field` set to True to obtain the flow value that corresponds to the 90% quantile for each row in the joined timeseries table.\n",
+    "\n",
+    "- The following portion takes the resulting table (with `AbovePercentileEventDetection` fields present) and executes a metrics query for our categorical deterministic metrics (with `threshold_field_name` arguments populated with the default `output_quantile_field_name` specified in the `AbovePercentileEventDetection` method). \n",
+    "\n",
+    "As you can see, we obtain a unique entry for the following metrics per unique combination of `primary_location_id` and `configuration_name` (as specified in the `group_by` argument in the `ev.metrics.query()` call).\n",
+    "\n",
+    "<b>When employing categorical deterministic metrics in TEEHR, keep in mind that each aggregation must correspond to exactly one 'threshold value'</b>. \n",
+    "\n",
+    "In this example, that criteria is achieved inherently given the `quantile_value` was generated using a `TimeSeriesAwareCalculatedField` which considers each 'unique timeseries' (where 'unique timeseries' is defined by `AbovePercentileEventDetection.uniqueness_fields`, by default=`['reference_time', 'primary_location_id', 'configuration_name', 'variable_name', 'unit_name']`). \n",
+    "\n",
+    "By that same logic, each unique combination of `primary_location_id` and `configuration_name` defined in the `group_by` argument in this example has exactly one 'threshold value' because `['reference_time', 'variable_name', 'unit_name']` fields are constant for each unique combination of `['primary_location_id, 'configuration_name']`."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 20,