Merge pull request #345 from posit-dev/feat-data-freshness

feat: add the `data_freshness()` validation method

Author: rich-iannone

docs/_quarto.yml

@@ -194,6 +194,7 @@ quartodoc:
         - name: Validate.rows_complete
         - name: Validate.col_exists
         - name: Validate.col_pct_null
+        - name: Validate.data_freshness
         - name: Validate.col_schema_match
         - name: Validate.row_count_match
         - name: Validate.col_count_match

docs/user-guide/validation-methods.qmd

@@ -357,6 +357,7 @@ These structural checks form a foundation for more detailed data quality assessm
 - `~~Validate.col_count_match()`: confirms the table has the expected number of columns
 - `~~Validate.row_count_match()`: verifies the table has the expected number of rows
 - `~~Validate.tbl_match()`: validates that the target table matches a comparison table
+- `~~Validate.data_freshness()`: checks that data is recent and not stale
 
 These structural validations provide essential checks on the fundamental organization of your data
 tables, ensuring they have the expected dimensions and components needed for reliable data analysis.
@@ -494,6 +495,70 @@ matches a specified count.
 Expectations on column and row counts can be useful in certain situations and they align nicely with
 schema checks.
 
+### Validating Data Freshness
+
+Late or missing data is one of the most common (and costly) data quality issues in production
+systems. When data pipelines fail silently or experience delays, downstream analytics and ML models
+can produce stale or misleading results. The `~~Validate.data_freshness()` validation method helps
+catch these issues early by verifying that your data contains recent records.
+
+Data freshness validation works by checking a datetime column against a maximum allowed age. If the
+most recent timestamp in that column is older than the specified threshold, the validation fails.
+This simple check can prevent major downstream problems caused by stale data.
+
+Here's an example that validates data is no older than 2 days:
+
+```{python}
+import polars as pl
+from datetime import datetime, timedelta
+
+# Simulate a data feed that should be updated daily
+recent_data = pl.DataFrame({
+    "event": ["login", "purchase", "logout", "signup"],
+    "event_time": [
+        datetime.now() - timedelta(hours=1),
+        datetime.now() - timedelta(hours=6),
+        datetime.now() - timedelta(hours=12),
+        datetime.now() - timedelta(hours=18),
+    ],
+    "user_id": [101, 102, 103, 104]
+})
+
+(
+    pb.Validate(data=recent_data)
+    .data_freshness(column="event_time", max_age="2d")
+    .interrogate()
+)
+```
+
+The `max_age=` parameter accepts a flexible string format: `"30m"` for 30 minutes, `"6h"` for 6
+hours, `"2d"` for 2 days, or `"1w"` for 1 week. You can also combine units: `"1d 12h"` for 1.5 days.
+
+When validation succeeds, the report includes details about the data's age in the footer. When it
+fails, you'll see exactly how old the most recent data is and what threshold was exceeded. This
+context helps quickly diagnose whether you're dealing with a minor delay or a major pipeline
+failure.
+
+Data freshness validation is particularly valuable for:
+
+- monitoring ETL pipelines to catch failures before they cascade to reports and dashboards
+- validating data feeds to ensure third-party data sources are delivering as expected
+- including freshness checks in automated data quality tests as part of continuous integration
+- building alerting systems that trigger notifications when critical data becomes stale
+
+You might wonder why not just use `~~Validate.col_vals_gt()` with a datetime threshold. While that
+approach works, `~~Validate.data_freshness()` offers several advantages: the method name clearly
+communicates your intent, the `max_age=` string format (e.g., `"2d"`) is more readable than datetime
+arithmetic, it auto-generates meaningful validation briefs, the report footer shows helpful context
+about actual data age and thresholds, and timezone mismatches between your data and comparison time
+are handled gracefully with informative warnings.
+
+::: {.callout-note}
+When comparing timezone-aware and timezone-naive datetimes, Pointblank will include a warning in the
+validation report. For consistent results, ensure your data and comparison times use compatible
+timezone settings.
+:::
+
 ## 4. AI-Powered Validations
 
 AI-powered validations use Large Language Models (LLMs) to validate data based on natural language

pointblank/_constants.py

@@ -49,6 +49,7 @@
     "col_schema_match": "col_schema_match",
     "row_count_match": "row_count_match",
     "col_count_match": "col_count_match",
+    "data_freshness": "data_freshness",
     "tbl_match": "tbl_match",
     "conjointly": "conjointly",
     "specially": "specially",
@@ -725,6 +726,19 @@
             <path d="M11.5931863,12.5146694 C11.3836625,12.5146694 10.212234,12.5646694 10.212234,13.8480027 L10.212234,53.181336 C10.212234,54.4646694 11.3836625,54.5146694 11.5931863,54.5146694 L14.1646149,54.5146694 L14.1646149,12.5146694 L11.5931863,12.5146694 Z M20.1721771,12.5146694 L20.1721771,54.5146694 L16.2522908,54.5146694 L16.2522908,54.5146694 L16.2522908,12.5146694 L16.2522908,12.5146694 L20.1721771,12.5146694 Z M24.8656149,12.5150904 C25.1448786,12.521763 26.212234,12.6230027 26.212234,13.8480027 L26.212234,13.8480027 L26.212234,53.181336 C26.212234,54.4646694 25.0408054,54.5146694 24.8312816,54.5146694 L24.8312816,54.5146694 L22.259853,54.5146694 L22.259853,12.5146694 Z" id="rows_one" fill="#000000" fill-rule="nonzero" transform="translate(18.212234, 33.514669) rotate(-180.000000) translate(-18.212234, -33.514669) "></path>
         </g>
     </g>
+</svg>""",
+    "data_freshness": """<?xml version="1.0" encoding="UTF-8"?>
+<svg width="67px" height="67px" viewBox="0 0 67 67" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
+    <title>data_freshness</title>
+    <g id="All-Icons" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
+        <g id="data_freshness" transform="translate(0.000000, 0.275862)">
+            <path d="M56.712234,1 C59.1975153,1 61.4475153,2.00735931 63.076195,3.63603897 C64.7048747,5.26471863 65.712234,7.51471863 65.712234,10 L65.712234,10 L65.712234,65 L10.712234,65 C8.22695259,65 5.97695259,63.9926407 4.34827294,62.363961 C2.71959328,60.7352814 1.71223397,58.4852814 1.71223397,56 L1.71223397,56 L1.71223397,10 C1.71223397,7.51471863 2.71959328,5.26471863 4.34827294,3.63603897 C5.97695259,2.00735931 8.22695259,1 10.712234,1 L10.712234,1 Z" id="rectangle" stroke="#000000" stroke-width="2" fill="#FFFFFF"></path>
+            <circle id="clock-face" stroke="#000000" stroke-width="2" cx="33.5" cy="33" r="20"></circle>
+            <line x1="33.5" y1="33" x2="33.5" y2="20" id="hour-hand" stroke="#000000" stroke-width="2.5" stroke-linecap="round"></line>
+            <line x1="33.5" y1="33" x2="44" y2="33" id="minute-hand" stroke="#000000" stroke-width="2" stroke-linecap="round"></line>
+            <circle id="center-dot" fill="#000000" cx="33.5" cy="33" r="2"></circle>
+        </g>
+    </g>
 </svg>""",
     "tbl_match": """<?xml version="1.0" encoding="UTF-8"?>
 <svg width="67px" height="67px" viewBox="0 0 67 67" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">