Data quality diagnostics for `processed_cube` objects

[wlangera]

We would like to propose the implementation of data quality diagnostics for processed_cube objects in dubicube. The goal is to support exploratory data analysis and to assess whether a data cube is fit for use, either in general or in relation to specific biodiversity indicators.

Motivation

Biodiversity indicators can be highly sensitive to uneven data coverage across space, time, and taxonomy. Currently, users have limited tooling to diagnose these issues before (or alongside) indicator calculation. Providing standardized robustness diagnostics would:

• Improve transparency and interpretability of indicators
• Support informed indicator selection
• Enable early detection of problematic datasets

Scope (high level)

We envision diagnostics along three main dimensions:

Spatial

• Data distribution (clustered vs evenly spread)
• Geographical coverage (localized vs widespread species)

Temporal

• Temporal variation in occurrences
• Stability over time, optionally compared to higher taxonomic levels

Taxonomical

• Species prevalence or abundance
• Overrepresentation of certain species in multispecies indicators

Proposed functionality (overview)

1. Measuring data cube robustness

   • Input: processed_cube, optionally an indicator function
   • Output: a structured summary with warnings / flags and short explanations
   • Intended for data exploration and pre-indicator checks

2. Filtering observations based on robustness criteria

   • Input: processed_cube
   • Output: filtered processed_cube
   • Includes sensible defaults (e.g. excluding species observed only once in a single year for trend analysis)

[wlangera]

Some brainstorming with ChatGPT:

Below is a concrete, technical proposal, grounded in how dubicube already works and in established R ecosystem patterns.

tl;dr

• Return a structured cube_diagnostics object, not messages
• Use rule-based diagnostics
• Separate metrics, severity, and messages
• Messages are presentation, not data
• Reuse diagnostics for filtering
• Keep dependencies light and tidyverse-adjacent

---

1. Core design principle

Diagnostics should be:

• Structured (machine-readable)
• Composable (many small checks, not one giant function)
• Explainable (human-readable messages)
• Indicator-aware, but not indicator-dependent

2. Proposed output type: cube_diagnostics (S3)

Why not just messages?

• Messages are hard to test
• Impossible to program against
• Lost once printed
• Not filterable or plottable

Recommended approach

Return a structured object with:

• Metrics
• Flags
• Severity levels
• Text explanations

diag <- diagnose_cube(cube)
class(diag)
#> [1] "cube_diagnostics"

Internal structure (example)

A tibble-based structure works best:

tibble::tibble(
  dimension = c("spatial", "temporal", "taxonomical"),
  metric = c("spatial_coverage", "temporal_continuity", "species_dominance"),
  value = c(0.42, 0.63, 0.81),
  threshold = c(0.6, 0.5, 0.7),
  severity = c("important", "note", "very_important"),
  message = c(
    "Observations are clustered in limited spatial areas",
    "Temporal coverage varies strongly between years",
    "Few species dominate the multispecies index"
  )
)

This gives you:

• Printing
• Filtering
• Summarising
• Plotting
• Testing

3. Severity / flag system (simple but powerful)

Avoid free text; use fixed levels:

severity_levels <- c(
  "ok",
  "note",
  "important",
  "very_important"
)

These can later map to:

• Colors
• Warnings
• Stop conditions (optional)

print(diag)
# 🟡 IMPORTANT: Observations are clustered in limited spatial areas

4. User-facing API

Main function

diagnose_cube(
  cube,
  indicator = NULL,
  rules = default_cube_rules(),
  verbose = TRUE
)

Supporting functions

default_cube_rules()
custom_cube_rules(...)
add_cube_rule(...)

Rules are data, not hard-coded logic.

5. Rule-based diagnostics (key insight)

Each diagnostic should be a rule object:

list(
  id = "spatial_coverage",
  dimension = "spatial",
  compute = function(cube) {...},
  threshold = 0.6,
  severity = function(value, threshold) {...},
  message = function(value) {...}
)

This allows:

• Easy extension
• Project-specific rules (e.g. B3)
• Transparent logic

6. Filtering based on diagnostics

Filtering should reuse diagnostics, not reimplement logic.

filter_cube(
  cube,
  diagnostics = diag,
  min_severity = "important"
)

or directly:

filter_cube(
  cube,
  rules = list(
    min_years = 3,
    min_occurrences = 5
  )
)

Internally:

1. Evaluate rules
2. Attach flags to observations or species
3. Drop flagged units
4. Return new processed_cube

---

7. Printing, messages, and UX

Messages should be derived, not primary

• print.cube_diagnostics() → colored summary
• summary() → counts per severity & dimension
• autoplot() → coverage / dominance plots

Messages (message(), warning()) only when:

• verbose = TRUE
• very_important severity

8. Useful packages (lightweight, mainstream)

✅ Strongly recommended

Package	Why
tibble	structured diagnostics
dplyr	rule evaluation
rlang	tidy evaluation for rules
cli	clean, colored console output
pillar	nice printing
scales	rescaling metrics
vctrs	stable S3 objects

⚠️ Optional (but nice)

Package	Use
checkmate	input validation
glue	readable messages
ggplot2	diagnostics plots
sf	spatial coverage metrics
terra	raster-based coverage

Avoid heavy dependencies unless strictly needed.

9. Minimal viable implementation

Start with:

1. diagnose_cube()
2. 3–5 core rules
3. print() method
4. No plotting
5. No filtering yet

Then expand