Merge branch 'main' into fix_url

Author: jqnatividad

.github/workflows/main.yml

@@ -0,0 +1 @@
+3.10.19

.python-version

@@ -6,19 +6,72 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## What's Changed
+* Update README with some fixes by @tino097 in https://github.com/dathere/datapusher-plus/pull/178
+* Druf apr2025 by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/180
+* Refactor upload log level by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/181
+* feat: zip file support by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/182
+* feat: shapefile support by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/183
+* Refactor jobs py by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/184
+* feat: make frequency limit configurable; move stats/freq copying to datastore from jobs.py to qsv_utils.py by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/185
+* Lat lon columns inferencing for use in Formulas by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/188
+* Configurable Date/Datetime inferencing and dataset stats by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/190
+* refactor: move pii-screening to a separate module by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/191
+* chore: add WIP geojson update by @rzmk in https://github.com/dathere/datapusher-plus/pull/186
+* "smart" formula spatial functions by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/192
+* Jobs cleanup by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/193
+* Fix datastore upload log timestamps by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/194
+* DCAT 3 formula helpers by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/195
+* fix: tmp input was being wrongfully assigned  by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/197
+* refactored SQL-enabled formulas by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/199
+* auto unzip one file setting by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/200
+* add LRU caches to potentially expensive Formula methods by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/201
+* feat: add `dpp_suggestions.STATUS` to track formulae processing progress by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/202
+* refactor dpp_suggestions.STATUS to sync with Suggestions UI by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/203
+* Refactor: remove stats & freq  table save to datastore by @jqnatividad in https://github.com/dathere/datapusher-plus/pull/204
+
+## New Contributors
+* @rzmk made their first contribution in https://github.com/dathere/datapusher-plus/pull/186
+
+**Full Changelog**: https://github.com/dathere/datapusher-plus/compare/2.0.0...2.1.0
+
 ## [2.0.0] - 2025-04-25
 
-## Highlights
-Data Resource Upload First (DRUF) Workflow is here!
+## 🎉 Data Resource Upload First (DRUF) Workflow is finally here! 🎉 
 A workflow that flips the old CKAN traditional data ingestion on its head.
  * Instead of filling out the metadata first and then uploading the data, users upload data resources first 
  * In a few seconds, even for very large datasets, analysis and validation is done while precompiling statistical metadata
- * This precompiled metadata are then used by formulas defined in the scheming yaml files to either precompute other metadata fields and/or to offer metadata suggestions
- * Formulas use the same powerful Jinja2 template engine that powers CKAN's templating system.
- * It comes with an extensible library of Jinja2 filters/functions that can be used in formulas ala Excel.
-
-The DRUF reinvents CKAN data ingestion - making it easier for Data Publishers to ensure their data catalog has high-quality, high-resolution metadata that actually reflects the and describes the data in the catalog.
-
+ * This precompiled metadata are then used by Metadata Formulae defined in the [scheming](https://github.com/ckan/ckanext-scheming?tab=readme-ov-file#ckanext-scheming) yaml files to either precompute other metadata fields (on both package & resource levels) or to offer metadata suggestions
+ * Metadata Formulae use the same powerful Jinja2 template engine that powers CKAN's templating system.
+ * It comes with an extensible library of Jinja2 filters/functions that can be used in Metadata Formulae ala Excel.
+
+The DRUF reinvents CKAN data ingestion - by automatically calculating/suggesting "**Automagical Metadata**" - high-quality, high-resolution metadata that reflects and describes what's **INSIDE** the dataset (e.g. summary stats; frequency table; spatial extent, date range, outliers, etc. calculated with Metadata Formulae) in addition to metadata about the dataset **FILE** (e.g. last updated, size of the file, owner, format, license, etc - what's normally found in traditional data catalogs).
+
+Future improvements planned:
+- **Expanded Data Dictionary**
+
+- **"entry-time" Metadata Formulae**
+In addition to the two formula types (`formula` to set a metadata field directly during creation/update; and `suggestion_formula` to suggest values using the Bootstap Popover UI), we'll add the ability to allow Data Publishers to enter formulas while they're entering metadata - fully embracing the Excel formula UI/UX aesthetic.
+- **DCAT3-optimized reference profiles**
+Following implementation guidance for both [DCAT-US v3](https://doi-do.github.io/dcat-us/) and [DCAT-AP 3](https://semiceu.github.io/DCAT-AP/releases/3.0.0/) scheming profiles with Metadata Formulae to compute recommended and optional properties that allow publishers to more fully take advantage of DCAT3 features and improvements - metadata properties that are often too laborious to manually compile.
+- **Co-Curator AI**
+"Automagical metadata" is the perfect context for AI engines - as it summarizes even very large datasets in just a few kilobytes. It allows the Co-Curator[^1] to suggest tags, descriptions, links to related data sets and chat about the corpus WHILE the Data Publisher is curating the data.
+- **Inline Data Validation**
+Optional ability to [infer an initial JSON Schema validation file](https://github.com/dathere/qsv?tab=readme-ov-file#schema_deeplink), and then [validate future updates](https://github.com/dathere/qsv?tab=readme-ov-file#validate_deeplink) to the dataset using it, leveraging the same blazing-fast qsv engine (validating up to 340,000 records/per second[^2]). 
+- **Customizable DRUF Data ingestion pipeline**
+Currently, there are [numerous configuration settings](https://github.com/dathere/datapusher-plus/blob/main/ckanext/datapusher_plus/config.py) to fine-tune the DRUF data-ingestion pipeline. However, the built-in default pipeline can only be customized to a limit without customizing the code. We will expose hooks that CKAN operators can take advantage of to tailor their DRUF pipelines to meet their requirements, while preserving the ability to access the precompiled statistical metadata that DP+ maintains.
+- **Dynamic loading of Formula filters/functions**
+So users can share custom Jinja2 filters and functions they developed for their Metadata Formulae.
+- **Inline Data Enrichment**
+Data can be optionally enriched while it's being ingested from other reference datasets within the same CKAN instance or external sources (e.g. enriched against high value curated sources like the Census; geocoding, etc.)
+- **and more!**
+It took a while for us to bake 2.0.0, but we look forward to picking up the pace and co-innovating with the CKAN ecosystem.
+
+
+> NOTE: To fully experience the DRUF workflow, you'll need to use [scheming dataset form pages](https://excess.org/scheming-formpages/) and apply some CKAN core changes. A detailed installation procedure will be published on the Wiki shortly.
+
+[^1]: Inspired by the [Curator in Ready Player One](https://hero.fandom.com/wiki/Curator)
+[^2]: `validate_index benchmark` -  https://qsv.dathere.com/benchmarks
 ---
 
 ### Added

CHANGELOG.md

@@ -0,0 +1,96 @@
+# DataPusher Plus Configuration
+
+## Optional Features
+
+DataPusher Plus includes some optional features that can be enabled through configuration. These features are disabled by default to ensure compatibility with different CKAN versions.
+
+### IFormRedirect Support
+
+The IFormRedirect interface provides custom redirect behavior after dataset and resource form submissions. This interface is only available in certain CKAN branches and is not yet merged into the main CKAN codebase.
+
+**Note**: IFormRedirect methods are only defined when this feature is enabled, keeping the plugin completely clean when disabled.
+
+**Configuration:**
+```ini
+# Enable IFormRedirect functionality (default: false)
+ckanext.datapusher_plus.enable_form_redirect = true
+```
+
+**What it does:**
+- **Dynamically adds IFormRedirect methods** only when enabled
+- Provides custom redirect URLs after dataset/resource creation or editing
+- Redirects to dataset page after dataset metadata submission
+- Redirects to resource view after resource editing
+- Allows "add another resource" workflow
+- **Works best with DRUF** for complete resource-first workflow
+
+**Requirements:**
+- CKAN version with IFormRedirect interface support
+- If the interface is not available, the feature will be automatically disabled with a warning
+- **Recommended**: Enable together with DRUF for optimal resource-first experience
+
+### DRUF (Dataset Resource Upload First) Support
+
+DRUF allows users to upload resources before creating the dataset metadata, providing a resource-first workflow.
+
+**Configuration:**
+```ini
+# Enable DRUF functionality (default: false)  
+ckanext.datapusher_plus.enable_druf = true
+```
+
+**What it does:**
+- Adds a `/resource-first/new` endpoint
+- Creates a temporary dataset and redirects to resource upload
+- Useful for workflows where users want to upload data files first
+- **Overrides templates**: Modifies "Add Dataset" buttons and form stages to support resource-first workflow
+
+**Template Overrides:**
+When DRUF is enabled, the following templates are overridden:
+- `snippets/add_dataset.html`: Changes "Add Dataset" to redirect to resource upload
+- `package/snippets/package_form.html`: Modifies form stages to show "Add data" first
+- `scheming/package/snippets/package_form.html`: Modifies scheming form stages
+
+**Requirements:**
+- No special CKAN version requirements
+- Works with standard CKAN installations
+- Compatible with ckanext-scheming
+
+## Example Configuration
+
+Add these lines to your CKAN configuration file (e.g., `/etc/ckan/default/ckan.ini`):
+
+```ini
+# Enable DRUF (Dataset Resource Upload First) workflow
+ckanext.datapusher_plus.enable_druf = true
+
+# Enable IFormRedirect for better form redirects (recommended with DRUF)
+ckanext.datapusher_plus.enable_form_redirect = true
+```
+
+**Recommended combinations:**
+- **Standard mode**: Both disabled (default) - maintains standard CKAN behavior
+- **Resource-first workflow**: Both enabled - complete resource-first experience
+- **DRUF only**: Only `enable_druf = true` - resource-first without custom redirects
+
+## Template Organization
+
+DataPusher Plus uses a conditional template loading system to avoid conflicts when optional features are disabled:
+
+- **Base templates** (`templates/`): Always loaded, provides standard DataPusher Plus functionality
+- **DRUF templates** (`templates/druf/`): Only loaded when `enable_druf = true`, overrides default dataset creation workflow
+
+This ensures that when DRUF is disabled, your CKAN installation maintains completely standard behavior without any template modifications.
+
+## Backwards Compatibility
+
+When these features are disabled (default), DataPusher Plus maintains full backwards compatibility with standard CKAN installations. The plugin will automatically detect if required interfaces are available and disable features gracefully if they are not supported.
+
+## Logging
+
+The plugin will log the status of these features:
+- Info messages when features are successfully enabled
+- Warning messages when features are configured but not available
+- Debug messages for DRUF blueprint registration
+
+Check your CKAN logs to verify the status of these optional features.