Merge pull request #15 from OpenDataServices/2024-10-09

2024 10 09

Author: James (ODSC)

CHANGELOG.md

@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## Changed
+
+- In `TaskWithState.process_get_state()` any changes to `process_data` are now ignored. https://github.com/OpenDataServices/lib-cove-web-2/issues/14
+
 ## [0.3.0] - 2023-09-14
 
 When upgrading to this version, `ALLOWED_UNKNOWN_CONTENT_TYPES` must be set in the Django settings file, ideally from the settings file included with this library.

docs/hosting/index.rst

@@ -0,0 +1,10 @@
+Hosting
+=======
+
+
+
+.. toctree::
+   :maxdepth: 2
+
+   requirements.rst
+   with_dokku.rst

docs/hosting/requirements.rst

@@ -1,11 +1,6 @@
 Hosting Requirements
 ====================
 
-Python server for the Django app
-----------------------------------
-
-Normal options
-
 Database
 ----------
 
@@ -19,7 +14,36 @@ Normal options
 File Storage
 ------------
 
-TODO
+A file storage area is needed. This holds the files supplied by users, and the cached results of processing.
+
+All code access in this library to the file storage area should be through Django `DefaultStorage` API.
+This means in theory, you are free to use:
+
+* file storage by setting `MEDIA_ROOT` and `MEDIA_URL`.
+* cloud bucket solutions (eg AWS S3, Azure Storage Container).
+* any other Django compatible Storage API.
+
+Nothing in this library requires the contents of this storage to be served on the web.
+
+However, code in apps that uses this library may have:
+
+* direct access to the file system, in which case file storage by setting `MEDIA_ROOT` and `MEDIA_URL` is required.
+* direct users to URL's for download, in which case the contents of this storage must be served on the web.
+
+All apps that use this library currently just use direct file storage by setting `MEDIA_ROOT` and `MEDIA_URL`,
+and that should be considered the only tested solution.
+
+
+Python web server for the Django app
+------------------------------------
+
+Normal options
+
+
+A background worker
+-------------------
+
+Run using Celery's normal run options.
 
 Cron tasks
 ----------

docs/hosting/with_dokku.rst

@@ -0,0 +1,8 @@
+Hosting with Dokku
+==================
+
+For examples of this, see:
+
+* https://github.com/openownership/cove-bods
+* https://github.com/Open-Telecoms-Data/cove-ofds
+* https://github.com/GFDRR/rdls-cove

docs/index.rst

@@ -26,6 +26,6 @@ The application consists of:
    django-settings.rst
    python-api/index.rst
    migration-from-lib-cove-web.rst
-   hosting/requirements.rst
+   hosting/index.rst 
    used-by.rst
 

libcoveweb2/process/common_tasks/task_with_state.py

@@ -10,12 +10,18 @@
 class TaskWithState(ProcessDataTask):
     """An abstract task that helps you save state from
     the processing step and add it to the context.
+
+    It will cache one JSON blob for you, and add
+    it to the view context when a user is looking at the results.
+    (So make sure you choose keys in the JSON blob carefully
+    so as not to clash with other view context variables!)
+
     Extend and provide your own state_filename and process_get_state.
     """
 
     #: Set state_filename to a unique name for each task.
     #:
-    #: If you change this name the task will be rerun, so this is is a good way to
+    #: If you change this name the task will be rerun, so this is a good way to
     #: make sure all underlying data changes if a new version of this bit of cove
     #: is released.
     state_filename: str = "task_with_state.json"
@@ -28,14 +34,19 @@ def process_get_state(self, process_data: dict):
 
         Should return a tuple.
         The first item is the results to save, as a dictionary.
-        The second item is process_data, as a dictionary."""
+        The second item is process_data, as a dictionary.
+
+        Do NOT change process_data in this function!
+        The fact it's returned is a mistake:
+        https://github.com/OpenDataServices/lib-cove-web-2/issues/14
+        """
         return {}, process_data
 
     def process(self, process_data: dict) -> dict:
         if self.does_state_exist():
             return process_data
 
-        state, process_data = self.process_get_state(process_data)
+        state, process_data_throw_away = self.process_get_state(process_data)
 
         default_storage.save(
             os.path.join(self.supplied_data.storage_dir(), self.state_filename),