Merge branch '1232-process-the-full-text-dump' of github.com:NASA-IMPACT/COSMOS into 1232-process-the-full-text-dump

Author: CarsonDavis

.github/workflows/run_full_test_suite.yml

@@ -4,6 +4,8 @@ on:
   pull_request:
     branches:
       - dev
+    paths-ignore:
+      - '**/*.md'
 
 jobs:
   run-tests:

CHANGELOG.md

@@ -12,6 +12,24 @@ For each PR made, an entry should be added to this changelog. It should contain
   - etc.
 
 ## Changelog
+
+- 1209-bug-fix-document-type-creator-form
+  - Description: The dropdown on the pattern creation form needs to be set as multi as the default option since this is why the doc type creator form is used for the majority of multi-URL pattern creations. This should be applied to doc types, division types, and titles as well.
+  - Changes:
+    - Set the default value for `match_pattern_type` in `BaseMatchPattern` class is set to `2`
+    - Changed `test_create_simple_exclude_pattern` test within `TestDeltaExcludePatternBasics`
+    - Changed `test_create_division_pattern` and `test_create_document_type_pattern_single` within `TestFieldModifierPatternBasics`
+
+- 1052-update-cosmos-to-create-jobs-for-scrapers-and-indexers
+  - Description: The original automation set up to generate the scrapers and indexers automatically based on a collection workflow status change needed to be updated to more accurately reflect the curation workflow. It would also be good to generate the jobs during this process to streamline the same.
+  - Changes:
+    - Updated function nomenclature. Scrapers are Sinequa connector configurations that are used to scrape all the URLs prior to curation. Indexers are Sienqua connector configurations that are used to scrape the URLs post to curation, which would be used to index content on production. Jobs are used to trigger the connectors which are included as parts of joblists.
+    - Parameterized the convert_template_to_job method to include the job_source to streamline the value added to the `<Collection>` tag in the job XML.
+    - Updated the fields that are pertinenet to transfer from a scraper to an indexer. Also added a third level of XML processing to facilitate the same.
+    - scraper_template.xml and indexer_template.xml now contains the templates used for the respective configuration generation.
+    - Deleted the redundant webcrawler_initial_crawl.xml file.
+    - Added and updated tests on workflow status triggers.
+
 - 2889-serialize-the-tdamm-tags
   - Description: Have TDAMM serialzed in a specific way and exposed via the Curated URLs API to be consumed into SDE Test/Prod
   - Changes:
@@ -35,3 +53,144 @@ For each PR made, an entry should be added to this changelog. It should contain
     - Defined a class `HTMLFreeCharField` which inherits `serializers.CharField`
     - Used regex to catch any HTML content comming in as an input to form fields
     - Called this class within the serializer for necessary fields
+
+- 1030-resolve-0-value-document-type-in-nasa_science
+  - Description: Around 2000 of the docs coming out of the COSMOS api for nasa_science have a doc type value of 0.
+  - Changes:
+    - Added `obj.document_type != 0` as a condition in the `get_document_type` method within the `CuratedURLAPISerializer`
+
+- 1014-add-logs-when-importing-urls-so-we-know-how-many-were-expected-how-many-succeeded-and-how-many-failed
+  - Description: When URLs of a given collection are imported into COSMOS, a Slack notification is sent. This notification includes the name of the collection imported,count of the existing curated URLs, total URLs count as per the server, URLs successfully imported from the server, delta URLs identified and delta URLs marked for deletion.
+  - Changes:
+    - The get_full_texts() function in sde_collections/sinequa_api.py is updated to yeild total_count along with rows.
+    - fetch_and_replace_full_text() function in sde_collections/tasks.py captures the total_server_count and triggers send_detailed_import_notification().
+    - Added a function send_detailed_import_notification() in sde_collections/utils/slack_utils.py to structure the notification to be sent.
+    - Updated the associated tests effected due to inclusion of this functionality.
+
+- 3228-bugfix-preserve-scroll-position--document-type-selection-behavior-on-individual-urls
+  - Description: Upon selecting a document type on any individual URL, the page refreshes and returns to the top. This is not necessarily a bug but an inconvenience, especially when working at the bottom of the page. Fix the JS code.
+  - Changes:
+    - Added a constant `scrollPosition` within `postDocumentTypePatterns` to store the y coordinate postion on the page
+    - Modified the ajax relaod to navigate to this position upon posting/saving the document type changes.
+
+- 3227-bugfix-title-patterns-selecting-multi-url-pattern-does-nothing
+  - Description: When selecting options from the match pattern type filter, the system does not filter the results as expected. Instead of displaying only the chosen variety of patterns, it continues to show all patterns.
+  - Changes:
+    - In `title_patterns_table` definition, corrected the column reference
+    - Made `match_pattern_type` searchable
+    - Corrected the column references and made code consistent on all the other tables, i.e., `exclude_patterns_table`, `include_patterns_table`, `division_patterns_table` and `document_type_patterns_table`
+
+- 1190-add-tests-for-job-generation-pipeline
+  - Description: Tests have been added to enhance coverage for the config and job creation pipeline, alongside comprehensive tests for XML processing.
+  - Changes:
+    - Added config_generation/tests/test_config_generation_pipeline.py which tests the config and job generation pipeline, ensuring all components interact correctly
+    - config_generation/tests/test_db_to_xml.py is updated to include comprehensive tests for XML Processing
+
+- 1001-tests-for-critical-functionalities
+  - Description: Critical functionalities have been identified and listed, and critical areas lacking tests listed
+  - Changes:
+    - Integrated coverage.py as an indicative tool in the workflow for automated coverage reports on PRs, with separate display from test results.
+    - Introduced docs/architecture-decisions/testing_strategy.md, which includes the coverage report, lists critical areas, and specifically identifies those critical areas that are untested or under-tested.
+
+- 1192-finalize-the-infrastructure-for-frontend-testing
+  - Description: Set up comprehensive frontend testing infrastructure using Selenium WebDriver with Chrome, establishing a foundation for automated UI testing.
+  - Changes:
+    - Added Selenium testing dependency to `requirements/local.txt`
+    - Updated Dockerfile to support Chrome and ChromeDriver
+    - Created BaseTestCase and AuthenticationMixin for reusable test components
+    - Implemented core authentication test suite
+
+- 1195-implement-unit-test-for-forms-on-the-frontend
+  - Description: Implemented comprehensive frontend test suite covering authentication, collection management, search functionality, and pattern application forms.
+  - Changes:
+    - Added tests for authentication flows
+    - Implemented collection display and data table tests
+    - Added universal search functionality tests
+    - Created search pane filter tests
+    - Added pattern application form tests with validation checks
+
+- 1101-bug-fix-quotes-not-escaped-in-titles
+  - Description: Title rules that include single quotes show up correctly in the sinequa frontend (and the COSMOS api) but not in the delta urls page.
+  - Changes:
+    - Added `escapeHtml` function in the `delta_url_list.js` file to handle special character escaping correctly.
+    - Called this function while retrieving the titles in `getGeneratedTitleColumn()` and `getCuratedGeneratedTitleColumn()` functions.
+
+- 1240-fix-code-scanning-alert-inclusion-of-functionality-from-an-untrusted-source
+  - Description: Ensured all external resources load securely by switching to HTTPS and adding Subresource Integrity (SRI) checks.
+  - Changes:
+    - Replaced protocol‑relative URLs with HTTPS.
+    - Added SRI (integrity) and crossorigin attributes to external script tags.
+
+- 1196-arrange-the-show-100-csv-customize-columns-boxes-to-be-in-one-line-on-the-delta-urls-page
+  changelog-update-Issue-1001
+  - Description: Formatting the buttons - 'Show 100','CSV' and 'Customize Columns' to be on a single line for an optimal use of space.
+  - Changes:
+    - Updated delta_url_list.css and delta_url_list.js files with necessary modifications
+
+- 1246-minor-enhancement-document-type-pattern-form-require-document-type-or-show-appropriate-error
+  - Description: In the Document Type Pattern Form, if the user does not select a Document Type while filling out the form, an appropriate error message is displayed.
+  - Changes:
+    - Added a JavaScript validation check on form submission to ensure the document type (stored in a hidden input) is not empty.
+    - Display an error message and prevent form submission if the field is empty.
+
+- 1249-add-https-link-to-cors_allowed_origins-for-sde-lrm
+  - Description: The feedback form API was throwing CORS errors and to rectify that, we need to add the apt https link for sde-lrm.
+  - Changes:
+    - Added `https://sde-lrm.nasa-impact.net` to `CORS_ALLOWED_ORIGINS` in the base settings.
+
+- 1252-document-type-filter-not-working-in-delta-urls-page
+  - Description: Fixed document type filtering functionality in the "Document Type Patterns" tab in Delta URLs page.
+  - Changes:
+    - Added a new event listener to the Document Type Patterns dropdown to trigger the filtering of the table results based on the selected value.
+
+- 1251-column-sorting-issue-curated-urls-count-sorts-by-delta-urls-count
+  - Description: Fixed incorrect sorting behavior in Collections table where sorting by Curated URLs column was not working as expected.
+  - Changes:
+    - Added `data-order` attribute to URL count columns for proper numeric sorting
+    - Updated SearchPane comparisons to use `@data-order` values instead of string-based loose equality checks to ensure correct numeric filtering
+
+- 1182-ml-classification-queue
+  - Description: The inference API will be providing confidence levels to the classification results to COSMOS. We require a robust job processing mechanism to batch URLs based on the load the API can handle, and track every individual job sent to the API and ultimately evaluate the status of jobs tied to each collection based on the results retrieved. This also needs to take the translation of the classification labels from the API to the tags used internally within COSMOS.
+  - Changes:
+    - New environment values have been created called `INFERENCE_API_URL` and `TDAMM_CLASSIFICATION_THRESHOLD` set in base settings.
+    - New models added:
+      - ModelVersion: Tracking system for multiple versions of classification models with API identifiers
+      - InferenceJob: Manages inference jobs for collections of URLs with a specific model version
+      - ExternalJob: Represents a batched job sent to the inference API, with multiple ExternalJobs per InferenceJob
+      - Status Tracking: Enum classes for job status tracking (queued, pending, completed, failed, cancelled, etc.)
+      - BatchProcessor: Handles batching of URLs for efficient API processing
+        - Text Length Management: Smart batching based on total text length with configurable maximum (default 10,000 chars)
+        - Oversized Text Handling: Automatic truncation of URLs that exceed maximum batch size
+        - Iterator Management: Safe handling of QuerySet iterators including proper cleanup
+      - InferenceAPIClient: Handles direct interaction with the Inference API
+        - Model Management: Loading, unloading, and status checking for models
+        - Job Submission: Support for batch submission with proper error handling
+        - Retry Logic: Robust retry mechanisms for model loading operations
+        - Health Checking: API health verification before operations
+      - ClassificationThresholdProcessor: A class to handle the class-based thresholding of classification results
+        - Separte classmethods for tdamm and division classifiers
+        - Config file to handle the thresholds for each class
+    - Celery Integration: Scheduled processing of inference job queue with configurable interval, executes `process_inference_job_queue`
+      - Time-Based Execution: Configured to run during off-hours on weekdays (6pm-7am) and all the time on weekends
+    - Concurrency and Safety:
+      - AdvisoryLock: Utility class for managing Postgres advisory locks
+      - Transaction Management: Context managers for safe lock acquisition and release
+      - ID Generation: Hash-based lock ID generation from string names
+    - Updated TDAMMTags to remove redundant tags (MMA_M_EM, MMA_O_BI, MMA_O_BH, MMA_O_N) and add a missing one (MMA_S_FBOT). Also updated the enum value for the NOT_TDAMM tag.
+    - Classification results coming in from the inference API will need to be translated to the TDAMMTags model we have, and that is handled by the `map_classification_to_tdamm_tags` in the `classification_utils` that will contain any relevant utilities for subsequent classifiers
+    - The collections that will be run through the pipeline are limited to the following right now:
+      - imagine_the_universe
+      - physics_of_the_cosmos
+      - stsci_space_telescope_science_institute
+    - Once the front end has been updated to allow for tag edits, all astrophysics collections will be marked to be run through the pipeline
+
+- 1298-csv-export-command-for-urls
+  - Description: Added a new Django management command to export URLs (DumpUrl, DeltaUrl, or CuratedUrl) to CSV files for analysis or backup purposes. The command allows filtering by collection and provides configurable export options.
+  - Changes:
+    - Created a new management command `export_urls_to_csv.py` to extract URL data to CSV format
+    - Implemented options to filter exports by model type and specific collections
+    - Added support for excluding full text content with the `--full_text` flag to reduce file size
+    - Included proper handling for paired fields (tdamm_tag_manual, tdamm_tag_ml)
+    - Added automatic creation of a dedicated `csv_exports` directory for storing export files
+    - Implemented batched processing to efficiently handle large datasets
+    - Added progress reporting during export operations

compose/local/django/start

@@ -1,3 +1,4 @@
+#compose/local/django/start
 #!/bin/bash
 
 set -o errexit

compose/production/django/Dockerfile

@@ -1,3 +1,4 @@
+# compose/production/django/Dockerfile
 # define an alias for the specfic python version used in this file.
 FROM python:3.10.14-slim-bullseye AS python
 

compose/production/django/start

@@ -1,3 +1,4 @@
+# compose/production/django/start
 #!/bin/bash
 
 set -o errexit

compose/production/traefik/traefik.yml

@@ -1,3 +1,4 @@
+# compose/production/traefik/traefik.yml
 log:
   level: INFO
 

config/celery.py

@@ -0,0 +1,15 @@
+# config/celery.py
+import os
+
+from celery import Celery
+
+# Set the default Django settings module
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.local")
+
+app = Celery("cosmos")
+
+# Configure Celery using Django settings
+app.config_from_object("django.conf:settings", namespace="CELERY")
+
+# Load task modules from all registered Django app configs
+app.autodiscover_tasks()

config/settings/base.py

@@ -84,6 +84,7 @@
     "feedback",
     "sde_collections",
     "sde_indexing_helper.users",
+    "inference",
 ]
 
 # https://docs.djangoproject.com/en/dev/ref/settings/#installed-apps
@@ -92,6 +93,7 @@
 CORS_ALLOWED_ORIGINS = [
     "http://localhost:3000",
     "http://sde-lrm.nasa-impact.net",
+    "https://sde-lrm.nasa-impact.net",
     "https://sde-qa.nasa-impact.net",
     "https://sciencediscoveryengine.test.nasa.gov",
     "https://sciencediscoveryengine.nasa.gov",
@@ -288,11 +290,9 @@
 # https://docs.celeryq.dev/en/stable/userguide/configuration.html#std:setting-result_serializer
 CELERY_RESULT_SERIALIZER = "json"
 # https://docs.celeryq.dev/en/stable/userguide/configuration.html#task-time-limit
-# TODO: set to whatever value is adequate in your circumstances
-CELERY_TASK_TIME_LIMIT = 5 * 60
+CELERY_TASK_TIME_LIMIT = 30 * 60
 # https://docs.celeryq.dev/en/stable/userguide/configuration.html#task-soft-time-limit
-# TODO: set to whatever value is adequate in your circumstances
-CELERY_TASK_SOFT_TIME_LIMIT = 60
+CELERY_TASK_SOFT_TIME_LIMIT = 25 * 60
 # https://docs.celeryq.dev/en/stable/userguide/configuration.html#beat-scheduler
 CELERY_BEAT_SCHEDULER = "django_celery_beat.schedulers:DatabaseScheduler"
 # https://docs.celeryq.dev/en/stable/userguide/configuration.html#worker-send-task-events
@@ -349,3 +349,5 @@
 LRM_QA_PASSWORD = env("LRM_QA_PASSWORD")
 LRM_DEV_TOKEN = env("LRM_DEV_TOKEN")
 XLI_TOKEN = env("XLI_TOKEN")
+INFERENCE_API_URL = env("INFERENCE_API_URL", default="http://host.docker.internal:8000")
+TDAMM_CLASSIFICATION_THRESHOLD = env("TDAMM_CLASSIFICATION_THRESHOLD", default="0.5")

config/settings/local.py

@@ -52,6 +52,11 @@
 
 # https://docs.celeryq.dev/en/stable/userguide/configuration.html#task-eager-propagates
 CELERY_TASK_EAGER_PROPAGATES = True
+
+# Inference API
+# ------------------------------------------------------------------------------
+INFERENCE_API_URL = env("INFERENCE_API_URL", default="http://host.docker.internal:8000")
+
 # Your stuff...
 # ------------------------------------------------------------------------------
 

config/settings/production.py

@@ -166,6 +166,9 @@
     traces_sample_rate=env.float("SENTRY_TRACES_SAMPLE_RATE", default=0.0),
 )
 
+# Inference API
+# ------------------------------------------------------------------------------
+INFERENCE_API_URL = env("INFERENCE_API_URL", default="http://172.17.0.1:8000")
 
 # Your stuff...
 # ------------------------------------------------------------------------------