Merge branch 'dev' into 1052-update-cosmos-to-create-jobs-for-scrapers-and-indexers

Author: dhanur-sharma

.github/workflows/run_full_test_suite.yml

@@ -4,6 +4,8 @@ on:
   pull_request:
     branches:
       - dev
+    paths-ignore:
+      - '**/*.md'
 
 jobs:
   run-tests:
@@ -33,5 +35,10 @@ jobs:
           DJANGO_ENV: test
         run: docker-compose -f local.yml run --rm django bash ./init.sh
 
+      - name: Generate Coverage Report
+        env:
+          DJANGO_ENV: test
+        run: docker-compose -f local.yml run --rm django bash -c "coverage report"
+
       - name: Cleanup
         run: docker-compose -f local.yml down --volumes

CHANGELOG.md

@@ -12,13 +12,72 @@ For each PR made, an entry should be added to this changelog. It should contain
   - etc.
 
 ## Changelog
-
 - 1052-update-cosmos-to-create-jobs-for-scrapers-and-indexers
   - Description: The original automation set up to generate the scrapers and indexers automatically based on a collection workflow status change needed to be updated to more accurately reflect the curation workflow. It would also be good to generate the jobs during this process to streamline the same.
   - Changes:
     - Updated function nomenclature. Scrapers are Sinequa connector configurations that are used to scrape all the URLs prior to curation. Indexers are Sienqua connector configurations that are used to scrape the URLs post to curation, which would be used to index content on production. Jobs are used to trigger the connectors which are included as parts of joblists.
-    - Parameterized the convert_template_to_job method to include the job_source to streamline the value added to the <Collection> tag in the job XML.
+    - Parameterized the convert_template_to_job method to include the job_source to streamline the value added to the `<Collection>` tag in the job XML.
     - Updated the fields that are pertinenet to transfer from a scraper to an indexer. Also added a third level of XML processing to facilitate the same.
     - scraper_template.xml and indexer_template.xml now contains the templates used for the respective configuration generation.
     - Deleted the redundant webcrawler_initial_crawl.xml file.
     - Added and updated tests on workflow status triggers.
+
+- 2889-serialize-the-tdamm-tags
+  - Description: Have TDAMM serialzed in a specific way and exposed via the Curated URLs API to be consumed into SDE Test/Prod
+  - Changes:
+    - Changed `get_tdamm_tag` method in the `CuratedURLAPISerializer` to process the TDAMM tags and pass them to the API endpoint
+
+- 960-notifications-add-a-dropdown-with-options-on-the-feedback-form
+  - Description: Generate an API endpoint and publish all the dropdown options necessary as a list for LRM to consume it.
+  - Changes:
+    - Created a new model `FeedbackFormDropdown`
+    - Added the migration file
+    - Added the `dropdown_option` field to the `Feedback` model
+    - Updated the slack notification structure by adding the dropdown option text
+    - Created a new serializer called `FeedbackFormDropdownSerializer`
+    - Added a new API endpoint `feedback-form-dropdown-options-api/` where the list is going to be accesible
+    - Added a list view called `FeedbackFormDropdownListView`
+    - Added tests
+
+- 1217-add-data-validation-to-the-feedback-form-api-to-restrict-html-content
+  - Description: The feedback form API does not currently have any form of data validation on the backend which makes it easy for the user with the endpoint to send in data with html tags. We need to have a validation scheme on the backend to protect this from happening.
+  - Changes:
+    - Defined a class `HTMLFreeCharField` which inherits `serializers.CharField`
+    - Used regex to catch any HTML content comming in as an input to form fields
+    - Called this class within the serializer for necessary fields
+
+- 3227-bugfix-title-patterns-selecting-multi-url-pattern-does-nothing
+  - Description: When selecting options from the match pattern type filter, the system does not filter the results as expected. Instead of displaying only the chosen variety of patterns, it continues to show all patterns.
+  - Changes:
+    - In `title_patterns_table` definition, corrected the column reference
+    - Made `match_pattern_type` searchable
+    - Corrected the column references and made code consistent on all the other tables, i.e., `exclude_patterns_table`, `include_patterns_table`, `division_patterns_table` and `document_type_patterns_table`
+
+- 1190-add-tests-for-job-generation-pipeline
+  - Description: Tests have been added to enhance coverage for the config and job creation pipeline, alongside comprehensive tests for XML processing.
+  - Changes:
+    - Added config_generation/tests/test_config_generation_pipeline.py which tests the config and job generation pipeline, ensuring all components interact correctly
+    - config_generation/tests/test_db_to_xml.py is updated to include comprehensive tests for XML Processing
+
+- 1001-tests-for-critical-functionalities
+  - Description: Critical functionalities have been identified and listed, and critical areas lacking tests listed
+  - Changes:
+    - Integrated coverage.py as an indicative tool in the workflow for automated coverage reports on PRs, with separate display from test results.
+    - Introduced docs/architecture-decisions/testing_strategy.md, which includes the coverage report, lists critical areas, and specifically identifies those critical areas that are untested or under-tested.
+
+- 1192-finalize-the-infrastructure-for-frontend-testing
+  - Description: Set up comprehensive frontend testing infrastructure using Selenium WebDriver with Chrome, establishing a foundation for automated UI testing.
+  - Changes:
+    - Added Selenium testing dependency to `requirements/local.txt`
+    - Updated Dockerfile to support Chrome and ChromeDriver
+    - Created BaseTestCase and AuthenticationMixin for reusable test components
+    - Implemented core authentication test suite
+
+- 1195-implement-unit-test-for-forms-on-the-frontend
+  - Description: Implemented comprehensive frontend test suite covering authentication, collection management, search functionality, and pattern application forms.
+  - Changes:
+    - Added tests for authentication flows
+    - Implemented collection display and data table tests
+    - Added universal search functionality tests
+    - Created search pane filter tests
+    - Added pattern application form tests with validation checks

compose/local/django/Dockerfile

@@ -52,6 +52,7 @@ RUN apt-get update && apt-get install --no-install-recommends -y \
   && wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - \
   && apt-get update \
   && apt-get install -y postgresql-15 postgresql-client-15 \
+  && apt-get install -y chromium chromium-driver \
   # cleaning up unused files
   && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
   && rm -rf /var/lib/apt/lists/*

config_generation/tests/test_config_generation_pipeline.py

@@ -0,0 +1,90 @@
+from unittest.mock import MagicMock, call, patch
+
+from django.test import TestCase
+
+from sde_collections.models.collection import Collection
+from sde_collections.models.collection_choice_fields import WorkflowStatusChoices
+
+"""
+Workflow status change → Opens template → Applies XML transformation → Writes to GitHub.
+
+- When the `workflow_status` changes, it triggers the relevant config creation method.
+- The method reads an template and processes it using `XmlEditor`.
+- `XmlEditor` modifies the template by injecting collection-specific values and transformations.
+- The generated XML is passed to `_write_to_github()`, which commits it directly to GitHub.
+
+Note: This test verifies that the correct methods are triggered and XML content is passed to GitHub.
+The actual XML structure and correctness are tested separately in `test_db_xml.py`.
+"""
+
+
+class TestConfigCreation(TestCase):
+    def setUp(self):
+        self.collection = Collection.objects.create(
+            name="Test Collection", division="1", workflow_status=WorkflowStatusChoices.RESEARCH_IN_PROGRESS
+        )
+
+    @patch("sde_collections.utils.github_helper.GitHubHandler")  # Mock GitHubHandler
+    @patch("sde_collections.models.collection.Collection._write_to_github")
+    @patch("sde_collections.models.collection.XmlEditor")
+    def test_ready_for_engineering_triggers_config_and_job_creation(
+        self, MockXmlEditor, mock_write_to_github, MockGitHubHandler
+    ):
+        """
+        When the collection's workflow status is updated to READY_FOR_ENGINEERING,
+        it should trigger the creation of scraper configuration and job files.
+        """
+        # Mock GitHubHandler to avoid actual API calls
+        mock_github_instance = MockGitHubHandler.return_value
+        mock_github_instance.create_file.return_value = None
+        mock_github_instance.create_or_update_file.return_value = None
+
+        # Set up the XmlEditor mock for both config and job
+        mock_editor_instance = MockXmlEditor.return_value
+        mock_editor_instance.convert_template_to_scraper.return_value = "<scraper_config>config_data</scraper_config>"
+        mock_editor_instance.convert_template_to_job.return_value = "<scraper_job>job_data</scraper_job>"
+
+        # Simulate the status change to READY_FOR_ENGINEERING
+        self.collection.workflow_status = WorkflowStatusChoices.READY_FOR_ENGINEERING
+        self.collection.save()
+
+        # Verify that the XML for both config and job are generated and written to GitHub
+        expected_calls = [
+            call(self.collection._scraper_config_path, "<scraper_config>config_data</scraper_config>", False),
+            call(self.collection._scraper_job_path, "<scraper_job>job_data</scraper_job>", False),
+        ]
+        mock_write_to_github.assert_has_calls(expected_calls, any_order=True)
+
+    @patch("sde_collections.models.collection.GitHubHandler")  # Mock GitHubHandler in the correct module path
+    @patch("sde_collections.models.collection.Collection._write_to_github")
+    @patch("sde_collections.models.collection.XmlEditor")
+    def test_ready_for_curation_triggers_indexer_config_and_job_creation(
+        self, MockXmlEditor, mock_write_to_github, MockGitHubHandler
+    ):
+        """
+        When the collection's workflow status is updated to READY_FOR_CURATION,
+        it should trigger indexer config and job creation methods.
+        """
+        # Mock GitHubHandler to avoid actual API calls
+        mock_github_instance = MockGitHubHandler.return_value
+        mock_github_instance.check_file_exists.return_value = True  # Assume scraper exists
+        mock_github_instance._get_file_contents.return_value = MagicMock()
+        mock_github_instance._get_file_contents.return_value.decoded_content = (
+            b"<scraper_config>Mock Data</scraper_config>"
+        )
+
+        # Set up the XmlEditor mock for both config and job
+        mock_editor_instance = MockXmlEditor.return_value
+        mock_editor_instance.convert_template_to_indexer.return_value = "<indexer_config>config_data</indexer_config>"
+        mock_editor_instance.convert_template_to_job.return_value = "<indexer_job>job_data</indexer_job>"
+
+        # Simulate the status change to READY_FOR_CURATION
+        self.collection.workflow_status = WorkflowStatusChoices.READY_FOR_CURATION
+        self.collection.save()
+
+        # Verify that the XML for both indexer config and job are generated and written to GitHub
+        expected_calls = [
+            call(self.collection._indexer_config_path, "<indexer_config>config_data</indexer_config>", True),
+            call(self.collection._indexer_job_path, "<indexer_job>job_data</indexer_job>", False),
+        ]
+        mock_write_to_github.assert_has_calls(expected_calls, any_order=True)

config_generation/tests/test_db_to_xml.py

@@ -1,4 +1,7 @@
-import xml.etree.ElementTree as ET
+# docker-compose -f local.yml run --rm django pytest config_generation/tests/test_db_to_xml.py
+from xml.etree.ElementTree import ElementTree, ParseError, fromstring
+
+import pytest
 
 from ..db_to_xml import XmlEditor
 
@@ -28,39 +31,112 @@ def elements_equal(e1, e2):
             return False
         return all(elements_equal(c1, c2) for c1, c2 in zip(e1, e2))
 
-    tree1 = ET.fromstring(xml1)
-    tree2 = ET.fromstring(xml2)
-    return elements_equal(tree1, tree2)
+    tree1 = ElementTree(fromstring(xml1))
+    tree2 = ElementTree(fromstring(xml2))
 
+    return elements_equal(tree1.getroot(), tree2.getroot())
 
-def test_update_or_add_element_value():
-    xml_string = """<root>
-    <child>
-        <grandchild>old_value</grandchild>
-    </child>
-    </root>"""
 
+# Tests for valid and invalid XML initializations
+def test_valid_xml_initialization():
+    xml_string = "<root><child>Test</child></root>"
     editor = XmlEditor(xml_string)
+    assert editor.get_tag_value("child") == ["Test"]
 
-    # To update an existing element's value
-    updated_xml = editor.update_or_add_element_value("child/grandchild", "new_value")
-    expected_output = """<root>
-        <child>
-            <grandchild>new_value</grandchild>
-        </child>
-    </root>
-    """
-    assert xmls_equal(updated_xml, expected_output)
-
-    # To create a new element and set its value
-    new_xml = editor.update_or_add_element_value("newchild", "some_value")
-    expected_output = """<root>
-        <child>
-            <grandchild>new_value</grandchild>
-        </child>
-        <newchild>
-            some_value
-        </newchild>
-    </root>
-    """
-    assert xmls_equal(new_xml, expected_output)
+
+def test_invalid_xml_initialization():
+    with pytest.raises(ParseError):
+        XmlEditor("<root><child></root>")
+
+
+# Test retrieval of single and multiple tag values
+def test_get_single_tag_value():
+    xml_string = "<root><child>Test</child></root>"
+    editor = XmlEditor(xml_string)
+    assert editor.get_tag_value("child", strict=True) == "Test"
+
+
+def test_get_nonexistent_tag_value():
+    xml_string = "<root><child>Test</child></root>"
+    editor = XmlEditor(xml_string)
+    assert editor.get_tag_value("nonexistent", strict=False) == []
+
+
+def test_get_tag_value_strict_multiple_elements():
+    xml_string = "<root><child>One</child><child>Two</child></root>"
+    editor = XmlEditor(xml_string)
+    with pytest.raises(ValueError):
+        editor.get_tag_value("child", strict=True)
+
+
+# Test updating and adding XML elements
+def test_update_existing_element():
+    xml_string = "<root><child>Old</child></root>"
+    editor = XmlEditor(xml_string)
+    editor.update_or_add_element_value("child", "New")
+    updated_xml = editor.update_config_xml()
+    assert "New" in updated_xml and "Old" not in updated_xml
+
+
+def test_add_new_element():
+    xml_string = "<root></root>"
+    editor = XmlEditor(xml_string)
+    editor.update_or_add_element_value("newchild", "Value")
+    updated_xml = editor.update_config_xml()
+    assert "Value" in updated_xml and "<newchild>Value</newchild>" in updated_xml
+
+
+def test_add_third_level_hierarchy():
+    xml_string = "<root></root>"
+    editor = XmlEditor(xml_string)
+    editor.update_or_add_element_value("parent/child/grandchild", "DeeplyNested")
+    updated_xml = editor.update_config_xml()
+    root = fromstring(updated_xml)
+    grandchild = root.find(".//grandchild")
+    assert grandchild is not None, "Grandchild element not found"
+    assert grandchild.text == "DeeplyNested", "Grandchild does not contain the correct text"
+
+    # Check complete path
+    parent = root.find(".//parent/child/grandchild")
+    assert parent is not None, "Complete path to grandchild not found"
+    assert parent.text == "DeeplyNested", "Complete path to grandchild does not contain correct text"
+
+
+# Test transformations and generic mapping
+def test_convert_indexer_to_scraper_transformation():
+    xml_string = """<root><Plugin>Indexer</Plugin></root>"""
+    editor = XmlEditor(xml_string)
+    editor.convert_indexer_to_scraper()
+    updated_xml = editor.update_config_xml()
+    assert "<Plugin>SMD_Plugins/Sinequa.Plugin.ListCandidateUrls</Plugin>" in updated_xml
+    assert "<Plugin>Indexer</Plugin>" not in updated_xml
+
+
+def test_generic_mapping_addition():
+    xml_string = "<root></root>"
+    editor = XmlEditor(xml_string)
+    editor._generic_mapping(name="id", value="doc.url1", selection="url1")
+    updated_xml = editor.update_config_xml()
+    assert "<Mapping>" in updated_xml
+    assert "<Name>id</Name>" in updated_xml
+    assert "<Value>doc.url1</Value>" in updated_xml
+
+
+# Test XML serialization with headers
+def test_xml_serialization_with_header():
+    xml_string = "<root><child>Value</child></root>"
+    editor = XmlEditor(xml_string)
+    xml_output = editor.update_config_xml()
+    assert '<?xml version="1.0" encoding="utf-8"?>' in xml_output
+    assert "<root>" in xml_output and "<child>Value</child>" in xml_output
+
+
+# Test handling multiple changes accumulation
+def test_multiple_changes_accumulation():
+    xml_string = "<root><child>Initial</child></root>"
+    editor = XmlEditor(xml_string)
+    editor.update_or_add_element_value("child", "Modified")
+    editor.update_or_add_element_value("newchild", "Added")
+    updated_xml = editor.update_config_xml()
+    assert "Modified" in updated_xml and "Added" in updated_xml
+    assert "Initial" not in updated_xml