Merge branch 'dev' into 960-notifications-add-a-dropdown-with-options-on-the-feedback-form

Author: dhanur-sharma

.github/workflows/run_full_test_suite.yml

@@ -33,5 +33,10 @@ jobs:
           DJANGO_ENV: test
         run: docker-compose -f local.yml run --rm django bash ./init.sh
 
+      - name: Generate Coverage Report
+        env:
+          DJANGO_ENV: test
+        run: docker-compose -f local.yml run --rm django bash -c "coverage report"
+
       - name: Cleanup
         run: docker-compose -f local.yml down --volumes

CHANGELOG.md

@@ -23,3 +23,10 @@ For each PR made, an entry should be added to this changelog. It should contain
     - Added a new API endpoint `feedback-form-dropdown-options-api/` where the list is going to be accesible
     - Added a list view called `FeedbackFormDropdownListView`
     - Added tests
+
+- 1217-add-data-validation-to-the-feedback-form-api-to-restrict-html-content
+  - Description: The feedback form API does not currently have any form of data validation on the backend which makes it easy for the user with the endpoint to send in data with html tags. We need to have a validation scheme on the backend to protect this from happening.
+  - Changes:
+    - Defined a class `HTMLFreeCharField` which inherits `serializers.CharField`
+    - Used regex to catch any HTML content comming in as an input to form fields
+    - Called this class within the serializer for necessary fields

compose/local/django/Dockerfile

@@ -52,6 +52,7 @@ RUN apt-get update && apt-get install --no-install-recommends -y \
   && wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - \
   && apt-get update \
   && apt-get install -y postgresql-15 postgresql-client-15 \
+  && apt-get install -y chromium chromium-driver \
   # cleaning up unused files
   && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
   && rm -rf /var/lib/apt/lists/*

docs/architecture-decisions/testing_strategy.md

@@ -0,0 +1,283 @@
+# Frontend Testing Methodologies for Django Projects
+## Overview
+
+This document outlines testing methodologies for Django projects with HTML forms and JavaScript enhancements, focusing on Python-based testing solutions. While going through the codebase, I can see it is primarily a JavaScript-heavy frontend that uses plain HTML forms enhanced with JavaScript/jQuery rather than server-rendered Django forms. Django forms are being used only in the admin panel of the project.
+
+## Primary Testing Tools
+
+### 1. Selenium with Python (Chosen)
+
+#### Capabilities
+- Full browser automation
+- JavaScript execution support
+- Real DOM interaction
+- Cross-browser testing
+- Modal dialog handling
+- AJAX request testing
+- File upload testing
+- DataTables interaction
+
+#### Implementation
+```python
+from selenium import webdriver
+from selenium.webdriver.common.by import By
+from selenium.webdriver.support.ui import WebDriverWait
+from selenium.webdriver.support import expected_conditions as EC
+
+class TestCollectionDetail:
+    def setup_method(self):
+        self.driver = webdriver.Chrome()
+        self.wait = WebDriverWait(self.driver, 10)
+
+    def test_title_change_modal(self):
+        # Example test for title change modal
+        self.driver.get("/collections/1/")
+        title_button = self.wait.until(
+            EC.element_to_be_clickable((By.ID, "change-title-btn"))
+        )
+        title_button.click()
+        
+        modal = self.wait.until(
+            EC.visibility_of_element_located((By.ID, "title-modal"))
+        )
+        
+        form = modal.find_element(By.TAG_NAME, "form")
+        input_field = form.find_element(By.NAME, "title")
+        input_field.send_keys("New Title")
+        form.submit()
+        
+        # Wait for AJAX completion
+        self.wait.until(
+            EC.text_to_be_present_in_element((By.ID, "collection-title"), "New Title")
+        )
+```
+
+#### Pros
+- Complete end-to-end testing
+- Real browser interaction
+- JavaScript support
+- Comprehensive API
+- Strong community support
+
+#### Drawbacks
+- Slower execution
+- Browser dependencies
+- More complex setup
+- Can be flaky with timing issues
+
+### 2. pytest-django with django-test-client
+
+#### Capabilities
+- Form submission testing
+- Response validation
+- Header verification
+- Status code checking
+- Session handling
+- Template rendering testing
+
+#### Implementation
+```python
+import pytest
+from django.urls import reverse
+
+@pytest.mark.django_db
+class TestCollectionForms:
+    def test_collection_create(self, client):
+        url = reverse('collection_create')
+        data = {
+            'title': 'Test Collection',
+            'division': 'division1',
+            'workflow_status': 'active'
+        }
+        response = client.post(url, data)
+        assert response.status_code == 302  # Redirect after success
+        
+        # Verify creation
+        response = client.get(reverse('collection_detail', kwargs={'pk': 1}))
+        assert 'Test Collection' in response.content.decode()
+```
+
+#### Pros
+- Fast execution
+- No browser dependency
+- Simpler setup
+- Integrated with Django
+
+#### Drawbacks
+- **No JavaScript support (Dealbreaker)**
+- Limited DOM interaction
+- Can't test real user interactions
+
+### 3. Playwright for Python
+
+#### Capabilities
+- Modern browser automation
+- Async/await support
+- Network interception
+- Mobile device emulation
+- Automatic waiting
+- Screenshot and video capture
+
+#### Implementation
+```python
+from playwright.sync_api import sync_playwright
+
+def test_modal_form_submission():
+    with sync_playwright() as p:
+        browser = p.chromium.launch()
+        page = browser.new_page()
+        
+        page.goto("/collections/")
+        
+        # Click button to open modal
+        page.click("#add-collection-btn")
+        
+        # Fill form in modal
+        page.fill("#title-input", "New Collection")
+        page.fill("#division-input", "Division A")
+        
+        # Submit form
+        page.click("#submit-btn")
+        
+        # Wait for success message
+        success_message = page.wait_for_selector(".toast-success")
+        assert "Collection created" in success_message.text_content()
+        
+        browser.close()
+```
+
+#### Pros
+- Modern API design
+- Better stability than Selenium
+- Built-in async support
+- Powerful debugging tools
+
+#### Drawbacks
+- **Newer tool, smaller community (Dealbreaker)**
+- Additional system dependencies
+- Learning curve for async features
+
+
+### 4. Beautiful Soup with Requests
+A combination for testing HTML structure and content.
+
+**Capabilities:**
+- HTML parsing and validation
+- Content extraction
+- Structure verification
+- Link checking
+- Form field validation
+- Template testing
+
+**Pros:**
+- Lightweight solution
+- Flexible HTML parsing
+- No browser dependency
+- Fast execution
+- Simple API
+- Low resource usage
+
+**Drawbacks:**
+- **No JavaScript support (Dealbreaker)**
+- Limited interaction testing
+- No visual testing
+- Basic functionality only
+- No real browser simulation
+
+## Feature Comparison Table
+
+| Feature                    | Selenium | Django Test Client | Playwright | Beautiful Soup |
+|---------------------------|----------|-------------------|------------|----------------|
+| JavaScript Support        | ✅ Yes    | ❌ No             | ✅ Yes      | ❌ No          |
+| Setup Complexity          | 🟡 Medium   | 🟢 Low            | 🟡 Medium   | 🟢 Low         |
+| Execution Speed           | 🔴 Slow   | 🟢 Fast           | 🟡 Medium   | 🟢 Fast        |
+| Modal Testing             | ✅ Yes    | ❌ No             | ✅ Yes      | ❌ No          |
+| AJAX Testing              | ✅ Yes    | ❌ No             | ✅ Yes      | ❌ No          |
+| Cross-browser Testing     | ✅ Yes    | ❌ No             | ✅ Yes      | ❌ No          |
+| Real User Interaction     | ✅ Yes    | ❌ No             | ✅ Yes      | ❌ No          |
+| Documentation Quality    | ✅ Excellent| ✅ Good          | ✅ Good     | ✅ Good        |
+| Community Support        | ✅ Large   | ✅ Large          | 🟡 Growing  | ✅ Large       |
+
+## Testing Strategy Recommendations
+
+1. **Primary Testing Tool**: Selenium with Python
+   - Best suited for your JavaScript-heavy interface
+   - Handles modals and AJAX naturally
+   - Extensive documentation and community support
+
+2. **Test Coverage Areas**:
+   - Modal form interactions
+   - AJAX submissions
+   - DataTables functionality
+   - Form validation
+   - Success/error messages
+   - URL routing
+   - DOM updates
+
+## Implementation Steps
+
+1. Add testing dependencies to requirements file `requirements/local.txt`:
+```text
+# Testing Dependencies
+selenium>=4.15.2
+pytest-xdist>=3.3.1
+pytest-cov>=4.1.0
+```
+
+2. Update Dockerfile `compose/local/django/Dockerfile` to install Chrome and ChromeDriver:
+```dockerfile
+# Install Chrome and ChromeDriver for Selenium tests
+RUN apt-get update && apt-get install -y \
+    chromium \
+    chromium-driver \
+    && rm -rf /var/lib/apt/lists/*
+```
+
+3. Rebuild Docker container to apply changes:
+```bash
+docker-compose -f local.yml build django
+```
+
+4. Create test directory structure:
+```bash
+mkdir -p tests/frontend
+touch tests/frontend/__init__.py
+touch tests/frontend/base.py
+touch tests/frontend/test_setup.py
+```
+
+5. Create base test classes:
+```python
+import pytest
+from selenium import webdriver
+
+class BaseUITest:
+    @pytest.fixture(autouse=True)
+    def setup_class(self):
+        self.driver = webdriver.Chrome()
+        yield
+        self.driver.quit()
+
+    def login(self):
+        # Common login logic
+        pass
+```
+
+6. Organize tests by feature:
+```python
+class TestCollectionManagement(BaseUITest):
+    def test_create_collection(self):
+        pass
+
+    def test_edit_collection(self):
+        pass
+
+class TestURLPatterns(BaseUITest):
+    def test_add_include_pattern(self):
+        pass
+```
+
+7. Run tests:
+```bash
+docker-compose -f local.yml run --rm django pytest tests/frontend/test_setup.py -v
+```

docs/test/frontend_testing_methodologies.md

@@ -1,3 +1,5 @@
+import re
+
 from rest_framework import serializers
 
 from .models import ContentCurationRequest, Feedback, FeedbackFormDropdown
@@ -9,7 +11,23 @@ class Meta:
         fields = ["id", "name"]
 
 
+class HTMLFreeCharField(serializers.CharField):
+    def to_internal_value(self, data):
+        value = super().to_internal_value(data)
+
+        if re.search(r"<[^>]+>", value):
+            raise serializers.ValidationError("HTML tags are not allowed in this field")
+
+        return value
+
+
 class FeedbackSerializer(serializers.ModelSerializer):
+
+    name = HTMLFreeCharField()
+    subject = HTMLFreeCharField()
+    comments = HTMLFreeCharField()
+    source = HTMLFreeCharField()
+
     class Meta:
         model = Feedback
         fields = [

feedback/serializers.py

@@ -1,5 +1,5 @@
 #!/bin/bash
-echo "Running all test cases across the project..."
+echo "Running all test cases across the project with coverage analysis..."
 
 # Initialize a failure counter
 failure_count=0
@@ -10,10 +10,20 @@ excluded_dirs="document_classifier functional_tests"
 # Find all test files except those in excluded directories
 test_files=$(find . -type f -name "test_*.py" | grep -Ev "$(echo $excluded_dirs | sed 's/ /|/g')")
 
-# Run each test file
+coverage erase  # Clear any existing coverage data
+
+# Setup .coveragerc configuration to include all Python files
+echo "[run]
+source = .
+include = */*.py
+
+[report]
+show_missing = True" > .coveragerc
+
+# Run each test file with coverage (without generating report yet)
 for test_file in $test_files; do
     echo "Running $test_file..."
-    pytest "$test_file"
+    coverage run --append -m pytest "$test_file"  # Collect coverage data
 
     # Check the exit status of pytest
     if [ $? -ne 0 ]; then
@@ -22,10 +32,11 @@ for test_file in $test_files; do
     fi
 done
 
-# Report the results
+# Report the results without generating the coverage report
 if [ $failure_count -ne 0 ]; then
-    echo "$failure_count test(s) failed."
+    echo "$failure_count test(s) failed. Refer to the terminal output for details."
     exit 1
 else
     echo "All tests passed successfully!"
+    echo "Coverage data collected. Coverage report will be generated separately."
 fi

init.sh

@@ -13,6 +13,10 @@ pytest==8.0.0  # https://github.com/pytest-dev/pytest
 pytest-sugar==1.0.0  # https://github.com/Frozenball/pytest-sugar
 types-requests # maybe instead, we should add `mypy --install-types` to the dockerfile?
 types-xmltodict
+pytest-xdist>=3.3.1
+pytest-cov>=4.1.0
+selenium>=4.15.2    # Selenium (Frontend Testing)
+coverage==7.4.1
 
 # Documentation
 # ------------------------------------------------------------------------------