TODO: get away from server-side javascript libraries in forms

[tlambert03]

here's a proposal generated by claude:

Modern Architecture: Server-Side Validation + Client-Side Widgets

Key Principles

1. Server stays dumb about widgets - Django just renders standard HTML forms
2. Progressive enhancement - Works without JS, enhanced with JS
3. API-first - Clean JSON endpoints for autocomplete
4. Type-safe - TypeScript for the frontend
5. Framework-agnostic - Can use any modern widget library

---

1. Server Side: Clean Django Forms

Form Definition (No DAL dependencies!)

# backend/proteins/forms/microscope.py

from django import forms
from proteins.models import Filter, OpticalConfig

class OpticalConfigForm(forms.ModelForm):
    """Clean Django form - no widget library dependencies"""

    ex_filters = forms.ModelMultipleChoiceField(
        queryset=Filter.objects.all(),
        required=False,
        label="Excitation Filter(s)",
        # Just use standard Django widget - JS will enhance it
        widget=forms.SelectMultiple(attrs={
            'class': 'filter-select',  # Hook for JS enhancement
            'data-autocomplete-url': '/api/filters/autocomplete/',  # API endpoint
            'data-field-type': 'filter',  # Semantic type
        })
    )

    em_filters = forms.ModelMultipleChoiceField(
        queryset=Filter.objects.all(),
        required=False,
        label="Emission Filter(s)",
        widget=forms.SelectMultiple(attrs={
            'class': 'filter-select',
            'data-autocomplete-url': '/api/filters/autocomplete/',
            'data-field-type': 'filter',
        })
    )

    class Meta:
        model = OpticalConfig
        fields = ['name', 'laser', 'light', 'camera', 'comments']

    def clean_ex_filters(self):
        """Server-side validation - still works!"""
        filters = self.cleaned_data.get('ex_filters')
        if filters and filters.count() > 5:
            raise forms.ValidationError("Maximum 5 excitation filters allowed")
        return filters

    def save(self, commit=True):
        """Same clean business logic"""
        oc = super().save(commit=False)
        if commit:
            oc.save()
            # Handle M2M after save
            self.save_m2m()
            self._save_filter_placements()  # Custom logic
        return oc

    def _save_filter_placements(self):
        """Business logic stays on server"""
        # Same logic as before - create FilterPlacement instances
        # with proper path, reflects, etc.
        pass

Key Points:

• No DAL imports
• Standard Django SelectMultiple widget
• Data attributes provide hints to JS
• Full validation and business logic intact

---

2. API Endpoints: Clean REST/GraphQL

Option A: Django REST Framework

# backend/api/views.py

from rest_framework.decorators import api_view
from rest_framework.response import Response
from proteins.models import Filter

@api_view(['GET'])
def filter_autocomplete(request):
    """
    GET /api/filters/autocomplete/?q=semrock&limit=10

    Returns: {
        results: [
            {id: 45, name: "Semrock FF01-475/35", part: "FF01-475/35", ...},
            ...
        ],
        pagination: {more: false, total: 23}
    }
    """
    query = request.GET.get('q', '').strip()
    limit = int(request.GET.get('limit', 10))

    qs = Filter.objects.all()

    if query:
        # More sophisticated search
        qs = qs.filter(
            Q(name__icontains=query) |
            Q(part__icontains=query) |
            Q(manufacturer__name__icontains=query)
        ).distinct()

    total = qs.count()
    filters = qs[:limit].values('id', 'name', 'part', 'manufacturer__name')

    return Response({
        'results': list(filters),
        'pagination': {
            'more': total > limit,
            'total': total
        }
    })

Option B: GraphQL (if you're feeling fancy)

# backend/schema.py

import graphene
from graphene_django import DjangoObjectType

class FilterType(DjangoObjectType):
    class Meta:
        model = Filter
        fields = ['id', 'name', 'part', 'manufacturer']

class Query(graphene.ObjectType):
    filters = graphene.List(
        FilterType,
        search=graphene.String(),
        limit=graphene.Int(default_value=10)
    )

    def resolve_filters(self, info, search=None, limit=10):
        qs = Filter.objects.all()
        if search:
            qs = qs.filter(name__icontains=search)
        return qs[:limit]

Key Points:

• Clean, documented API
• Independent of form rendering
• Can be used by any client (web, mobile, etc.)
• Versioned if needed (/api/v1/...)

---

3. Client Side: Modern Widget Library

Let's use a hypothetical modern library, but I'll show patterns that work with any:

Installation

pnpm add @fpbase/autocomplete-select
# Or any modern library: tom-select, choices.js, etc.

TypeScript Widget Wrapper

// frontend/src/components/FilterSelect.ts

import { AutocompleteSelect } from '@fpbase/autocomplete-select'

interface FilterSelectOptions {
  element: HTMLSelectElement
  apiUrl: string
  multiple?: boolean
  placeholder?: string
}

export class FilterSelect {
  private widget: AutocompleteSelect
  private element: HTMLSelectElement

  constructor(options: FilterSelectOptions) {
    this.element = options.element

    // Initialize the widget
    this.widget = new AutocompleteSelect({
      element: options.element,

      // Fetch function for autocomplete
      async search(query: string) {
        const url = new URL(options.apiUrl, window.location.origin)
        url.searchParams.set('q', query)
        url.searchParams.set('limit', '20')

        const response = await fetch(url.toString())
        const data = await response.json()

        return data.results.map((filter: any) => ({
          value: filter.id.toString(),
          label: filter.name,
          meta: filter  // Store full object for rich display
        }))
      },

      // Custom rendering
      renderOption(option) {
        return `
          <div class="filter-option">
            <strong>${option.label}</strong>
            ${option.meta.part ? `<span class="part">${option.meta.part}</span>` : ''}
          </div>
        `
      },

      // Configuration
      multiple: options.multiple ?? false,
      placeholder: options.placeholder,
      closeOnSelect: !options.multiple,

      // Validation hooks
      validate: (values: string[]) => {
        if (values.length > 5) {
          return 'Maximum 5 filters allowed'
        }
        return null  // Valid
      }
    })

    // Sync back to original <select> for form submission
    this.widget.on('change', (values) => {
      this.syncToSelect(values)
    })
  }

  private syncToSelect(values: string[]) {
    // Clear existing selections
    Array.from(this.element.options).forEach(opt => {
      opt.selected = false
    })

    // Set new selections (or create options if needed)
    values.forEach(value => {
      let option = this.element.querySelector(`option[value="${value}"]`)
      if (!option) {
        option = document.createElement('option')
        option.value = value
        option.textContent = value  // Will be replaced by server
        this.element.appendChild(option)
      }
      option.selected = true
    })

    // Trigger change event for Django's benefit
    this.element.dispatchEvent(new Event('change', { bubbles: true }))
  }

  destroy() {
    this.widget.destroy()
  }
}

Auto-initialization

// frontend/src/microscope-form.ts

import { FilterSelect } from './components/FilterSelect'

// Auto-enhance all filter selects
document.addEventListener('DOMContentLoaded', () => {
  const filterSelects = document.querySelectorAll<HTMLSelectElement>('.filter-select')

  filterSelects.forEach(element => {
    const apiUrl = element.dataset.autocompleteUrl
    if (!apiUrl) return

    new FilterSelect({
      element,
      apiUrl,
      multiple: element.multiple,
      placeholder: element.dataset.placeholder || 'Select filters...'
    })
  })
})

---

4. HTML Output (Progressive Enhancement)

Django renders standard HTML:

<form method="post" action="/microscope/123/edit/">
  {% csrf_token %}

  <div class="form-group">
    <label for="id_optical_configs-0-ex_filters">
      Excitation Filter(s)
    </label>

    <!-- Standard HTML select - works without JS! -->
    <select
      name="optical_configs-0-ex_filters"
      id="id_optical_configs-0-ex_filters"
      class="filter-select"
      data-autocomplete-url="/api/filters/autocomplete/"
      data-field-type="filter"
      data-placeholder="Select filters..."
      multiple>

      <!-- Pre-selected options (if editing) -->
      {% for filter in form.initial.ex_filters %}
      <option value="{{ filter.id }}" selected>
        {{ filter.name }}
      </option>
      {% endfor %}
    </select>

    <!-- Django validation errors still work! -->
    {% if form.ex_filters.errors %}
      <div class="invalid-feedback d-block">
        {{ form.ex_filters.errors }}
      </div>
    {% endif %}
  </div>

  <!-- Other fields... -->

  <button type="submit">Save</button>
</form>

Progressive Enhancement:

1. Without JS: Plain <select multiple> - still functional
2. With JS: Enhanced autocomplete widget - better UX
3. Server validation: Always runs, regardless of client-side

---

5. Form Submission Flow

Client-side (optional enhancement)

// frontend/src/microscope-form.ts

const form = document.getElementById('microscopeform') as HTMLFormElement

form.addEventListener('submit', async (e) => {
  // Optional: client-side validation
  const exFilters = document.querySelectorAll('#id_optical_configs-0-ex_filters option:checked')
  if (exFilters.length > 5) {
    e.preventDefault()
    alert('Maximum 5 excitation filters allowed')
    return
  }

  // Optional: AJAX submission instead of full page reload
  e.preventDefault()

  const formData = new FormData(form)
  const response = await fetch(form.action, {
    method: 'POST',
    body: formData,
    headers: {
      'X-Requested-With': 'XMLHttpRequest',  // Tell Django it's AJAX
    }
  })

  if (response.ok) {
    const data = await response.json()
    if (data.success) {
      window.location.href = data.redirect_url
    } else {
      // Display server-side errors
      displayErrors(data.errors)
    }
  }
})

Server-side (same as before!)

def form_valid(self, form):
    """Server-side validation ALWAYS runs"""
    try:
        self.object = form.save()  # Full validation + business logic

        if is_ajax(self.request):
            return JsonResponse({
                'success': True,
                'redirect_url': self.get_success_url()
            })
        else:
            return redirect(self.get_success_url())

    except ValidationError as e:
        if is_ajax(self.request):
            return JsonResponse({
                'success': False,
                'errors': form.errors
            }, status=400)
        else:
            return self.form_invalid(form)

---

6. Comparison: Before vs After

Before (with DAL):

# Tightly coupled to DAL
from dal import autocomplete

widget=autocomplete.ModelSelect2Multiple(
    url="proteins:filter-autocomplete",  # Named URL
    attrs={'data-theme': 'bootstrap'}
)

Problems:

• Python dependency on JS library
• Hard to customize
• Tight coupling between backend and frontend
• Different autocomplete libraries require different Python packages

After (clean separation):

# Clean Django form
widget=forms.SelectMultiple(attrs={
    'class': 'filter-select',
    'data-autocomplete-url': '/api/filters/autocomplete/',
})

Benefits:

• ✅ No Python dependencies on JS libraries
• ✅ Swap widget library without touching Django
• ✅ API can be used by other clients
• ✅ Progressive enhancement
• ✅ TypeScript for type safety
• ✅ Server-side validation unchanged
• ✅ Easier testing (API and forms separate)

---

7. Migration Strategy

If you want to migrate gradually:

Step 1: Create API endpoints

# Keep DAL forms working, add new API
urlpatterns = [
    # Old DAL endpoints (keep for now)
    path('autocomplete/filter/', FilterAutocomplete.as_view(), ...),

    # New API endpoints
    path('api/filters/autocomplete/', filter_autocomplete),
]

Step 2: Update one form at a time

class OpticalConfigForm(forms.ModelForm):
    # New fields use standard widgets
    ex_filters = forms.ModelMultipleChoiceField(
        queryset=Filter.objects.all(),
        widget=forms.SelectMultiple(attrs={
            'class': 'filter-select',
            'data-autocomplete-url': '/api/filters/autocomplete/',
        })
    )

    # Old fields keep DAL (temporarily)
    bs_filters = MultipleFilterField("Dichroic Filter(s)")  # Still uses DAL

Step 3: Remove DAL when all forms migrated

pip uninstall django-autocomplete-light

---

Summary

Modern Stack:

Client:  TypeScript + Modern Widget Library (tom-select, choices.js, etc.)
         ↕ JSON
API:     Django REST Framework or GraphQL
         ↕ ORM
Forms:   Standard Django Forms (no DAL)
         ↕ Validation
Database: PostgreSQL

Key Wins:

1. Separation of concerns - API, forms, widgets are independent
2. Server-side validation - Never compromised
3. Framework flexibility - Switch widget libraries without touching Python
4. Progressive enhancement - Works without JS
5. Type safety - TypeScript on frontend
6. Testability - Each layer tests independently
7. Reusability - API can serve web, mobile, etc.

This is the modern way! No Python code should know about JavaScript widget libraries.