Skip to content

Commit f871ebb

Browse files
dhanur-sharmadhanur-sharma
authored
Merge branch 'dev' into 1195-implement-unit-test-for-forms-on-the-frontend
2 parents b78f4bf + 24fb7eb commit f871ebb

File tree

6 files changed

+231
-5
lines changed

6 files changed

+231
-5
lines changed

‎.github/workflows/run_full_test_suite.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,5 +33,10 @@ jobs:
3333
DJANGO_ENV: test
3434
run: docker-compose -f local.yml run --rm django bash ./init.sh
3535

36+
- name: Generate Coverage Report
37+
env:
38+
DJANGO_ENV: test
39+
run: docker-compose -f local.yml run --rm django bash -c "coverage report"
40+
3641
- name: Cleanup
3742
run: docker-compose -f local.yml down --volumes

‎CHANGELOG.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,10 @@ For each PR made, an entry should be added to this changelog. It should contain
1212
- etc.
1313

1414
## Changelog
15+
16+
- 1217-add-data-validation-to-the-feedback-form-api-to-restrict-html-content
17+
- Description: The feedback form API does not currently have any form of data validation on the backend which makes it easy for the user with the endpoint to send in data with html tags. We need to have a validation scheme on the backend to protect this from happening.
18+
- Changes:
19+
- Defined a class `HTMLFreeCharField` which inherits `serializers.CharField`
20+
- Used regex to catch any HTML content comming in as an input to form fields
21+
- Called this class within the serializer for necessary fields

‎docs/architecture-decisions/testing_strategy.md‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
## Overview
2+
As of early 2025, we have only recently been writing tests for new features, and have about 250 tests in total, mostly centered around the EJ portal, the reindexing process, and pattern applications.
3+
4+
Although this covers much of the core system logic, there still remain a number of untested logical areas such as the config file generation, core project settings, frontend features, etc.
5+
6+
This document outlines a testing strategy for the project, which will guide us towards adding tests in the most critical areas first, followed by a plan to fully cover the remaining areas.
7+
8+
## Current Coverage
9+
Using the coverage library, the following report was generated:
10+
Name | Stmts | Miss | Cover | Missing
11+
----------|--------------------------------------------------------------------------------------------------|-------|--------|--------
12+
config/__init__.py | 2 | 0 | 100% |
13+
config/celery_app.py | 6 | 0 | 100% |
14+
config/settings/__init__.py | 0 | 0 | 100% |
15+
config/settings/base.py | 94 | 0 | 100% |
16+
config/settings/local.py | 20 | 20 | 0% | 1-65
17+
config/settings/production.py | 48 | 48 | 0% | 1-162
18+
config/urls.py | 14 | 4 | 71% | 26-47
19+
config/wsgi.py | 8 | 8 | 0% | 17-36
20+
config_generation/__init__.py | 0 | 0 | 100% |
21+
config_generation/api.py | 34 | 34 | 0% | 1-88
22+
config_generation/config_example.py | 15 | 15 | 0% | 1-69
23+
config_generation/db_to_xml.py | 203 | 133 | 34% | 45, 47, 50, 96, 119-125, 129-136, 142-149, 197-200, 206-214, 225-230, 242-271, 274-278, 285-292, 303-308, 311, 317, 326-332, 342-349, 361-368, 371-374, 377-378, 382-390, 393-399, 402-412, 415-429
24+
config_generation/db_to_xml_file_based.py | 52 | 52 | 0% | 4-119
25+
config_generation/delete_config_folders.py | 24 | 24 | 0% | 9-50
26+
config_generation/delete_server_content.py | 12 | 12 | 0% | 3-25
27+
config_generation/delete_webapp_collections.py | 5 | 5 | 0% | 6-12
28+
config_generation/export_collections.py | 36 | 36 | 0% | 1-73
29+
config_generation/export_whole_index.py | 28 | 28 | 0% | 1-58
30+
config_generation/generate_collection_list.py | 29 | 29 | 0% | 8-69
31+
config_generation/generate_commands.py | 41 | 41 | 0% | 6-87
32+
config_generation/generate_emac_indexer.py | 24 | 24 | 0% | 1-81
33+
config_generation/generate_jobs.py | 42 | 42 | 0% | 8-100
34+
config_generation/generate_scrapers.py | 15 | 15 | 0% | 2-54
35+
config_generation/minimum_api.py | 33 | 33 | 0% | 1-81
36+
config_generation/preprocess_sources.py | 25 | 25 | 0% | 1-50
37+
config_generation/sources_to_scrape.py | 28 | 28 | 0% | 2-1631
38+
docs/__init__.py | 0 | 0 | 100% |
39+
docs/conf.py | 17 | 17 | 0% | 13-62
40+
environmental_justice/__init__.py | 0 | 0 | 100% |
41+
environmental_justice/admin.py | 5 | 0 | 100% |
42+
environmental_justice/apps.py | 4 | 0 | 100% |
43+
environmental_justice/models.py | 29 | 1 | 97% | 44
44+
environmental_justice/serializers.py | 6 | 0 | 100% |
45+
environmental_justice/views.py | 23 | 0 | 100% |
46+
feedback/__init__.py | 0 | 0 | 100% |
47+
feedback/admin.py | 14 | 0 | 100% |
48+
feedback/apps.py | 4 | 0 | 100% |
49+
feedback/models.py | 42 | 15 | 64% | 20-29, 35-44, 61-63
50+
feedback/serializers.py | 10 | 0 | 100% |
51+
feedback/urls.py | 4 | 0 | 100% |
52+
feedback/views.py | 9 | 0 | 100% |
53+
manage.py | 16 | 16 | 0% | 2-31
54+
merge_production_dotenvs_in_dotenv.py | 15 | 1 | 93% | 26
55+
scripts/ej/cmr_processing.py | 241 | 5 | 98% | 160, 186-188, 397, 410
56+
scripts/ej/config.py | 6 | 0 | 100% |
57+
scripts/ej/test_cmr_processing.py | 225 | 1 | 99% | 610
58+
scripts/ej/test_threshold_processing.py | 97 | 1 | 99% | 209
59+
scripts/ej/threshold_processing.py | 20 | 0 | 100% |
60+
sde_collections/__init__.py | 0 | 0 | 100% |
61+
sde_collections/admin.py | 212 | 72 | 66% | 22-24, 29, 34, 40-60, 65-81, 86-89, 98-101, 110-112, 120-134, 143, 148, 153, 158, 163, 168, 173, 178-189, 196-197, 260, 265, 270, 275, 302-303, 308-309, 314-316, 345-372, 478-480
62+
sde_collections/apps.py | 4 | 0 | 100% |
63+
sde_collections/forms.py | 15 | 0 | 100% |
64+
sde_collections/management/commands/database_backup.py | 62 | 1 | 98% | 68
65+
sde_collections/management/commands/database_restore.py | 83 | 8 | 90% | 34, 36, 87-89, 142-145
66+
sde_collections/models/__init__.py | 0 | 0 | 100% |
67+
sde_collections/models/candidate_url.py | 89 | 16 | 82% | 124, 128-134, 138-142, 145, 176-177
68+
sde_collections/models/collection.py | 414 | 144 | 65% | 241, 269, 277-287, 291-301, 305-315, 319-344, 348-357, 361, 365, 369-376, 380-387, 394, 403-406, 419, 436-439, 449-470, 478, 482-515, 519, 523, 527, 531-532, 536, 540-546, 550-553, 558-567, 575-617, 640, 679, 689, 703, 707-732, 765, 769-777, 785
69+
sde_collections/models/collection_choice_fields.py | 138 | 20 | 86% | 14-17, 36-39, 56-59, 74-77, 168-171
70+
sde_collections/models/delta_patterns.py | 313 | 33 | 89% | 119, 123, 139, 226-227, 263, 267, 291, 382-389, 439-449, 498, 503-506, 592, 627-641
71+
sde_collections/models/delta_url.py | 81 | 19 | 77% | 117-125, 129-135, 139-143, 146
72+
sde_collections/models/pattern.py | 145 | 79 | 46% | 40-48, 56-63, 66, 69, 73-74, 78-79, 87, 94-96, 105, 117-119, 128, 139-151, 163-205, 208-212, 215-216, 230-233, 243, 257-260, 268
73+
sde_collections/serializers.py | 191 | 47 | 75% | 80-81, 84-85, 88-89, 92-93, 129-130, 133-134, 137-138, 141-142, 197, 201, 211-214, 244-247, 257-260, 271, 274, 307-315, 335-343, 358-366
74+
sde_collections/sinequa_api.py | 102 | 3 | 97% | 65, 255, 289
75+
sde_collections/tasks.py | 119 | 67 | 44% | 25-67, 72-108, 113-117, 122-125, 130-148, 153-155, 215-216
76+
sde_collections/urls.py | 17 | 0 | 100% |
77+
sde_collections/utils/__init__.py | 0 | 0 | 100% |
78+
sde_collections/utils/bulk_github_push.py | 8 | 8 | 0% | 7-22
79+
sde_collections/utils/generate_deployment_message.py | 8 | 8 | 0% | 1-24
80+
sde_collections/utils/github_helper.py | 115 | 93 | 19% | 12-18, 30-42, 49-52, 60-68, 81-96, 104-110, 119-123, 127-129, 132-142, 145-152, 155-172, 175, 178-185, 189-192, 196-224, 227
81+
sde_collections/utils/health_check.py | 123 | 106 | 14% | 33-46, 51-57, 61-98, 102-143, 155-165, 172-187, 191-273
82+
sde_collections/utils/paired_field_descriptor.py | 33 | 2 | 94% | 35, 52
83+
sde_collections/utils/slack_utils.py | 19 | 4 | 79% | 57-58, 66-67
84+
sde_collections/utils/title_resolver.py | 90 | 5 | 94% | 64, 75, 83, 85, 92
85+
sde_collections/views.py | 368 | 229 | 38% | 70, 82-89, 102-141, 144-187, 194, 208-212, 215-223, 226-237, 246, 249-251, 256-265, 273-277, 280-306, 309-315, 323-327, 330-336, 339-345, 353-355, 358-368, 410, 413-422, 430, 433-442, 450, 458, 461-475, 483, 486-490, 505-511, 523-530, 538-566, 577-583, 586-607, 610-613, 628-634
86+
sde_indexing_helper/__init__.py | 2 | 0 | 100% |
87+
sde_indexing_helper/conftest.py | 9 | 0 | 100% |
88+
sde_indexing_helper/contrib/__init__.py | 0 | 0 | 100% |
89+
sde_indexing_helper/contrib/sites/__init__.py | 0 | 0 | 100% |
90+
sde_indexing_helper/users/__init__.py | 0 | 0 | 100% |
91+
sde_indexing_helper/users/adapters.py | 11 | 11 | 0% | 1-16
92+
sde_indexing_helper/users/admin.py | 13 | 0 | 100% |
93+
sde_indexing_helper/users/apps.py | 10 | 0 | 100% |
94+
sde_indexing_helper/users/context_processors.py | 3 | 0 | 100% |
95+
sde_indexing_helper/users/forms.py | 15 | 0 | 100% |
96+
sde_indexing_helper/users/models.py | 10 | 0 | 100% |
97+
sde_indexing_helper/users/tasks.py | 6 | 0 | 100% |
98+
sde_indexing_helper/users/urls.py | 4 | 0 | 100% |
99+
sde_indexing_helper/users/views.py | 27 | 0 | 100% |
100+
sde_indexing_helper/utils/__init__.py | 0 | 0 | 100% |
101+
sde_indexing_helper/utils/exceptions.py | 7 | 0 | 100% |
102+
sde_indexing_helper/utils/storages.py | 7 | 7 | 0% | 1-11
103+
tests/test_merge_production_dotenvs_in_dotenv.py | 13 | 0 | 100 |%
104+
105+
## Critical Areas
106+
### Config Generation
107+
- config_generation/db_to_xml.py
108+
- update_or_add_element_value()
109+
- _update_config_xml()
110+
- convert_template_to_scraper()
111+
- add_document_type()
112+
- add_url_exclude()
113+
- add_title_mapping()
114+
- add_job_list_item()
115+
- get_tag_value()
116+
- fetch_treeroot()
117+
- fetch_document_type()
118+
- config_generation/generate_jobs.py
119+
- make_all_parallel_jobs()
120+
121+
### Models
122+
- environmental_justice/models.py
123+
- sde_collections/models/collection.py
124+
- clear_delta_urls()
125+
- clear_dump_urls()
126+
- refresh_url_lists_for_all_patterns ()
127+
- migrate_dump_to_delta ()
128+
- create_or_update_delta_url
129+
- promote_to_curate
130+
- add_to_public_query()
131+
- create_scraper_config()
132+
- create_indexer_config()
133+
- create_plugin_config()
134+
- _write_to_github()
135+
- update_config_xml()
136+
- apply_all_patterns()
137+
- create_configs_on_status_change()
138+
- sde_collections/models/collection_choice_fields.py
139+
- sde_collections/models/delta_patterns.py
140+
- sde_collections/models/delta_url.py
141+
- sde_collections/models/pattern.py
142+
- sde_indexing_helper/users/models.py
143+
144+
### Views
145+
- environmental_justice/views.py
146+
- sde_collections/views.py
147+
- sde_indexing_helper/users/views.py
148+
149+
### Serializers and APIs
150+
- environmental_justice/serializers.py
151+
- sde_collections/serializers.py
152+
153+
### Admin Interface
154+
- environmental_justice/admin.py
155+
- sde_collections/admin.py
156+
- fetch_full_text_lrm_dev_action()
157+
- fetch_full_text_xli_action()
158+
- sde_indexing_helper/users/admin.py
159+
160+
### Utilities and Helpers
161+
- sde_collections/utils/github_helper.py
162+
- sde_collections/utils/health_check.py
163+
- sde_collections/utils/title_resolver.py
164+
- sde_collections/utils/github_helper.py
165+
- fetch_metadata()
166+
- _get_contents_from_path()
167+
168+
### Task Automation and Background Jobs
169+
- sde_collections/tasks.py
170+
171+
### Key Operational Pipelines in the Repository
172+
The selection of critical areas for testing is guided by the following pipelines of the repository:
173+
1. Sinequa config files are generated
174+
2. COSMOS imports data from LRM Dev
175+
3. Imported data is processed
176+
4. Curators update URL metadata
177+
5. Sinequa reads results from the COSMOS APIs
178+
179+
### Critical Areas Lacking Tests
180+
- **Config Generation**: Config generation files are under-tested. Develop unit tests for all critical functions in the config_generation files.
181+
- **Project Settings**: Environment-specific configurations (`local.py`, `production.py`) have no tests.
182+
- **Frontend Features**: Currently, there are no tests covering frontend logic and interactions.
183+
- **Utilities and Helpers**: Essential utility modules like github_helper.py and health_check.py lack tests
184+

‎feedback/serializers.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,27 @@
1+
import re
2+
13
from rest_framework import serializers
24

35
from .models import ContentCurationRequest, Feedback
46

57

8+
class HTMLFreeCharField(serializers.CharField):
9+
def to_internal_value(self, data):
10+
value = super().to_internal_value(data)
11+
12+
if re.search(r"<[^>]+>", value):
13+
raise serializers.ValidationError("HTML tags are not allowed in this field")
14+
15+
return value
16+
17+
618
class FeedbackSerializer(serializers.ModelSerializer):
19+
20+
name = HTMLFreeCharField()
21+
subject = HTMLFreeCharField()
22+
comments = HTMLFreeCharField()
23+
source = HTMLFreeCharField()
24+
725
class Meta:
826
model = Feedback
927
fields = [

0 commit comments

Comments
 (0)