Merge upstream commits (#534)

* Add invalid ID tests for `main.views` (lucyparsons#1124)

There were a handful of routes where we did not validate the initial ID
parameters. Additionally, I standardized how object querying was done
within the `main.views` file.

- [x] This branch is up-to-date with the `develop` branch.
- [x] `pytest` passes on my local development environment.
- [x] `pre-commit` passes on my local development environment.

* Remove unnecessary `mockdata` parameters (lucyparsons#1126)

lucyparsons#1125

Removed the `mockdata` parameter from test functions that don't require
it.

- [x] This branch is up-to-date with the `develop` branch.
- [x] `pytest` passes on my local development environment.
- [x] `pre-commit` passes on my local development environment.

* Finish audit fields for `users` table (lucyparsons#1129)

lucyparsons#1004

Added `disabled_by`, `disabled_at`, `approved_at`, `approved_by`, and
`confirmed_at`, and `confirmed_by` fields to the `users` table so that
we could keep track of users actions on the platform at a more granular
level. I also removed the previously mentioned columns corresponding
boolean columns.

- [x] This branch is up-to-date with the `develop` branch.
- [x] `pytest` passes on my local development environment.
- [x] `pre-commit` passes on my local development environment.
- [x] The `flask db upgrade` and `flask db downgrade` functions work for
both migrations.

```console
I have no name!@c38d30934aa7:/usr/src/app$ %
(env) michaelp@MacBook-Air-18 OpenOversight % docker exec -it openoversight-web-1 bash
I have no name!@5027495d31c0:/usr/src/app$ flask db downgrade
...
INFO  [alembic.runtime.migration] Running downgrade 99c50fc8d294 -> bf254c0961ca, complete audit field addition to users
I have no name!@5027495d31c0:/usr/src/app$ flask db downgrade
...
INFO  [alembic.runtime.migration] Running downgrade bf254c0961ca -> 5865f488470c, add remaining audit fields for users table
I have no name!@5027495d31c0:/usr/src/app$ flask db downgrade
...
INFO  [alembic.runtime.migration] Running downgrade 5865f488470c -> 939ea0f2b26d, change salary column types
I have no name!@5027495d31c0:/usr/src/app$ flask db upgrade
...
INFO  [alembic.runtime.migration] Running upgrade 939ea0f2b26d -> 5865f488470c, change salary column types
INFO  [alembic.runtime.migration] Running upgrade 5865f488470c -> bf254c0961ca, add remaining audit fields for users table
INFO  [alembic.runtime.migration] Running upgrade bf254c0961ca -> 99c50fc8d294, complete audit field addition to users
I have no name!@5027495d31c0:/usr/src/app$
```

* Fix the `/find` department sort (lucyparsons#1133)

## Fixes issue
Addresses TO-DO in code.

## Description of Changes
The `/find` route was using an unsorted list of departments unlike the
rest of the application.

Before:
<img width="1334" alt="Screenshot 2024-11-06 at 10 13 13 PM"
src="https://github.com/user-attachments/assets/8c6b832a-109e-465a-a6ec-c102524eb43b">

After:
<img width="1324" alt="Screenshot 2024-11-06 at 10 13 36 PM"
src="https://github.com/user-attachments/assets/1f639518-a750-498e-8a26-6f5784ad052b">

## Tests and Linting
- [x] This branch is up-to-date with the `develop` branch.
- [x] `pytest` passes on my local development environment.
- [x] `pre-commit` passes on my local development environment.

* Standardize model `__repr__` functions (lucyparsons#1140)

## Description of Changes
Addressed inconsistencies with the model `__repr__` functions.

Mostly making this because this PR got a bit out of control:
lucyparsons#1138

## Tests and Linting
- [x] This branch is up-to-date with the `develop` branch.
- [x] `pytest` passes on my local development environment.
- [x] `pre-commit` passes on my local development environment.

* Restructure application routing (lucyparsons#1141)

Sets the groundwork for this issue:
lucyparsons#385

Simplifying the import logic for the `auth` and `main` routes of the
application. This also sets up the work for this PR:
lucyparsons#1138

- [x] This branch is up-to-date with the `develop` branch.
- [x] `pytest` passes on my local development environment.
- [x] `pre-commit` passes on my local development environment.

* Add generic `__repr__` and `to_dict` functions (lucyparsons#1143)

Adds a generic `__repr__` function and and a generic `to_dict` function
so that objects can be serialized in the future. The reason I didn't use
the SQLAlchemy class like we talked about in the previous PR was because
that would require a complete rework of most object instantiations and
how properties are ordered. For a greater view into that, you can look
at this PR: lucyparsons#1142

- [x] This branch is up-to-date with the `develop` branch.
- [x] `pytest` passes on my local development environment.
- [x] `pre-commit` passes on my local development environment.

* Add static `Department` functions (lucyparsons#1150)

Moved `Department`-related methods to the `Department` object.

- [x] This branch is up-to-date with the `develop` branch.
- [x] Ran `make create_db_diagram` command.
- [x] `pytest` passes on my local development environment.
- [x] `pre-commit` passes on my local development environment.

* Remove audit fields from `Description` download (lucyparsons#1151)

## Description of Changes
Removed audit fields from the `Description` download function. They leak
user information and provide data that isn't particularly useful outside
of internal auditing.

## Tests and Linting
- [x] This branch is up-to-date with the `develop` branch.
- [x] Ran `make create_db_diagram` command.
- [x] `pytest` passes on my local development environment.
- [x] `pre-commit` passes on my local development environment.

* UX Re-design (lucyparsons#1155)

* Fix submit image page and python sqlite bug (lucyparsons#1157)

## Description of Changes

_Changes carried over from
https://github.com/OrcaCollective/OpenOversight/pull/533_

Fix bug where submit image page department selector was not submitting
images to the correct department.

It looks like the current implementation was using `selectedIndex` (the
index of the selected `<option>` element in the `<select>`) to determine
the initial department id, which assumes the departments are loaded in
order:


https://github.com/OrcaCollective/OpenOversight/blob/2c5ad9a74687943eac5e71a874c0f4ceb4784fa0/OpenOversight/app/templates/submit_image.html#L79-L82

## Notes for Deployment
None!

## Screenshots (if appropriate)
N/A

## Testing instructions
1. Log in as admin user.
2. Create a new department "Peoria Police Department (PPD)" at
http://localhost:3000/departments/new.
3. Update the user's preferred department to PPD at
http://localhost:3000/auth/change-dept/.
4. Visit the Submit Image page (http://localhost:3000/submit) and
confirm that the populated department is BPD.
5. Upload an image.
6. Verify in devtools that an image was submitted to
http://localhost:3000/upload/departments/4.
7. Update the department selector to Springfield Police Department.
8. Upload an image.
9. Verify in devtools that an image was submitted to
http://localhost:3000/upload/departments/1.
10. Change the preferred department to Springfield Police Department at
http://localhost:3000/auth/change-dept/.
11. Visit the Submit Image page (http://localhost:3000/submit) and
confirm that the populated department is now SPD.
12. Upload an image.
13. Verify in devtools that an image was submitted to
http://localhost:3000/upload/departments/1.

## Checks

 - [x] I have rebased my changes on `main`

 - [x] `just lint` passes

 - [x] `just test` passes

* Default to "N/A" if dept state is invalid (lucyparsons#1159)

## Description of Changes
Default to "N/A" if department state is invalid

In [the
migration](https://github.com/lucyparsons/OpenOversight/blob/develop/OpenOversight/migrations/versions/2023-07-26-1551_18f43ac4622f_add_state_column_to_departments.py)
where we introduced the department state column, we defaulted to empty
string for existing departments. We think this may have broken the
browse page when trying to load the browse page.

## Notes for Deployment
None!

## Screenshots (if appropriate)
N/A

## Tests and Linting
Reproduced the internal server error by setting the department state to
empty string
```
openoversight-dev=# update departments set state='' where id = 1;                                                                                                                       
```

Checked that the error was no longer displayed once this change was
implemented

- [x] This branch is up-to-date with the `develop` branch.
- [ ] Ran `make create_db_diagram` command.
- [x] `pytest` passes on my local development environment.
- [x] `pre-commit` passes on my local development environment.

* Fix department selector on submit image page

* Re-add dropzone

* Update contact email

---------

Co-authored-by: Michael Plunkett <[email protected]>

Author: sea-kelp

OpenOversight/app/__init__.py

@@ -6,19 +6,19 @@
 from flask import Flask, jsonify, render_template, request
 from flask_bootstrap import Bootstrap5
 from flask_compress import Compress
-from flask_limiter import Limiter
-from flask_limiter.util import get_remote_address
 from flask_login import LoginManager
 from flask_migrate import Migrate
-from flask_sitemap import Sitemap
 from flask_wtf.csrf import CSRFProtect
 
+from OpenOversight.app.auth.views import auth as auth_blueprint
 from OpenOversight.app.email_client import EmailClient
 from OpenOversight.app.filters import instantiate_filters
+from OpenOversight.app.main.views import main as main_blueprint
 from OpenOversight.app.models.config import config
 from OpenOversight.app.models.database import db
 from OpenOversight.app.models.users import AnonymousUser
 from OpenOversight.app.utils.constants import MEGABYTE
+from OpenOversight.app.utils.flask import limiter, sitemap
 
 
 bootstrap = Bootstrap5()
@@ -29,11 +29,6 @@
 login_manager.anonymous_user = AnonymousUser
 login_manager.login_view = "auth.login"
 
-limiter = Limiter(
-    key_func=get_remote_address, default_limits=["100 per minute", "5 per second"]
-)
-
-sitemap = Sitemap()
 csrf = CSRFProtect()
 
 
@@ -52,15 +47,11 @@ def create_app(config_name="default"):
     sitemap.init_app(app)
     compress.init_app(app)
 
-    from OpenOversight.app.main import main as main_blueprint
-
+    # Register Blueprints for application routes
+    app.register_blueprint(auth_blueprint)
     app.register_blueprint(main_blueprint)
 
-    from OpenOversight.app.auth import auth as auth_blueprint
-
-    app.register_blueprint(auth_blueprint, url_prefix="/auth")
-
-    max_log_size = 10 * MEGABYTE  # start new log file after 10 MB
+    max_log_size = 10 * MEGABYTE  # Start new log file after 10 MB
     num_logs_to_keep = 5
     file_handler = RotatingFileHandler(
         "/tmp/openoversight.log", "a", max_log_size, num_logs_to_keep
@@ -77,7 +68,7 @@ def create_app(config_name="default"):
     app.logger.addHandler(file_handler)
     app.logger.info("OpenOversight startup")
 
-    # Also log when endpoints are getting hit hard
+    # Log when endpoints are getting hit hard
     limiter.logger.addHandler(file_handler)
 
     # Define error handlers

OpenOversight/app/auth/__init__.py

@@ -1,6 +0,0 @@
-from flask import Blueprint
-
-
-auth = Blueprint("auth", __name__)
-
-from . import views  # noqa: F401, E402

OpenOversight/app/auth/views.py

@@ -1,6 +1,8 @@
+from datetime import datetime, timezone
 from http import HTTPMethod, HTTPStatus
 
 from flask import (
+    Blueprint,
     current_app,
     flash,
     redirect,
@@ -11,8 +13,6 @@
 )
 from flask_login import current_user, login_required, login_user, logout_user
 
-from OpenOversight.app import sitemap
-from OpenOversight.app.auth import auth
 from OpenOversight.app.auth.forms import (
     ChangeDefaultDepartmentForm,
     ChangeEmailForm,
@@ -34,10 +34,13 @@
     ResetPasswordEmail,
 )
 from OpenOversight.app.utils.auth import admin_required
+from OpenOversight.app.utils.constants import KEY_APPROVE_REGISTRATIONS
+from OpenOversight.app.utils.flask import sitemap
 from OpenOversight.app.utils.forms import set_dynamic_default
 from OpenOversight.app.utils.general import validate_redirect_url
 
 
+auth = Blueprint("auth", __name__, url_prefix="/auth")
 js_loads = ["js/zxcvbn.js", "js/password.js"]
 sitemap_endpoints = []
 
@@ -57,7 +60,8 @@ def static_routes():
 def before_request():
     if (
         current_user.is_authenticated
-        and not current_user.confirmed
+        and not current_user.confirmed_at
+        and not current_user.confirmed_by
         and request.endpoint
         and request.endpoint[:5] != "auth."
         and request.endpoint not in ["static", "bootstrap.static"]
@@ -67,9 +71,15 @@ def before_request():
 
 @auth.route("/unconfirmed")
 def unconfirmed():
-    if current_user.is_anonymous or current_user.confirmed:
+    if current_user.is_anonymous or (
+        current_user.confirmed_at and current_user.confirmed_by
+    ):
         return redirect(url_for("main.index"))
-    if current_app.config["APPROVE_REGISTRATIONS"] and not current_user.approved:
+    if (
+        current_app.config[KEY_APPROVE_REGISTRATIONS]
+        and not current_user.approved_at
+        and not current_user.approved_by
+    ):
         return render_template("auth/unapproved.html")
     else:
         return render_template("auth/unconfirmed.html")
@@ -112,11 +122,16 @@ def register():
             email=form.email.data,
             username=form.username.data,
             password=form.password.data,
-            approved=False if current_app.config["APPROVE_REGISTRATIONS"] else True,
+            approved_at=None
+            if current_app.config[KEY_APPROVE_REGISTRATIONS]
+            else datetime.now(timezone.utc),
+            approved_by=None
+            if current_app.config[KEY_APPROVE_REGISTRATIONS]
+            else User.query.filter_by(is_administrator=True).first().id,
         )
         db.session.add(user)
         db.session.commit()
-        if current_app.config["APPROVE_REGISTRATIONS"]:
+        if current_app.config[KEY_APPROVE_REGISTRATIONS]:
             admins = User.query.filter_by(is_administrator=True).all()
             for admin in admins:
                 EmailClient.send_email(
@@ -141,9 +156,11 @@ def register():
 @auth.route("/confirm/<token>", methods=[HTTPMethod.GET])
 @login_required
 def confirm(token):
-    if current_user.confirmed:
+    if current_user.confirmed_at and current_user.confirmed_by:
         return redirect(url_for("main.index"))
-    if current_user.confirm(token):
+    if current_user.confirm(
+        token, User.query.filter_by(is_administrator=True).first().id
+    ):
         admins = User.query.filter_by(is_administrator=True).all()
         for admin in admins:
             EmailClient.send_email(
@@ -313,18 +330,31 @@ def edit_user(user_id):
                     flash("You cannot edit your own account!")
                     form = EditUserForm(obj=user)
                     return render_template("auth/user.html", user=user, form=form)
-                already_approved = user.approved
+                already_approved = (
+                    user.approved_at is not None and user.approved_by is not None
+                )
+                if form.approved.data:
+                    user.approve_user(current_user.id)
+
+                if form.confirmed.data:
+                    user.confirm_user(current_user.id)
+
+                if form.is_disabled.data:
+                    user.disable_user(current_user.id)
+
                 form.populate_obj(user)
                 db.session.add(user)
                 db.session.commit()
 
                 # automatically send a confirmation email when approving an
                 # unconfirmed user
                 if (
-                    current_app.config["APPROVE_REGISTRATIONS"]
+                    current_app.config[KEY_APPROVE_REGISTRATIONS]
                     and not already_approved
-                    and user.approved
-                    and not user.confirmed
+                    and user.approved_at
+                    and user.approved_by
+                    and not user.confirmed_at
+                    and not user.confirmed_by
                 ):
                     admin_resend_confirmation(user)
 
@@ -353,7 +383,7 @@ def delete_user(user_id):
 
 
 def admin_resend_confirmation(user):
-    if user.confirmed:
+    if user.confirmed_at and current_user.confirmed_by:
         flash(f"User {user.username} is already confirmed.")
     else:
         token = user.generate_confirmation_token()

OpenOversight/app/commands.py

@@ -93,10 +93,13 @@ def make_admin_user(
         username=username,
         email=email,
         password=password,
-        confirmed=True,
         is_administrator=True,
     )
     db.session.add(u)
+    db.session.flush()
+
+    u.confirmed_at = datetime.now()
+    u.confirmed_by = u.id
     db.session.commit()
     print(f"Administrator {username} successfully added")
     current_app.logger.info(f"Administrator {username} added with email {email}")

OpenOversight/app/filters.py

@@ -3,6 +3,7 @@
 from datetime import datetime
 from zoneinfo import ZoneInfo
 
+import us.states
 from flask import Flask, current_app, session
 
 from OpenOversight.app.utils.constants import (
@@ -87,6 +88,16 @@ def display_currency(value: float) -> str:
     return f"${value:,.2f}"
 
 
+def get_state_full_name(abbrev: str) -> str:
+    if abbrev == "FA":
+        return "Federal"
+
+    state = us.states.lookup(abbrev)
+    if state is None:
+        return "N/A"
+    return us.states.lookup(abbrev).name
+
+
 def instantiate_filters(app: Flask):
     """Instantiate all template filters"""
     app.template_filter("capfirst")(capfirst_filter)
@@ -99,3 +110,4 @@ def instantiate_filters(app: Flask):
     app.template_filter("local_time")(local_time)
     app.template_filter("thousands_separator")(thousands_separator)
     app.template_filter("display_currency")(display_currency)
+    app.template_filter("get_state_full_name")(get_state_full_name)

OpenOversight/app/main/__init__.py

@@ -1,6 +0,0 @@
-from flask import Blueprint
-
-
-main = Blueprint("main", __name__)
-
-from OpenOversight.app.main import views  # noqa: E402,F401

OpenOversight/app/main/downloads.py

@@ -5,7 +5,6 @@
 from typing import Any, Callable, Dict, List, TypeVar
 
 from flask import Response, abort
-from sqlalchemy.orm import Query
 
 from OpenOversight.app.models.database import (
     Assignment,
@@ -40,7 +39,7 @@ def check_output(output_str):
 
 
 def make_downloadable_csv(
-    query: Query,
+    query: List,
     department_id: int,
     csv_suffix: str,
     field_names: List[str],
@@ -161,8 +160,5 @@ def descriptions_record_maker(description: Description) -> _Record:
     return {
         "id": description.id,
         "text_contents": description.text_contents,
-        "created_by": description.created_by,
         "officer_id": description.officer_id,
-        "created_at": description.created_at,
-        "last_updated_at": description.last_updated_at,
     }

OpenOversight/app/main/forms.py

@@ -40,7 +40,7 @@
     STATE_CHOICES,
     SUFFIX_CHOICES,
 )
-from OpenOversight.app.utils.db import dept_choices, unit_choices, unsorted_dept_choices
+from OpenOversight.app.utils.db import dept_choices, unit_choices
 from OpenOversight.app.widgets import BootstrapListWidget, FormFieldWidget
 
 
@@ -102,8 +102,8 @@ class FindOfficerForm(Form):
     dept = QuerySelectField(
         "dept",
         validators=[DataRequired()],
-        query_factory=unsorted_dept_choices,
-        get_label="display_name",
+        query_factory=dept_choices,
+        get_label="name",
     )
     unit = StringField("unit", default="Not Sure", validators=[Optional()])
     current_job = BooleanField("current_job", default=None, validators=[Optional()])
@@ -122,12 +122,8 @@ class FindOfficerForm(Form):
         choices=GENDER_CHOICES,
         validators=[AnyOf(allowed_values(GENDER_CHOICES))],
     )
-    min_age = IntegerField(
-        "min_age", default=16, validators=[NumberRange(min=16, max=100)]
-    )
-    max_age = IntegerField(
-        "max_age", default=85, validators=[NumberRange(min=16, max=100)]
-    )
+    min_age = IntegerField("min_age", validators=[NumberRange(min=16, max=100)])
+    max_age = IntegerField("max_age", validators=[NumberRange(min=16, max=100)])
     require_photo = BooleanField(
         "require_photo", default=False, validators=[Optional()]
     )
@@ -287,7 +283,7 @@ class AddOfficerForm(Form):
         "Department",
         validators=[DataRequired()],
         query_factory=dept_choices,
-        get_label="display_name",
+        get_label="name",
     )
     first_name = StringField(
         "First name",
@@ -409,7 +405,7 @@ class EditOfficerForm(Form):
         "Department",
         validators=[Optional()],
         query_factory=dept_choices,
-        get_label="display_name",
+        get_label="name",
     )
     submit = SubmitField(label="Update")
 
@@ -424,7 +420,7 @@ class AddUnitForm(Form):
         "Department",
         validators=[DataRequired()],
         query_factory=dept_choices,
-        get_label="display_name",
+        get_label="name",
     )
     submit = SubmitField(label="Add")
 
@@ -434,7 +430,7 @@ class AddImageForm(Form):
         "Department",
         validators=[DataRequired()],
         query_factory=dept_choices,
-        get_label="display_name",
+        get_label="name",
     )
 
 
@@ -535,7 +531,7 @@ class IncidentForm(DateFieldForm):
         "Department*",
         validators=[DataRequired()],
         query_factory=dept_choices,
-        get_label="display_name",
+        get_label="name",
     )
     address = FormField(LocationForm)
     officers = FieldList(