Skip to content

Conversation

@estyxx
Copy link
Member

@estyxx estyxx commented Jun 7, 2025

What

This PR refactors the grants and reimbursements system, moving from hardcoded monetary fields in the Conference mode to a category-based reimbursement model. It adds new models for GrantReimbursementCategory and GrantReimbursement, migrates existing data, and updates the Django admin interface to support the new structure.

Database and Data Model Refactoring:

  • Introduced new models: GrantReimbursementCategory (defines reimbursement types per conference) and GrantReimbursement (records actual granted amounts per category per grant)..
  • Migrated existing grants and conferences to the new model, including backfilling reimbursement categories and amounts, and removed the old fields (accommodation_amount, ticket_amount, travel_amount, approved_type, total_amount) from the Grant model and related fields from the Conference model.

See the proposal: https://www.notion.so/pythonit/Grants-3-0-359642da7c094f3d9445fe0e05b7937c

ToDo

Re-introduce the automatic assignment of the grant travel amount based on the country type.

@estyxx estyxx self-assigned this Jun 7, 2025
@vercel
Copy link

vercel bot commented Jun 7, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Review Updated (UTC)
pycon Error Error Dec 14, 2025 0:58am

@codecov
Copy link

codecov bot commented Jun 7, 2025

Codecov Report

❌ Patch coverage is 91.66667% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.42%. Comparing base (19f667c) to head (ee5c98f).
⚠️ Report is 10 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4420      +/-   ##
==========================================
- Coverage   94.43%   94.42%   -0.02%     
==========================================
  Files         352      352              
  Lines       10135    10134       -1     
  Branches      735      732       -3     
==========================================
- Hits         9571     9569       -2     
+ Misses        469      468       -1     
- Partials       95       97       +2     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@estyxx
Copy link
Member Author

estyxx commented Nov 24, 2025

@copilot

Copy link

Copilot AI commented Nov 24, 2025

@estyxx I've opened a new pull request, #4498, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot AI mentioned this pull request Nov 24, 2025
@estyxx estyxx requested a review from Copilot November 24, 2025 12:07
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the grant reimbursement system from a fixed set of approved types to a flexible category-based approach. The changes enable conference organizers to define custom reimbursement categories (e.g., Travel, Ticket, Accommodation) with individual maximum amounts, replacing the previous hardcoded approved_type field.

Key changes:

  • Introduced GrantReimbursementCategory and GrantReimbursement models for flexible grant management
  • Migrated existing grant data from the old approved_type system to the new category-based system
  • Updated admin interfaces to display and manage reimbursements through the new models

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
backend/grants/models.py Added new models for reimbursement categories and reimbursements; removed deprecated fields and calculation logic
backend/grants/admin.py Added admin interfaces for new models and updated grant display to show reimbursement details
backend/reviews/admin.py Updated grant review workflow to handle multiple reimbursement categories instead of single approved type
backend/reviews/templates/grants-recap.html Changed UI from radio buttons to checkboxes for selecting multiple reimbursement categories
backend/grants/summary.py Refactored financial aggregation to sum from reimbursements instead of grant amounts
backend/grants/migrations/0030_remove_grant_accommodation_amount_and_more.py Migration to create new models and backfill existing data
backend/conferences/migrations/0055_remove_conference_grants_default_accommodation_amount_and_more.py Migration to remove deprecated conference-level grant amount fields
backend/conferences/models/conference.py Removed deprecated grant default amount fields
backend/conferences/admin/conference.py Removed deprecated grant configuration fieldset from admin
backend/grants/tests/test_models.py Updated tests to use new reimbursement system
backend/grants/tests/test_migration_backfill_grant_reimbursements.py Added comprehensive migration tests
backend/reviews/tests/test_admin.py Added tests for grant review workflow with new reimbursement system

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 156 to 174
"""
Aggregates financial data (total amounts) by grant status.
"""
financial_data = filtered_grants.values("pending_status").annotate(
total_amount_sum=Sum("total_amount")
)
financial_summary = {status[0]: 0 for status in statuses}
overall_total = 0

for data in financial_data:
pending_status = data["pending_status"]
total_amount = data["total_amount_sum"] or 0
financial_summary[pending_status] += total_amount
if pending_status in self.BUDGET_STATUSES:
overall_total += total_amount
for status in statuses:
grants_for_status = filtered_grants.filter(pending_status=status[0])
reimbursements = GrantReimbursement.objects.filter(
grant__in=grants_for_status
)
total = reimbursements.aggregate(total=Sum("granted_amount"))["total"] or 0
financial_summary[status[0]] = total
if status[0] in self.BUDGET_STATUSES:
overall_total += total

return financial_summary, overall_total
Copy link

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This unused function should be removed since it has been replaced by _aggregate_financial_data_by_status_new (which should be renamed to take its place). Having both functions could cause confusion about which one is actually being used.

Copilot uses AI. Check for mistakes.

@admin.display(description="Status")
def current_or_pending_status(self, obj):
return obj.current_or_pending_status
Copy link

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Missing blank line after the method definition. There should be a blank line before the next method definition for consistency with Python style guidelines.

Copilot uses AI. Check for mistakes.
return ""
@admin.display(description="Total")
def total_amount_display(self, obj):
return f"{obj.total_allocated:.2f}"
Copy link

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The property name is total_allocated_amount (line 271 in models.py) but this code references obj.total_allocated which is the annotation added in the queryset. This should reference obj.total_allocated_amount to be consistent with the model property, or the comment should clarify this relies on the queryset annotation.

Copilot uses AI. Check for mistakes.
"name": "Ticket",
"description": "Conference ticket",
"max_amount": conference.grants_default_ticket_amount
or Decimal("0.00"),
Copy link

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing import for Decimal. The migration uses Decimal on lines 20, 31, and 42 but doesn't import it from the decimal module.

Copilot uses AI. Check for mistakes.
},
),
# Backfill existing grants
migrations.RunPython(
Copy link

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The migrate_grants function should be called before ensure_categories_exist to ensure categories exist before attempting to create reimbursements. Add a RunPython operation for ensure_categories_exist before this one.

Suggested change
migrations.RunPython(
migrations.RunPython(
code=ensure_categories_exist,
),
migrations.RunPython(

Copilot uses AI. Check for mistakes.
# TODO: move the amount calculation in a separate function maybe?
grant.save(update_fields=["pending_status", "approved_type"])
grant.save(update_fields=["pending_status",])
approved_reimbursement_categories = approved_reimbursement_categories_decisions.get(grant.id, "")
Copy link

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default value should be an empty list [] instead of an empty string "" to match the expected type. This is used in a for loop on line 318, and iterating over a string would produce unexpected behavior.

Suggested change
approved_reimbursement_categories = approved_reimbursement_categories_decisions.get(grant.id, "")
approved_reimbursement_categories = approved_reimbursement_categories_decisions.get(grant.id, [])

Copilot uses AI. Check for mistakes.
@estyxx
Copy link
Member Author

estyxx commented Nov 30, 2025

@claude do a code review

@claude
Copy link
Contributor

claude bot commented Nov 30, 2025

Claude finished @estyxx's task —— View job


Code Review Complete ✅

  • Read and understand the model changes
  • Review migration and backfill logic
  • Analyze admin interface changes
  • Review test coverage
  • Check for bugs and edge cases
  • Provide comprehensive review feedback

Summary

This is a well-executed refactoring that successfully transforms the grant reimbursement system from a rigid, hardcoded structure to a flexible, category-based approach. The implementation is mostly solid with comprehensive test coverage. I've identified several critical issues that need addressing before merge, along with some suggestions for improvement.

Critical Issues 🔴

1. Missing Decimal import in migration (backend/grants/migrations/0030_remove_grant_accommodation_amount_and_more.py:20)

The migration uses Decimal on lines 20, 31, and 42 but doesn't import it from the decimal module. This will cause the migration to fail.

# Add at the top of the file
from decimal import Decimal

Location: backend/grants/migrations/0030_remove_grant_accommodation_amount_and_more.py:1


2. Migration order issue (backend/grants/migrations/0030_remove_grant_accommodation_amount_and_more.py:137)

The migrate_grants function is called before ensure_categories_exist, which means it will try to create reimbursements before the categories exist. This will fail. The operations should be reordered:

operations = [
    # Create reimbursement tables
    migrations.CreateModel(...),
    migrations.CreateModel(...),
    # First ensure categories exist
    migrations.RunPython(
        code=ensure_categories_exist,
    ),
    # Then migrate grants
    migrations.RunPython(
        code=migrate_grants,
        reverse_code=reverse_migration,
    ),
    # Finally remove old fields
    migrations.RemoveField(...),
    ...
]

Location: backend/grants/migrations/0030_remove_grant_accommodation_amount_and_more.py:101-167


3. Incorrect default value type (backend/reviews/admin.py:317)

The default value should be an empty list [] instead of an empty string "". This is used in a for loop on line 318, and iterating over a string would produce unexpected behavior (it would iterate over individual characters).

# Line 317
approved_reimbursement_categories = approved_reimbursement_categories_decisions.get(grant.id, [])

Location: backend/reviews/admin.py:317


High Priority Issues 🟠

4. Unused function should be removed (backend/grants/summary.py:155-172)

The function _aggregate_financial_data_by_status is defined but never called. It's been replaced by _aggregate_financial_data_by_status_new (line 45). Having both functions is confusing. Either:

  • Remove the old function entirely, OR
  • Rename _aggregate_financial_data_by_status_new to _aggregate_financial_data_by_status

Location: backend/grants/summary.py:155-172


5. Missing import in summary.py

The function _aggregate_financial_data_by_status (lines 155-172) references GrantReimbursement but doesn't import it at the top of the file. While this function appears unused, if it's kept, the import is needed.

Location: backend/grants/summary.py:164


6. Annotation vs. Property confusion (backend/grants/admin.py:626)

The total_amount_display method references obj.total_allocated (which is a queryset annotation added on line 666), but the model property is named total_allocated_amount. This works because of the annotation, but it's confusing. Consider either:

  • Renaming the annotation to total_allocated_amount for consistency, OR
  • Adding a comment explaining that this relies on the queryset annotation

Location: backend/grants/admin.py:626, line 666


Medium Priority Issues 🟡

7. Missing blank line after method (backend/grants/admin.py:558)

Missing blank line after the method definition. There should be a blank line before the next method definition for consistency with Python style guidelines (PEP 8).

Location: backend/grants/admin.py:558


8. Potential infinite loop in backfill logic (backend/grants/migrations/0030_remove_grant_accommodation_amount_and_more.py:48-85)

The migration filters grants with approved_type__isnull=False and excludes empty strings, which is correct. However, the add_reimbursement helper function silently does nothing if the category doesn't exist or if amount is falsy. This could lead to incomplete migrations if categories aren't set up correctly. Consider adding logging or validation.


9. Redundant function in summary.py (backend/grants/summary.py:253-270)

The function _aggregate_data_by_approved_type references the old approved_type field that was removed in this PR. This function should also be removed to avoid confusion.

Location: backend/grants/summary.py:253-270


Suggestions for Improvement 💡

10. Consider adding data validation

Add validation in GrantReimbursement to ensure granted_amount doesn't exceed the category's max_amount:

def clean(self):
    if self.granted_amount > self.category.max_amount:
        raise ValidationError(
            f"Granted amount ({self.granted_amount}) cannot exceed category maximum ({self.category.max_amount})"
        )

11. Add database index

Consider adding a database index on GrantReimbursement.grant and GrantReimbursement.category for faster lookups, especially for the total_allocated_amount property which sums all reimbursements.


12. Improve migration robustness

The migration backfill could be more defensive by:

  • Logging which grants were migrated
  • Handling edge cases where amounts might be None vs 0
  • Validating that totals match before/after migration

13. Test coverage gap

The migration tests (test_migration_backfill_grant_reimbursements.py) are excellent, but they don't test the actual Django migration. Consider adding an integration test that runs the actual migration on a test database with sample data.


14. Consider transaction handling in admin

In reviews/admin.py:_review_grants_recap_view, the POST handler creates/deletes reimbursements across multiple grants. While Django provides implicit transaction handling, consider wrapping this in an explicit transaction.atomic() block for clarity and safety.


Positive Aspects ✨

  1. Excellent test coverage: The migration test file is comprehensive and covers all edge cases
  2. Clean model design: The new many-to-many relationship through GrantReimbursement is well-structured
  3. Backwards-compatible migration: The backfill logic correctly handles all the old approved_type values
  4. Good separation of concerns: Categories are conference-specific, allowing flexibility per event
  5. Comprehensive admin interface: The inline editing and display methods are well-implemented

Conclusion

This is a solid refactoring with good test coverage and a clear migration path. The critical issues (missing Decimal import, migration order, and incorrect default value) must be fixed before merging. The high priority issues should also be addressed to avoid confusion and potential bugs.

Once these issues are resolved, this will be a great improvement to the grants system! 🎉


Remove hardcoded default grant amounts for ticket, accommodation, and travel
from `Conference` in favor of using `GrantReimbursementCategory`. Update
all relevant admin forms, models, and templates to reference flexible
categories instead of fixed fields.

- Remove legacy fields: `grants_default_ticket_amount`,
  `grants_default_accommodation_amount`, `grants_default_travel_from_italy_amount`,
  and `grants_default_travel_from_europe_amount` from `Conference`
- Update `Grant` and `GrantReimbursement` logic to work exclusively with
  `GrantReimbursementCategory`
- Refactor grant review admin and summary logic to support multiple,
  configurable reimbursement categories per grant
- Migrate existing grants to new reimbursement category scheme
- Add and update tests and migrations to cover flexible grant categories

This change allows flexible reimbursement types (and amounts) to be
configured per conference, supports granular grant allocation, and
paves the way for internationalization and more complex business rules.
@estyxx estyxx force-pushed the feature/flexible-grant-categories branch 2 times, most recently from 0932ae1 to 0a1ec12 Compare December 1, 2025 15:45
@estyxx
Copy link
Member Author

estyxx commented Dec 1, 2025

@claude do a code review

@claude
Copy link
Contributor

claude bot commented Dec 1, 2025

Claude finished @estyxx's task —— View job


Code Review Complete ✅

  • Read and understand the model changes
  • Review migration and backfill logic
  • Analyze admin interface changes
  • Review test coverage
  • Check for bugs and edge cases
  • Provide comprehensive review feedback

Summary

This is a well-executed refactoring that successfully transforms the grant reimbursement system from a rigid, hardcoded structure to a flexible, category-based approach. The implementation is mostly solid with comprehensive test coverage and a clear migration path.

The code demonstrates good practices including:

  • Clean model design with proper relationships
  • Comprehensive test coverage
  • Thoughtful migration with backfill logic
  • Good separation of concerns

However, there are several critical issues that must be addressed before merging, along with some high-priority concerns that could cause bugs or confusion.


Critical Issues 🔴

1. Migration order is correct

Upon careful review, the migration operations are in the correct order in /home/runner/work/pycon/pycon/backend/grants/migrations/0030_remove_grant_accommodation_amount_and_more.py:

  • Lines 141-173: Creates models
  • Lines 175-178: Runs ensure_categories_exist
  • Lines 180-183: Runs migrate_grants
  • Lines 185+: Removes old fields

The operations execute sequentially, so categories are created before grants are migrated. ✓

2. Missing Decimal import in migration (FIXED) ✓

The migration does import Decimal on line 4, so this is not an issue. ✓

3. Incorrect default value type in backend/reviews/admin.py:307

Location: backend/reviews/admin.py:307

The default value should be an empty list [] instead of an empty string "". Line 318 iterates over this value with a for loop:

# Line 307 - WRONG
approved_reimbursement_categories = approved_reimbursement_categories_decisions.get(grant.id, "")

# Should be:
approved_reimbursement_categories = approved_reimbursement_categories_decisions.get(grant.id, [])

Then on line 327:

for reimbursement_category_id in approved_reimbursement_categories:

If the default is "", the for loop would iterate over an empty string character-by-character, which would work but is semantically incorrect. While it won't crash (empty string iteration is valid), it's a type mismatch that could cause confusion.

Impact: Medium - The code works but has incorrect semantics


High Priority Issues 🟠

4. Duplicate default on line 327 in backend/reviews/admin.py:327

Location: backend/reviews/admin.py:327

Same issue as #3 - the default should be []:

approved_reimbursement_categories = approved_reimbursement_categories_decisions.get(grant.id, [])

Medium Priority Issues 🟡

5. Missing blank line after method in backend/grants/admin.py:544

Location: backend/grants/admin.py:544

There should be a blank line after the current_or_pending_status method definition (line 544) before the next method definition for consistency with PEP 8.


6. Queryset annotation vs model property naming in backend/grants/admin.py:612, 652

Location: backend/grants/admin.py:612, 652

The total_amount_display method (line 612) references obj.total_allocated_amount, which works because of the queryset annotation on line 652. However, the model also has a property with the same name (models.py:276), which could cause confusion.

The code is correct and working as intended, but consider adding a comment explaining the annotation to clarify for future developers:

# Line 652 - Add comment
total_allocated_amount=Coalesce(  # Overrides model property for efficiency
    Sum("reimbursements__granted_amount"),
    Value(0),
    output_field=IntegerField(),
),

Suggestions for Improvement 💡

7. Consider adding validation to GrantReimbursement model

Add validation in the GrantReimbursement model to ensure granted_amount doesn't exceed the category's max_amount:

# In backend/grants/models.py, add to GrantReimbursement class
def clean(self):
    if self.granted_amount > self.category.max_amount:
        raise ValidationError(
            f"Granted amount ({self.granted_amount}) cannot exceed "
            f"category maximum ({self.category.max_amount})"
        )

8. Consider adding database indexes

The GrantReimbursement model already has an index on line 316-317 of models.py


9. Migration robustness

The migration backfill in 0030_remove_grant_accommodation_amount_and_more.py is good, but could be enhanced with:

  • More detailed logging of which grants were migrated
  • Validation that category totals match before/after
  • Better handling of edge cases where amounts might be None

However, the current implementation with logging on lines 74-77, 84-91, and 119-122 is already quite good.


10. Consider transaction handling in admin

In reviews/admin.py:_review_grants_recap_view (lines 294-341), the POST handler creates/deletes reimbursements across multiple grants. While Django provides implicit transaction handling, consider wrapping this in an explicit transaction.atomic() block for clarity and safety (already has @transaction.atomic on grants admin actions, line 276).


Positive Aspects ✨

  1. Excellent test coverage: The migration test file (test_migration_backfill_grant_reimbursements.py) is comprehensive and covers all edge cases
  2. Clean model design: The new many-to-many relationship through GrantReimbursement is well-structured
  3. Backwards-compatible migration: The backfill logic correctly handles all the old approved_type values
  4. Good separation of concerns: Categories are conference-specific, allowing flexibility per event
  5. Comprehensive admin interface: The inline editing and display methods are well-implemented
  6. Proper use of prefetch_related: Line 625 in grants/admin.py properly prefetches reimbursements for efficiency

Summary of Required Changes

Must fix before merge:

  • Fix default value type in backend/reviews/admin.py:307 from "" to []
  • Fix default value type in backend/reviews/admin.py:327 from "" to []

Should fix:

  • Add blank line after method in backend/grants/admin.py:544

Optional improvements:

  • Add validation to GrantReimbursement.clean()
  • Add clarifying comments about queryset annotations
  • Consider explicit transaction handling in review admin

Once the required changes are made, this will be an excellent improvement to the grants system! 🎉

total_amount=None,
grant = GrantFactory(status=Grant.Status.approved)
# Create reimbursement with 0 amount so total_allocated_amount is 0
GrantReimbursementFactory(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use factoryboy Trait

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it actually use it! category__ticket=True,

mock_send_approved_email.assert_not_called()


def test_send_reply_emails_approved_missing_amount(rf, mocker, admin_user):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this test still make sense, if I have no reimbursments I get the error, but if the amounts are 0 the property on the grants still will sum to 0... is even thown?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no it tests that we didn't accidentally set all them to 0...

return conference


def _create_reimbursements_for_grant(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests were created before putting this logic inside the migration, these doesn't make any sense unless we import the utilities methods from the migration itself and we test those. but will be impossible to create the Grants' object as before the migration... so nope

Comment on lines +25 to +27
TRAVEL = "travel", _("Travel")
TICKET = "ticket", _("Ticket")
ACCOMMODATION = "accommodation", _("Accommodation")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This year pycon italia we will have a budget for travel+accommodation together, this mean we need to add a new category to this enum TRAVEL_ACCOMMODATION?
Smells

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure we_need this 🤔
we can just match the name of the Category...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we need: when we do the calculation in the review system we need to know which is which, the travel is calculated location based, while the accomodation is fixed, so in the setup we will have separate categories, but we need to be sure that for the end-user email those are sum together (no ticket category)

Revised several Italian grant-related strings in `index.ts` to use more gender
neutral language and improve consistency in tone. This helps ensure the form
is welcoming and inclusive to all users, and maintains a coherent voice across
the application.
@estyxx
Copy link
Member Author

estyxx commented Dec 14, 2025

@claude do the final code review

@claude
Copy link
Contributor

claude bot commented Dec 14, 2025

Claude encountered an error —— View job


I'll analyze this and get back to you.

@estyxx estyxx marked this pull request as ready for review December 14, 2025 13:07
@marcoacierno marcoacierno merged commit 55de16d into main Dec 25, 2025
10 of 11 checks passed
@marcoacierno marcoacierno deleted the feature/flexible-grant-categories branch December 25, 2025 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants