[Issue #8495] Adding custom fields

[jcrichlake]

Summary

Fixes #8495 and fixes #8490

Changes proposed

This PR adds CustomFields to the common grants OpenAPI endpoints via the updates to the Marshmallow schema. It also supports populating the custom fields from the SGG datasource into the expected field. This is done in the populate_custom_fields function.

Context for reviewers

Part of this PR is that this establishes the pattern for custom fields in the common grants endpoints going forward. There is a new file added common_grants_custom_fields.py that will serve as the container for all custom fields going forward.

These custom fields can then be imported and registered to any base object that is returned in an endpoint by adding it to the CustomFields Marshmallow class. Below is an example of the class registration.

class CustomFields(Schema):
    legacyId = fields.Nested(LegacyId, allow_none=True)
    federalOpportunityNumber = fields.Nested(FederalOpportunityNumber, allow_none=True)
    assistanceListing = fields.Nested(AssistanceListing, allow_none=True)
    agency = fields.Nested(Agency, allow_none=True)
    attachments = fields.Nested(Attachments, allow_none=True)
    category = fields.Nested(Category, allow_none=True)
    fiscalYear = fields.Nested(FiscalYear, allow_none=True)
    costSharing = fields.Nested(CostSharing, allow_none=True)
    additionalInfo = fields.Nested(AdditionalInfo, allow_none=True)
    agencyContact = fields.Nested(AgencyContact, allow_none=True)
    yourNewestField = fields.Nexted(YourNewestField, allow_none=True)

As well as registering the custom fields in the schema it also adds the population of the fields from the SGG base data. This occurs in populate_custom_fields. There are also changes in transform_opportunity_to_cg which is called from the GET endpoint before the populate call is made. These changes are to pre-populate some of the dictionary fields to ensure the data is pulled out of the initial request so that it can be added to the eventual response.

Validation steps

1. Checkout the branch jeff-8495-Ad…s-to-openapi
2. Run cd /api
3. Run make openapi-spec-common-grants to generate the new openAPI yaml file
4. Run make init make db-seed-local && make populate-search-opportunities make run-logs to stand up the API endpoints.
5. Open the SIMPLER-GRANTS-PROTOCOL repo and navigate to lib/python-sdk/examples6.
6. Update the ports in the 3 custom fields examples to be 8080
7. Update the Custom field examples to not take in an opportunity base parameter and update the print statement to list all custom fields (See below for examples of each file.
8. from lib/python-sdk Run `poetry run python examples/list_opportunities.py and validate the custom fields
9. Grab an Opportunity ID then run poetry run python examples/get_opportunity.py repeat validation
10. Run poetry run python examples/search_opportunities.py repeat validation

Updated list_opportunities.py

#!/usr/bin/env python3
"""Example script demonstrating basic client usage.

Run with: poetry run python list_opportunity.py
"""

from common_grants_sdk.client import Client
from common_grants_sdk.client.config import Config
from common_grants_sdk.extensions.specs import CustomFieldSpec
from common_grants_sdk.schemas.pydantic import OpportunityBase, CustomFieldType

config = Config(
    base_url="http://localhost:8080",
    api_key="two_org_user_key",
    timeout=5.0,
    page_size=10,
)
client = Client(config)


fields = {
    "legacyId": CustomFieldSpec(
        field_type=CustomFieldType.INTEGER,
        value=int,
        name="Legacy ID",
        description="Federal id legacy value",
    ),
    "groupName": CustomFieldSpec(field_type=CustomFieldType.STRING, value=str),
}

opp = OpportunityBase.with_custom_fields(custom_fields=fields, model_name="Opportunity")



response = client.opportunities.list(page=1)

print(f"Found {len(response.items)} opportunities:")


for item in response.items:

    if item.custom_fields is not None:

        print(
            f"  - {item.id}: {item.title}, custom field value: {item.custom_fields}"  # type: ignore[union-attr]
        )

Updated get_opportunity.py

#!/usr/bin/env python3
"""Example script demonstrating how to fetch a single opportunity by ID.

Run with: poetry run python get_opportunity.py <oppId>
"""

import sys

from common_grants_sdk.client import Client
from common_grants_sdk.client.config import Config
from common_grants_sdk.extensions.specs import CustomFieldSpec
from common_grants_sdk.schemas.pydantic import OpportunityBase, CustomFieldType

if len(sys.argv) < 2:
    print("Usage: get_opportunity.py <oppId>", file=sys.stderr)
    sys.exit(1)

opp_id = sys.argv[1]
config = Config(
    base_url="http://localhost:8080",
    api_key="two_org_user_key",
    timeout=5.0,
)
client = Client(config)

fields = {
    "legacyId": CustomFieldSpec(
        field_type=CustomFieldType.INTEGER,
        value=int,
        name="legacy_id",
        description="Federal ID Legacy Field",
    ),
    "groupName": CustomFieldSpec(field_type=CustomFieldType.STRING, value=str),
}

opp = OpportunityBase.with_custom_fields(custom_fields=fields, model_name="Opportunity")


opportunity = client.opportunities.get(opp_id)

print(f"Opportunity {opp_id}:")
print(f"  Title: {opportunity.title}")
print(f"  ID: {opportunity.id}")
print(f" Custom Fields: {opportunity.custom_fields}")

Updated search_opportunities.py

"""Example script demonstrating how to search for opportunities.

Run with: poetry run python search_opportunity.py <searchTerm>
"""

import sys

from common_grants_sdk.client import Client
from common_grants_sdk.client.config import Config
from common_grants_sdk.schemas.pydantic.models.opp_status import OppStatusOptions
from common_grants_sdk.extensions.specs import CustomFieldSpec
from common_grants_sdk.schemas.pydantic import OpportunityBase, CustomFieldType

if len(sys.argv) < 1:
    print("Usage: search_opportunity.py <searchTerm>", file=sys.stderr)


config = Config(
    base_url="http://localhost:8080",
    api_key="two_org_user_key",
    timeout=5.0,
    page_size=10,
)

search = sys.argv[1]
client = Client(config)


fields = {
    "legacyId": CustomFieldSpec(field_type=CustomFieldType.INTEGER, value=int),
    "groupName": CustomFieldSpec(field_type=CustomFieldType.STRING, value=str),
}

opp = OpportunityBase.with_custom_fields(custom_fields=fields, model_name="Opportunity")


response = client.opportunities.search(
    search=search, status=[OppStatusOptions.OPEN], page=1
)

print(f"Found {len(response.items)} opportunities: ")

for item in response.items:
    print(
        f" - {item.id}: {item.title} custom field value: {item.custom_fields}"  # type: ignore[union-attr]
    )

[jcrichlake]

@chouinar see below for the 3 curl commands to hit each of the endpoints.

ListOpportunities: curl -H 'accept: application/json' -H 'X-Api-Key: two_org_user_key' -X 'GET' 'http://localhost:8080/common-grants/opportunities'

Get Opportunity: curl -H 'accept: application/json' -H 'X-Api-Key: two_org_user_key' -X 'GET' 'http://localhost:8080/common-grants/opportunities/3d862f45-a525-460a-8134-5eeffd877611'

SearchOpportunities: curl -H 'accept: application/json' -H 'X-Api-Key: two_org_user_key' -X 'POST' 'http://localhost:8080/common-grants/opportunities/search' -d '{ "filters": { "status": { "operator": "in", "value": ["open", "forecasted"] }, "pagination": { "page": 1, "pageSize": 10 }, "search": "education", "sorting": { "sortBy": "lastModifiedAt", "sortOrder": "desc" } }'

[widal001]

This PR makes a ton of progress setting up the custom fields! So nice work. I left a quite a few comments but not all of them have to be addressed in this PR.

Things that should be addressed before merging:

1. Aligning these custom fields with the ones in the custom field catalog. If there are issues we run into validation-wise, we can always update the catalog, but they should be in sync.
2. Expanding the tests to handle the additional test cases I described -- this might require some updates to transformation.py to handle malformed custom fields data. In particular we want to be really certain how missing or malformed data will be handled so we're not returning 500 errors to API consumers if just one record has a bad value.

The other notes about the tests or the Pyright aren't urgent but can you create some tickets to account for that work?

[widal001]

This is probably better solved in the Python SDK itself, but I wanted to flag that I'm getting an IDE warning from Pylance/Pyright when we try to populate CustomField without explicitly setting CustomField.schema

In the next Python SDK release, we should make the default value for this be None.

[widal001]

It's not touched by this PR, but the same is true for SingleDateEvent which raises a type error unless you explicitly pass SingleDateEvent.eventType

We should have that default to EventType.SINGLE_DATE in the SDK since that's a literal value for that sub-class of EventBase

[widal001]

This doesn't need to be changed in this PR, but we should discuss the pros and cons of creating dedicated schemas for these custom fields, instead of just using the generic CustomField (similar to what we're doing for the Marshmallow schemas). Maybe this is a future KT topic with Bryan.

While the current pattern formats the data, we're not actually doing any validation of the data going into these custom fields at the transformation layer (via pydantic), which makes it harder to catch and handle malformed data before we try to serialize it with the Marshmallow models.

The plugin framework we're working on now will make it easier to define and auto-generate those pydantic schemas.

[widal001]

Instead of defining this over and over again in each mock test, can we bubble this up to a fixture or even just a separate utility function that returns default values that we can override as needed?

Also I notice that we're setting the same attributes in each test but the values are changing slightly, but there doesn't seem to be a reason for having them be different across tests.

Seeing the same values set over and over again makes it hard to track what's actually be tested.

[widal001]

Also can you add assert result is not None before these other asserts to remove the Pylance warnings?

[widal001]

Where is this "MockAttachment" defined and what purpose is it solving?

I only see the phrase MockAttachment in this file. Same with the other Mock<CustomField> examples

[jcrichlake]

It's mocking the opportunity_attachments custom fields property.

[widal001]

Can you move this out into its own TestPopulateCustomFields class and test a few additional cases, each as a separate method:

1. This base transformation test case, where everything is populated as expected.
2. A test case where some or all of these custom fields are missing values -- we want to check that None values are handled correctly.
3. A test case where some of these values have the incorrect type -- we want to check how we handle malformed data types

For test cases 2 and 3 we might need to also load the Marshmallow schemas you created to ensure that malformed data is caught before it gets to those values. Because otherwise we'll likely pass on some internal 500 errors to the API consumers -- which we don't want.

On a related we should have a similar set of rout-level test cases that tests how the new marshmallow models handle invalid data (e.g. does it raise a 500 error or a different error type like 422 or something else)