Ftr/dev 12639 update custom download #4447
Conversation
| """ | ||
| Returns True or False depending on if the expected_header_value matches what is sent with the request | ||
| """ | ||
| return request.headers.get(DOWNLOAD_API_HEADER) == DOWNLOAD_HEADER_VALUE |
There was a problem hiding this comment.
Using the same format as the experimental_elasticsearch_api wasn't coming through for me (using META.get and _ instead of -), but doing it this way did
There was a problem hiding this comment.
Django does some handling of the request headers (see https://docs.djangoproject.com/en/4.2/ref/request-response/#django.http.HttpRequest.META).
In short, the ES experimental header would normally be supplied to the API as X-Experimental-API and Django would convert it under request.META to be HTTP_X_EXPERIMENTAL_API.
With that said you should also be good to use request.headers in this case, but I would recommend changing the expected value to no include "HTTP" as that is something that Django prepends for the "request.META".
| download_job.json_request = json.dumps(str_to_json_original) | ||
| download_job.save() | ||
| else: | ||
| self.process_request(download_job) |
There was a problem hiding this comment.
I'm not sure if this is the best way to go about it, but this way the spark job is reading from the original download_job without files a or b in it and then the download_job runs without file c
| """ | ||
| Returns True or False depending on if the expected_header_value matches what is sent with the request | ||
| """ | ||
| return request.headers.get(DOWNLOAD_API_HEADER) == DOWNLOAD_HEADER_VALUE |
There was a problem hiding this comment.
Django does some handling of the request headers (see https://docs.djangoproject.com/en/4.2/ref/request-response/#django.http.HttpRequest.META).
In short, the ES experimental header would normally be supplied to the API as X-Experimental-API and Django would convert it under request.META to be HTTP_X_EXPERIMENTAL_API.
With that said you should also be good to use request.headers in this case, but I would recommend changing the expected value to no include "HTTP" as that is something that Django prepends for the "request.META".
usaspending_api/download/tests/integration/test_download_accounts.py
Outdated
Show resolved
Hide resolved
| from usaspending_api.download.v2.request_validations import DownloadValidatorBase | ||
| from usaspending_api.routers.replicas import ReadReplicaRouter | ||
| from usaspending_api.submissions.models import DABSSubmissionWindowSchedule | ||
| from usaspending_api.settings import IS_LOCAL |
There was a problem hiding this comment.
You can use settings.IS_LOCAL since settings is already imported above.
…ogram_activity and account_balances
….com/fedspendingtransparency/usaspending-api into ftr/dev-12639-update-custom-download
This reverts commit 654c4b8.
This reverts commit 9a49a2f.
usaspending_api/download/management/commands/delta_downloads/builders.py
Show resolved
Hide resolved
| from usaspending_api.common.helpers.spark_helpers import configure_spark_session, get_active_spark_session | ||
| from usaspending_api.common.spark.configs import LOCAL_EXTENDED_EXTRA_CONF | ||
|
|
||
| if settings.IS_LOCAL: |
There was a problem hiding this comment.
IS_LOCAL is hardcoded to True:
This is causing the import below to happen, which errors in not spark contexts because
pyspark is not a defined dependency of the app.
Description:
This has file c (award_financial) use Spark to process a download job when award_financial is a submission_type
Technical Details:
Uses the experimental header HTTP-X-DOWNLOAD-API to go through spark
If Files A and B are included in the request, they still go through the SQS queue
Requirements for PR Merge:
Explain N/A in above checklist: