Skip to content

Tst/emr pipeline testing #4519

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
boozallendanny merged 38 commits into from
Nov 5, 2025
Merged

Tst/emr pipeline testing #4519

boozallendanny merged 38 commits into from
Nov 5, 2025

Conversation

@sethstoudenmier
Copy link
Contributor

@sethstoudenmier sethstoudenmier commented Oct 16, 2025

Description:

Technical Details:

Requirements for PR Merge:

  1. Unit & integration tests updated
  2. API documentation updated (examples listed below)
    1. API Contracts
    2. API UI
    3. Comments
  3. Data validation completed (examples listed below)
    1. Does this work well with the current frontend? Or is the frontend aware of a needed change?
    2. Is performance impacted in the changes (e.g., API, pipeline, downloads, etc.)?
    3. Is the expected data returned with the expected format?
  4. Appropriate Operations ticket(s) created
  5. Jira Ticket(s)
    1. DEV-0

Explain N/A in above checklist:

collinwr and others added 30 commits June 10, 2025 18:31
@collinwr
Overrides the QAT Spark bucket for EMR testing
e9113ce
@collinwr
Makes orphaned_award_info a unmanaged table
0bfaa09
@collinwr
Makes orphaned_transaction_info a unmanaged table
bdb8a56
@collinwr
Makes orphaned_transaction_info a unmanaged table (fixes f-string usage)
8134cdf
@collinwr
Adds IF EXISTS for dropping view to avoid errors
fb41314
@collinwr
Merge branch 'qat' of ssh://ssh.github.com:443/fedspendingtransparenc…
76e842b 
…y/usaspending-api into tst/emr-pipeline-testing
@sethstoudenmier
Change ES delta views to use CAST(... AS STRING) syntax
33cd6ab
@sethstoudenmier
Move away from subquery to support hive syntax
d414a65
@sethstoudenmier
Update unlinked awards download to temp tables in S3
882acac
@sethstoudenmier
Update covid download to use S3 location for readme
1de89b1
@sethstoudenmier
Clearing the faba table if it exists prior to cloning
c072e36
@boozallendanny
sbx testing
dce98a2
@boozallendanny
Merge branch 'tst/emr-pipeline-testing' of https://github.com/fedspen…
68758f9 
…dingtransparency/usaspending-api into tst/emr-pipeline-testing
@boozallendanny
data_broker_url
c0b5041
@boozallendanny
change data_broker_url to data_Broker_db
a529acd
@zachflanders-frb
testing removing subquery
11a2879
@sethstoudenmier
remove subquery in award_search incremental merge
1bab179
@boozallendanny
replace data_broker_db with broker_db
6263cad
@zachflanders-frb
Merge branch 'tst/emr-pipeline-testing' into dev-13454-transaction-se…
bed5252 
…arch-improvements
@zachflanders-frb
Merge branch 'tst/emr-pipeline-testing-no-subquery' into tst/emr-pipe…
15e6b99 
…line-testing-transaction-search-dataframe-integration
@zachflanders-frb
Update transaction search gold columns
2d92ba9
@zachflanders-frb
Update transaction search delta columns
cb067e0
@zachflanders-frb
Cleanup on transaction search delta columns dict
313a803
@boozallendanny
update env config for staging and prod
fca63da
@zachflanders-frb
Merge branch 'dev-13454-transaction-search-improvements' into tst/emr…
573605f 
…-pipeline-testing-transaction-search-dataframe-integration
@sethstoudenmier
handle merge conflicts
91f80f2
@sethstoudenmier
handle merge conflicts
3c4b446
@sethstoudenmier
Merge pull request #4518 from fedspendingtransparency/tst/emr-pipelin…
b91912b 
…e-testing-transaction-search-dataframe-integration

Combine the two EMR testing branches
@sethstoudenmier
formatting
cc98bf3
@sethstoudenmier
update path for COVID download README
5e8a994
@sethstoudenmier sethstoudenmier added the do not merge [PR] shouldn't be merged label Oct 16, 2025
@sethstoudenmier sethstoudenmier changed the base branch from qat to staging October 27, 2025 14:27
@sethstoudenmier sethstoudenmier marked this pull request as ready for review October 30, 2025 20:36
sethstoudenmier
sethstoudenmier commented Oct 31, 2025
View reviewed changes
usaspending_api/etl/tests/data/delta_model_for_test.py
Copy link
Contributor Author

@sethstoudenmier sethstoudenmier Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was moved to avoid confusion as the model in the file is used by a test case and is not a test case itself.

@sethstoudenmier sethstoudenmier removed the do not merge [PR] shouldn't be merged label Nov 5, 2025
@boozallendanny boozallendanny merged commit 2cd0544 into staging Nov 5, 2025
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@boozallendanny boozallendanny boozallendanny approved these changes

Assignees

@sethstoudenmier sethstoudenmier

@boozallendanny boozallendanny

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sethstoudenmier @boozallendanny @collinwr @zachflanders-frb