Would like a command to get results (successes and failures) of a force:data:bulk:upsert run

[codeulike]

force:data:bulk:upsert is a really convenient way to use the Salesforce bulk api

BUT there isn't a command to fetch the results once your job has finished, you have to go into the Salesforce UI and download them (batch by batch) or use some other method like the REST API to retrieve them

When doing a serious data import into Salesforce (several objects, hundreds of thousands of records etc) the Results of a Bulk API job are very important because:

a) for your successful rows, you want to know the Salesforce Id that got allocated (because you might well need that for a child object you're going to import next, etc. You cant always use externalids in all situations to relate objects)

b) for your failed rows (and in real life there are always failed rows) you want to know what the errors are - locking, invalid picklists, valdiation rules etc

So a new command something like ...

sfdx force:data:bulk:results --jobid 750xx000000005sAAA

... which then gives you all the results files (from each batch) concatenated together into one big CSV would be REALLY useful

Many thanks

[github-actions[bot]]

Thank you for filing this feature request. We appreciate your feedback and will review the feature at our next grooming or sprint planning session. We prioritize feature requests with more upvotes and comments.

[git2gus[bot]]

This issue has been linked to a new work item: W-12408034

[WillieRuemmele]

Hi @codeulike, sorry that this issue has been quiet. Does the force data bulk status command meet your requirements? what else would you like to see it report. Note that while we've deprecated the force namespace, the data commands are the exception to that, non "force" data commands use Bulk API 2.0 instead of 1.0 in "force"

[codeulike]

Thanks I'll have a look at force data bulk status and report back

[codeulike]

You asked what else would you like to see it report?

Using version 2.1.7-355d833 sf force data bulk status --job-id 750nnnnnnnnnnnnnnn --batch-id 751nnnnnnnnnnnnnnn gives me:

=== Batch Status

apexProcessingTime:      0
apiActiveProcessingTime: 8427
createdDate:             2023-07-31T14:02:47.000Z
id:                      751nnnnnnnnnnnnnnn
jobId:                   750nnnnnnnnnnnnnnn
numberRecordsFailed:     162
numberRecordsProcessed:  4555
state:                   Completed
systemModstamp:          2023-07-31T14:02:59.000Z
totalProcessingTime:     11677
Getting Status... done

for the 162 records that failed, I want to see the error message for each row.
for the records that succeeded, I want to see the id that got generated

In the UI (for Bulk API 1) these are downloadable CSVs that give me this information:

For Bulk API 1 the rest call to fetch the results CSV is documented here: https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/asynch_api_batches_get_results.htm

For Bulk API 2 I think its this one: https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/get_job_failed_results.htm#get_job_failed_results

Being able to get row-by-row results of what succeeded and what failed (and why) is very very useful in real world loading scenarios

Many thanks

[WillieRuemmele]

Hey @codeulike - I just finished testing these changes. Keep an eye out in the release notes for a new data bulk results command

jsforce/jsforce#1603
salesforcecli/plugin-data#1097

[alan-morey]

I needed this last week 😅, I found sf api request rest to be very useful as a workaround to get the failure results:

sf api request rest '/jobs/ingest/750OO00000Frl6bYAB/failedResults' --stream-to-file failed-results.csv

Lookging forward to this new change landing to make life easier

[codeulike]

@alan-morey Thats really interesting, thanks

[jshackell-sfdc]

This feature was released in version 2.66.7 (November 13, 2024). Enjoy!

[codeulike]

Thanks for adding the feature, I got around to trying it out.

For anyone wondering, the output of the data bulk results command looks like this:

Status: JobComplete
Operation: upsert
Object: Account

Processed records: 1000000
Successful records: 1000000

Saved successful results to 750P100000R6345IAF-success-records.csv

The CSV file(s) get put in the current folder.

It works great, and there's a --json option to make getting the filename(s) easier (that is, the result files are still csv, but the output of the command is json)