new `data import|export beta tree` commands

[mshanemc]

There's been a number of longstanding problems and limitations with the tree commands whose fixes required a rewrite. There's a few small breaking changes and y'all run these commands A LOT so we're releasing them through our normal beta process.

Phase 1: Beta (complete)

1. introduce data import beta tree and data export beta tree
2. existing commands encourage users to try the new ones

Breaking changes between existing commands and the beta commands

1. import beta removes the hidden, deprecated content-type. Only json files are supported. Usage: ~5 per year.
2. import beta removes the --config-help flag. The schema information is now in the command help. Usage: ~1 per week.

Other changes

1. export --plan writes Object names as the file name. It used to append an s on the end. So the filename is now Account.json and Foo__c.json instead of Accounts.json and Foo__cs.json. Since the filenames are part of the plan file, this only affects newly generated plans/files.
2. export beta no longer writes empty child objects. Previously, you'd see properties with {records: []} that had no effect on imports.
3. import beta ignores the resolveRefs and saveRefs (export beta produces them to preserve compatibility with the non-beta import command).
4. export beta now handles more than 2 levels of child objects in a query using --plan (up to 5 because that's SOQL's limit). see Extract deeply nested (more than 1 level) plan #2663 and Extract deeply nested (more than 1 level) plan #2359
5. both export beta and import beta with --plan handle records that refer to records of the same object type (ex: Account with ParentId, User with ManagerId). Improve handling of self-referencing lookups by force:data:tree commands #2364
6. import beta using --plan handles files with more than 200 records. It will batch them into groups of 200 per object.
7. import supported plan files where the files property could contain objects (the export command did not produce such files). For import beta only strings are allowed and will throw an error otherwise.
8. import beta in --files mode (not --plan) will import the files in parallel (files can't reference each other without --plan)
9. import provides deprecation warnings for both content-type and config-help flags, since import beta removes them. --config-help is now hidden (content-type was already hidden)
10. import beta with --plan is optimized for references (using the saved ID of one record as the value of a Relationship Field in another) that follow the standard pattern @[ObjectName]Ref[SomeNumber], like @AccountRef54 or @Foo__cRef0. In the past, a ref could start with @ followed by anything. The export command historically produced refs in the preferred format (as does export beta), but you may have handcrafted your own at some point. Some of the enhancements (self-referential objects, support for more than 200 records in a file, etc) required the preferred pattern.

Export files created by export beta are compatible with import and import beta

The new record limit for import beta with --plan is not documented. You're most likely to hit a call stack depth (from recursion) or some org limit (data storage, api call limits) depending on your data and your org. If you're doing something really large, we still recommend the bulk commands.

During the beta, please submit issues with the new commands to https://github.com/forcedotcom/cli/issues

In general, you'll find the --plan options much more powerful. We encourage you to use those over the --files option.

Phase 2: GA the new commands, put the old under legacy. [July 10 2024] <=== Currently Here

1. the old commands will be renamed to data import legacy tree and data export legacy tree, and marked hidden and deprecated with the Phase 3 date
2. the new commands will become data import tree and data export tree. They will still work with their beta alias, but you'll get a deprecation warning if you use it.
3. the force: aliases will go to the new commands
4. data export tree with --json will being warning that it will change json output (the saveRefs/resolveRefs properties are in the json output) after Phase 3 date

Phase 3: retire the legacy commands and all their dependent code [Nov 10 2024]

1. make export stop writing the unused saveRefs and resolveRefs properties on plan files, and stop returning them in json
2. tighten schema to remove the object part of files, and remove saveRefs and resolveRefs
3. remove the beta alias from import|export

[RemiLeGuin]

Hello @mshanemc,
Thank you for this new release! I particularly like to query more than 2 levels of child object, the import of more than 200 records and the possibility to refer to the same object type in a relation.
Do you have any plan to support record type definition at import?

[mshanemc]

Hello @mshanemc, Thank you for this new release! I particularly like to query more than 2 levels of child object, the import of more than 200 records and the possibility to refer to the same object type in a relation. Do you have any plan to support record type definition at import?

Nobody's asked for that before. You can open a request for it https://github.com/forcedotcom/cli/discussions (that's what we use to prioritize what gets done next). It's a good feature idea but probably needs some design discussions.

[jasonAtThinkLP]

For the record, I'm hitting a limit of about ~2800 records (of a single custom object type) using the new import command before I hit Maximum call stack Exceeded.

Example individual record:

{
      "attributes": {
        "type": "MyNamespace__Car__c",
        "referenceId": "MyNamespace__Car__cRef9"
      },
      "MyNamespace__Is_Insured__c": "Yes",
      "MyNamespace__Location__c": "001As00000SEX15IAH",
      "RecordTypeId": "012As000001wsWqIAI",
      "MyNamespace__Manufacture_Date__c": "2016-09-01",
      "MyNamespace__Manufacture_Time__c": "04:00:00.000Z",
      "MyNamespace__Car_Type__c": "Sedan",
      "MyNamespace__Car_Outcome__c": "Totaled",
      "MyNamespace__Report_Date__c": "2016-09-01",
      "MyNamespace__Report_Time__c": "04:00:00.000Z",
      "MyNamespace__Mechanic__c": "Dr. Smith",
      "MyNamespace__Repair_Facility__c": "Auto Shop",
      "MyNamespace__Repair_Facility_Street__c": "123 Main St",
      "MyNamespace__Repair_Facility_City__c": "New York",
      "MyNamespace__Repair_Facility_State__c": "NY",
      "MyNamespace__Repair_Facility_ZIP__c": "10001",
      "MyNamespace__Time_Car_Began_Work__c": "09:00:00",
      "MyNamespace__Activity_Before_Incident__c": "Driving",
      "MyNamespace__What_Happened__c": "Accident",
      "MyNamespace__What_Was_Damage__c": "Scratch",
      "MyNamespace__What_Was_Object_Or_Substance__c": "Pole",
      "MyNamespace__Incident_Location__c": "Garage",
      "MyNamespace__Towed_Away__c": true
}

[cristiand391]

We'll be pushing Phase 3 for next week, final schedule:
removed on Wed. 13 RC (gets promoted to stable on Wed. 20).

[cristiand391]

the legacy commands have been removed in sf v2.72.21, release notes:
https://github.com/forcedotcom/cli/tree/main/releasenotes