pac data improvements

[Laskewitz]

💡 Idea

Currently, it's possible to use the data commands to import and export data with the Power Platform CLI. You can only use the commands with the .NET full framework version of the CLI. I would love to see these commands also on MacOS and Linux.

Apart from that, I would love to be able to generate the schema file without using the CMT tool. It would be great to have a pac data generate-schema command that will generate the schema file.

Also - what could be a cool copilot scenario, would be the creation of test data. For instance, have a pac copilot generate-data command that can generate data based on the schema file.

🧑‍💻 Commands

pac data generate-schema

Command usage: pac data generate-schema --tables [--select-fields] [--select-relationships] [--input-file] [--output-file]

Command	Description
--tables	The internal name(s) of the table(s) you want to create a schema for. (alias: -t)
--select-fields	Optional: Fields that should be exported. If not specified, select all fields. (alias: -sf)
--select-relationships	Optional: The internal name(s) of the relationship(s) you want to include. (alias: -sr)
--input-file	Optional: Path and name of a JSON file with all the details about the tables (including import settings), fields, and relationships in there. (alias: -i)
--output-file	Optional: Path and name of the xml schema file location. If not specified, use current directory + schema.xml (alias: -o)

pac copilot generate-data

Command usage: pac copilot generate-data --schema [--rows] [--output-file]

Command	Description
--schema	The schema that will be used for the data generation. (alias: -s)
--rows	Optional: Number of rows that need to be generated. Default: 10. Make sure to have a maximum as well. (alias: -r)
--output-file	Optional: Path and name of the zip file location. If not specified, use current directory + data.zip (alias: -o)

[jboliveira]

As an extra proposal, I would incorporate the following commands into the concept:

pac data validate

Validation would be useful if there were any discrepancies in the schema and/or data xml structure.

Command usage: pac data validate [--schema-file] [--data-file]

Command	Description
--schema-file	Path and name of the xml schema file location. If not specified, use current directory + schema.xml. (alias: -s)
--data-file	Path and name of the data zip file location. If not specified, use current directory + data.zip. (alias: -d)

pac copilot generate-data

I'd add the [--delta-since] command to this recommendation.

Command usage: pac copilot generate-data --schema [--rows] [--output-file] [--delta-since]

Command	Description
--delta-since	Specifies the delta filter condition in order to export only the data that has has been created/modified since the specified date. (alias: -d)

E.g. pac copilot generate-data --schema schema.xml --delta-since 20230901

Adding this functionality will reduce the size of the data.xml file to merely the delta changes from the previous release.

Remarks

With the completion of these ideas, we will be able to remove pac tool cmt from the CLI and then fully automate the steps via CI/CD.

[Leeuwrik]

The generate data one!!!

Reminds me of back in the days of SharePoint. Grandpa is speaking now; this was back in 2009 :P But there we started using the SharePoint automation functionality to generate customer-specific testdata (so sets of documents, presentations, excelfiles, textdocuments) around a specific model, in order to highlight what the system would look like.

Now, working with Dynamics, it would indeed be /awesome/ if we could have a system that doesn't just generate default test data for the out of the box entities, but indeed also based on the schema for the custom ones <3 And now I'm also thinking that this would be an awesome little tool just to create on our own even, for our project... Hmm...!

That said, Jader, aren't you mixing up the generate-data and the delta export in your comment...? ;)

[JJGriffin]

The lack of Config Migration Tool like functionality in the PAC CLI means that developers often need to write custom code / build tasks to handle this process. Adding this to the PAC CLI would save a great amount of time.

[ryanperrymba]

Hi @Laskewitz, second this.

Recently completed a very large T2T migration. To address this gap in functionality, we wrote scripting to generate the schema - it wasn't perfect, and the code is IP of the client, but definitely a need here.

I would add, the ability to include User Mappings with pac data import is also needed.
Also, CMT itself doesn't support image columns, but that is a broader issue beyond the PAC CLI.

@marcelbf ? PP ProDev team weigh in on these ideas?

[parvezghumra]

On the subject of the Configuration Migration Tool, it would be really helpful to have the ability to apply custom mappings for data packages to allow environment-specific value overrides, since certain tables have values in columns that are somehow environment specific (eg. GUID of root-level Business Unit, GUIDs of Currency records representing the same ISO currency, Units, Unit Groups, several config data tables that form part of the scheduling engine etc).