Get-AzPolicyDefinition object type doesn't work with | convertto-json -depth 10 when using az.resources 7.11.0

[harken350]

Description

I am using az.resources 7.11.0 and have upgraded from 6.16.1 recently. When running the command "get-azpolicydefinition" it will return the object but not work when trying to convert it to a json unless you use "$variable | select * | convertto-json -depth 100" or "$variable=get-azpolicydefinition -id -BackwardCompatible" or some other select command when you either exclude properties, or drill down into specific properties. However, on a previous version I am able to do what I need to without issue. It also changes the type of the object. There is no error. Note that you'll need to change your PS modules by renaming the folders to something else so that it doesn't load the latest PS module version

Issue script & Debug output

import-module az.resources -Version 6.16.1
connect-azaccount
#change the definition ID to any you wish. This will work when you have a valid policy definition
$def=Get-AzPolicyDefinition -Id /providers/Microsoft.Authorization/policyDefinitions/7509877f-d414-4d79-8d1f-d600ea78d087
$def.gettype()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     False    PsPolicyDefinition                       System.Object

$def | convertto-json -depth 100
{
  "Name": "7509877f-d414-4d79-8d1f-d600ea78d087",
  "ResourceId": "/providers/Microsoft.Authorization/policyDefinitions/7509877f-d414-4d79-8d1f-d600ea78d087",
  "ResourceName": "7509877f-d414-4d79-8d1f-d600ea78d087",
  "ResourceType": "Microsoft.Authorization/policyDefinitions",
  "SubscriptionId": null,
  "Properties": {
    "Description": "This policy definition blocks resource creation and updates in West Europe.",
    "DisplayName": "Resources should not be created in West Europe",
    "Metadata": {
      "version": "1.0.0",
      "category": "System Policy"
    },
    "Mode": "Indexed",
    "Parameters": null,
    "PolicyRule": {
      "if": {
        "field": "location",
        "equals": "westeurope"
      },
      "then": {
        "effect": "deny"
      }
    },
    "PolicyType": 2
  },
  "PolicyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/7509877f-d414-4d79-8d1f-d600ea78d087"
}


#Load a new powershell terminal and try this
import-module az.resources -Version 7.11.0
connect-azaccount
#change the definition ID to any you wish. This will work when you have a valid policy definition
$def=Get-AzPolicyDefinition -Id /providers/Microsoft.Authorization/policyDefinitions/7509877f-d414-4d79-8d1f-d600ea78d087
$def.gettype()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     False    PolicyDefinition                         System.Object

#Notice how the output is not right
$def | convertto-json -depth 100
{
  "Description": "This policy definition blocks resource creation and updates in West Europe.",
  "DisplayName": "Resources should not be created in West Europe",
  "Id": "/providers/Microsoft.Authorization/policyDefinitions/7509877f-d414-4d79-8d1f-d600ea78d087/versions/1.0.0",
  "Metadata": {},
  "Mode": "Indexed",
  "Name": "1.0.0",
  "Parameter": {},
  "PolicyRule": {},
  "PolicyType": "BuiltIn",
  "SystemDataCreatedAt": null,
  "SystemDataCreatedBy": null,
  "SystemDataCreatedByType": null,
  "SystemDataLastModifiedAt": null,
  "SystemDataLastModifiedBy": null,
  "SystemDataLastModifiedByType": null,
  "Type": "Microsoft.Authorization/policyDefinitions/versions",
  "Version": "1.0.0",
  "Versions": null
}

($def | select *).gettype()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     False    PSCustomObject                           System.Object

$def | select * | convertto-json -depth 100
{
  "Metadata": {
    "version": "1.0.0",
    "category": "System Policy"
  },
  "Parameter": {},
  "PolicyRule": {
    "if": {
      "field": "location",
      "equals": "westeurope"
    },
    "then": {
      "effect": "deny"
    }
  },
  "Versions": [
    "1.0.0"
  ],
  "Description": "This policy definition blocks resource creation and updates in West Europe.",
  "DisplayName": "Resources should not be created in West Europe",
  "Id": "/providers/Microsoft.Authorization/policyDefinitions/7509877f-d414-4d79-8d1f-d600ea78d087",
  "Mode": "Indexed",
  "Name": "7509877f-d414-4d79-8d1f-d600ea78d087",
  "PolicyType": "BuiltIn",
  "SystemDataCreatedAt": null,
  "SystemDataCreatedBy": null,
  "SystemDataCreatedByType": null,
  "SystemDataLastModifiedAt": null,
  "SystemDataLastModifiedBy": null,
  "SystemDataLastModifiedByType": null,
  "Type": "Microsoft.Authorization/policyDefinitions",
  "Version": "1.0.0"
}

#For some reason this also works but I do not know why
$def=get-azPolicyDefinition -Id /providers/Microsoft.Authorization/policyDefinitions/7509877f-d414-4d79-8d1f-d600ea78d087 -BackwardCompatible
WARNING: Parameter 'BackwardCompatible' is obsolete. This parameter is a temporary bridge to new types and formats and will be removed in a future release.
$def.gettype()                                          

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     False    PolicyDefinition                         System.Object

$def  | convertto-json -depth 100          
{
  "Description": "This policy definition blocks resource creation and updates in West Europe.",
  "DisplayName": "Resources should not be created in West Europe",
  "Id": "/providers/Microsoft.Authorization/policyDefinitions/7509877f-d414-4d79-8d1f-d600ea78d087",
  "Metadata": {},
  "Mode": "Indexed",
  "Name": "7509877f-d414-4d79-8d1f-d600ea78d087",
  "Parameter": {},
  "PolicyRule": {},
  "PolicyType": "BuiltIn",
  "SystemDataCreatedAt": null,
  "SystemDataCreatedBy": null,
  "SystemDataCreatedByType": null,
  "SystemDataLastModifiedAt": null,
  "SystemDataLastModifiedBy": null,
  "SystemDataLastModifiedByType": null,
  "Type": "Microsoft.Authorization/policyDefinitions",
  "Version": "1.0.0",
  "Versions": [
    "1.0.0"
  ],
  "Properties": {
    "Parameters": {},
    "DisplayName": "Resources should not be created in West Europe",
    "Description": "This policy definition blocks resource creation and updates in West Europe.",
    "Metadata": {
      "version": "1.0.0",
      "category": "System Policy"
    },
    "PolicyRule": {
      "if": {
        "field": "location",
        "equals": "westeurope"
      },
      "then": {
        "effect": "deny"
      }
    },
    "PolicyType": "BuiltIn",
    "Mode": "Indexed"
  },
  "ResourceId": "/providers/Microsoft.Authorization/policyDefinitions/7509877f-d414-4d79-8d1f-d600ea78d087",
  "ResourceName": "7509877f-d414-4d79-8d1f-d600ea78d087",
  "ResourceType": "Microsoft.Authorization/policyDefinitions",
  "PolicyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/7509877f-d414-4d79-8d1f-d600ea78d087"
}

Environment data

$psversionTable.PSVersion

Major  Minor  Patch  PreReleaseLabel BuildLabel
-----  -----  -----  --------------- ----------
7      4      13

Module versions

#In the failed test it is
get-module

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     5.1.1                 Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script     7.11.0                az.resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment…}
Manifest   7.0.0.0               Microsoft.PowerShell.Management     {Add-Content, Clear-Content, Clear-Item, Clear-ItemProperty…}
Manifest   7.0.0.0               Microsoft.PowerShell.Utility        {Add-Member, Add-Type, Clear-Variable, Compare-Object…}
Script     2.3.6                 PSReadLine                          {Get-PSReadLineKeyHandler, Get-PSReadLineOption, Remove-PSReadLineKeyHandler, Set-PSReadLineKeyHandler…}

#In the good test it is
get-module

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     4.2.0                 Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script     6.16.2                az.resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment…}
Manifest   7.0.0.0               Microsoft.PowerShell.Management     {Add-Content, Clear-Content, Clear-Item, Clear-ItemProperty…}
Manifest   7.0.0.0               Microsoft.PowerShell.Utility        {Add-Member, Add-Type, Clear-Variable, Compare-Object…}
Script     2.3.6                 PSReadLine                          {Get-PSReadLineKeyHandler, Get-PSReadLineOption, Remove-PSReadLineKeyHandler, Set-PSReadLineKeyHandler…}

Error output

[VeryEarly]

Hi @mentat9 , can you please help take a look at this issue? Thanks.

[mentat9]

Hi @mentat9 , can you please help take a look at this issue? Thanks.

@VeryEarly - This is not a bug. It's a direct result of our required move to autorest.Powershell. The new serializer created a new type and requires that RP properties within the property bag are no longer returned as a Properties object but are instead promoted to the top level. This breaking change was required by the Powershell team. At the time my team discovered the new autorest behavior it was discussed in-depth. We wanted to preserve the existing behavior but were overruled by the Powershell team.

The new type does not convert directly to JSON anymore. The simplest workaround is to insert select -Property * into the pipeline like this:

Get-AzPolicyDefinition | select -Property * | ConvertTo-Json -Depth 100

We did everything we could to mitigate this for our customers:

1. Added the breaking change notice within the previous module version using the standard Powershell process.
2. Provided the new -BackwardCompatible switch as a temporary bridge providing the old behavior in the new module.
3. Released the breaking change first as a preview release before official release at the breaking change release cycle.
4. Validated both versions of the output with in-depth tests.

Feel free to provide additional context on behalf of the Powershell team: I don't know what factors drove the design of the serializer and the requirement to promote properties in the property bag to the top level. If there is nothing else to add, please close this issue as "By design".

[harken350]

Thanks @mentat9 , I should have gone and checked things more thoroughly.

Closing as it is by design

[mentat9]

There is another workaround that could also be of interest. This uses an internal function on the type that creates the JSON instead of converting via Powershell. That works like this:

Get-AzPolicyDefinition | %{ $_.ToJsonString() }