Skip to content

Fix 429 service operation #64

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Splaxi wants to merge 10 commits into
base: development
Choose a base branch
from
Open

Fix 429 service operation #64

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ca0fb26
Merge pull request #58 from d365collaborative/development
Splaxi Mar 11, 2022
d113d74
Version: 0.4.36
Splaxi Mar 14, 2022
c9f4d31
Merge branch 'development'
Splaxi Mar 14, 2022
7785f3b
Merge branch 'fix-missing-systemurl-parameters'
Splaxi Mar 14, 2022
97b259e
Merge pull request #61 from Splaxi/master
Splaxi Mar 14, 2022
0f23884
Merge branch 'development'
Splaxi Feb 9, 2023
c37f34b
Version: 0.4.36
Splaxi Feb 9, 2023
018a665
Merge branch 'master' of https://github.com/d365collaborative/d365fo.…
Splaxi Feb 9, 2023
082fae5
Version: 0.4.38
Splaxi Feb 9, 2023
b0fb593
Fix: Refactor Invoke-D365RestEndpoint to handle 429
Splaxi Feb 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion d365fo.integrations/d365fo.integrations.psd1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
RootModule = 'd365fo.integrations.psm1'

# Version number of this module.
ModuleVersion = '0.4.35'
ModuleVersion = '0.4.38'

# ID used to uniquely identify this module
GUID = 'd2667b62-1436-42b3-a840-ab6b4a0e5aa0'
Expand Down
2 changes: 1 addition & 1 deletion d365fo.integrations/d365fo.integrations.psm1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
$script:ModuleRoot = $PSScriptRoot
$script:ModuleVersion = '0.4.35'
$script:ModuleVersion = '0.4.38'

$Script:TimeSignals = @{}

Expand Down
18 changes: 0 additions & 18 deletions d365fo.integrations/functions/add-d365odataconfig.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,24 +55,6 @@
Tags: Integrations, Integration, Bearer Token, Token, OData, Configuration

Author: Mötz Jensen (@Splaxi)

.LINK
Clear-D365ActiveBroadcastMessageConfig

.LINK
Get-D365ActiveBroadcastMessageConfig

.LINK
Get-D365BroadcastMessageConfig

.LINK
Remove-D365BroadcastMessageConfig

.LINK
Send-D365BroadcastMessage

.LINK
Set-D365ActiveBroadcastMessageConfig
#>

function Add-D365ODataConfig {
Expand Down
18 changes: 0 additions & 18 deletions d365fo.integrations/functions/get-d365activeodataconfig.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,24 +22,6 @@
Tags: OData, Environment, Config, Configuration, ClientId, ClientSecret

Author: Mötz Jensen (@Splaxi)

.LINK
Add-D365BroadcastMessageConfig

.LINK
Clear-D365ActiveBroadcastMessageConfig

.LINK
Get-D365BroadcastMessageConfig

.LINK
Remove-D365BroadcastMessageConfig

.LINK
Send-D365BroadcastMessage

.LINK
Set-D365ActiveBroadcastMessageConfig
#>

function Get-D365ActiveODataConfig {
Expand Down
18 changes: 0 additions & 18 deletions d365fo.integrations/functions/get-d365odataconfig.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,24 +40,6 @@
Tags: OData, Environment, Config, Configuration, ClientId, ClientSecret

Author: Mötz Jensen (@Splaxi)

.LINK
Add-D365BroadcastMessageConfig

.LINK
Clear-D365ActiveBroadcastMessageConfig

.LINK
Get-D365ActiveBroadcastMessageConfig

.LINK
Remove-D365BroadcastMessageConfig

.LINK
Send-D365BroadcastMessage

.LINK
Set-D365ActiveBroadcastMessageConfig
#>

function Get-D365ODataConfig {
Expand Down
1 change: 0 additions & 1 deletion d365fo.integrations/functions/import-d365odataentity.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,6 @@ function Import-D365ODataEntity {
Invoke-RequestHandler -Method POST -Uri $odataEndpoint.Uri.AbsoluteUri -Headers $headers -ContentType "application/json;charset=$PayloadCharset" -Payload $Payload -RetryTimeout $RetryTimeout

if (Test-PSFFunctionInterrupt) { return }

}
catch {
$messageString = "Something went wrong while importing data through the OData endpoint for the entity: $EntityName"
Expand Down
78 changes: 77 additions & 1 deletion d365fo.integrations/functions/invoke-d365restendpoint.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,30 @@

The charset has to be a valid http charset like: ASCII, ANSI, ISO-8859-1, UTF-8

.PARAMETER RetryTimeout
The retry timeout, before the cmdlet should quit retrying based on the 429 status code

Needs to be provided in the timspan notation:
"hh:mm:ss"

hh is the number of hours, numerical notation only
mm is the number of minutes
ss is the numbers of seconds

Each section of the timeout has to valid, e.g.
hh can maximum be 23
mm can maximum be 59
ss can maximum be 59

Not setting this parameter will result in the cmdlet to try for ever to handle the 429 push back from the endpoint

.PARAMETER ThrottleSeed
Instruct the cmdlet to invoke a thread sleep between 1 and ThrottleSeed value

This is to help to mitigate the 429 retry throttling on the OData / Custom Service endpoints

It makes most sense if you are running things a outer loop, where you will hit the OData / Custom Service endpoints with a burst of calls in a short time

.PARAMETER Tenant
Azure Active Directory (AAD) tenant id (Guid) that the D365FO environment is connected to, that you want to access through REST endpoint

Expand Down Expand Up @@ -89,6 +113,26 @@
The ServiceName used for the import is "UserSessionService/AifUserSessionService/GetUserSessionInfo".
The Payload is a valid json string, containing all the needed properties.

.EXAMPLE
PS C:\> $Payload = '{"RateTypeName": "TEST", "FromCurrency": "DKK", "ToCurrency": "EUR", "StartDate": "2019-01-03T00:00:00Z", "Rate": 745.10, "ConversionFactor": "Hundred", "RateTypeDescription": "TEST"}'
PS C:\> Invoke-D365RestEndpoint -ServiceName "UserSessionService/AifUserSessionService/GetUserSessionInfo" -Payload $Payload -RetryTimeout "00:01:00"

This will invoke the REST endpoint in the Dynamics 365 Finance & Operations environment, and try for 1 minute to handle 429.
First the desired json data is put into the $Payload variable.
The ServiceName used for the import is "UserSessionService/AifUserSessionService/GetUserSessionInfo".
The $Payload variable is passed to the cmdlet.
It will only try to handle 429 retries for 1 minute, before failing.

.EXAMPLE
PS C:\> $Payload = '{"RateTypeName": "TEST", "FromCurrency": "DKK", "ToCurrency": "EUR", "StartDate": "2019-01-03T00:00:00Z", "Rate": 745.10, "ConversionFactor": "Hundred", "RateTypeDescription": "TEST"}'
PS C:\> Invoke-D365RestEndpoint -ServiceName "UserSessionService/AifUserSessionService/GetUserSessionInfo" -Payload $Payload -ThrottleSeed 2

This will invoke the REST endpoint in the Dynamics 365 Finance & Operations environment, and sleep/pause between 1 and 2 seconds.
First the desired json data is put into the $Payload variable.
The ServiceName used for the import is "UserSessionService/AifUserSessionService/GetUserSessionInfo".
The $Payload variable is passed to the cmdlet.
It will use the ThrottleSeed 2 to sleep/pause the execution, to mitigate the 429 pushback from the endpoint.

.NOTES
Tags: REST, Endpoint, Custom Service, Services

Expand All @@ -107,6 +151,10 @@ function Invoke-D365RestEndpoint {

[string] $PayloadCharset = "UTF-8",

[Timespan] $RetryTimeout = "00:00:00",

[int] $ThrottleSeed,

[Alias('$AadGuid')]
[string] $Tenant = $Script:ODataTenant,

Expand All @@ -130,6 +178,28 @@ function Invoke-D365RestEndpoint {
)

begin {
if ([System.String]::IsNullOrEmpty($SystemUrl)) {
Write-PSFMessage -Level Verbose -Message "The SystemUrl parameter was empty, using the Url parameter as the OData endpoint base address." -Target $SystemUrl
$SystemUrl = $Url
}

if ([System.String]::IsNullOrEmpty($Url) -or [System.String]::IsNullOrEmpty($SystemUrl)) {
$messageString = "It seems that you didn't supply a valid value for the Url parameter. You need specify the Url parameter or add a configuration with the <c='em'>Add-D365ODataConfig</c> cmdlet."
Write-PSFMessage -Level Host -Message $messageString -Exception $PSItem.Exception -Target $entityName
Stop-PSFFunction -Message "Stopping because of errors." -Exception $([System.Exception]::new($($messageString -replace '<[^>]+>', ''))) -ErrorRecord $_
return
}

if ($Url.Substring($Url.Length - 1) -eq "/") {
Write-PSFMessage -Level Verbose -Message "The Url parameter had a tailing slash, which shouldn't be there. Removing the tailling slash." -Target $Url
$Url = $Url.Substring(0, $Url.Length - 1)
}

if ($SystemUrl.Substring($SystemUrl.Length - 1) -eq "/") {
Write-PSFMessage -Level Verbose -Message "The SystemUrl parameter had a tailing slash, which shouldn't be there. Removing the tailling slash." -Target $Url
$SystemUrl = $SystemUrl.Substring(0, $SystemUrl.Length - 1)
}

if (-not $Token) {
$bearerParms = @{
Url = $Url
Expand Down Expand Up @@ -186,7 +256,9 @@ function Invoke-D365RestEndpoint {

try {
Write-PSFMessage -Level Verbose -Message "Executing http request against the REST endpoint." -Target $($restEndpoint.Uri.AbsoluteUri)
Invoke-RestMethod @params
Invoke-RequestHandler -Method POST -Uri $restEndpoint.Uri.AbsoluteUri -Headers $headers -ContentType "application/json;charset=$PayloadCharset" -Payload $Payload -RetryTimeout $RetryTimeout

if (Test-PSFFunctionInterrupt) { return }
}
catch {
$messageString = "Something went wrong while importing data through the REST endpoint for the entity: $ServiceName"
Expand All @@ -195,6 +267,10 @@ function Invoke-D365RestEndpoint {
return
}

if ($ThrottleSeed) {
Start-Sleep -Seconds $(Get-Random -Minimum 1 -Maximum $ThrottleSeed)
}

Invoke-TimeSignal -End
}
}
62 changes: 62 additions & 0 deletions d365fo.integrations/functions/remove-d365odataconfig.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@

<#
.SYNOPSIS
Remove an OData config

.DESCRIPTION
Removes an OData config from the configuration store

.PARAMETER Name
The name of the OData configuration you are about to remove from the configuration store

.PARAMETER Temporary
Instruct the cmdlet to only temporarily remove the OData configuration from the configuration store

.EXAMPLE
PS C:\> Remove-D365ODataConfig -Name "UAT"

This will create an new OData configuration with the name "UAT".

.NOTES
Tags: Integrations, Integration, Bearer Token, Token, OData, Configuration

Author: Mötz Jensen (@Splaxi)
#>

function Remove-D365ODataConfig {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true)]
[string] $Name,

[switch] $Temporary
)

$Name = $Name.ToLower()

if ($Name -match '\*') {
Write-PSFMessage -Level Host -Message "The name cannot contain <c='em'>wildcard character</c>."
Stop-PSFFunction -Message "Stopping because the name contains wildcard character."
return
}

if (-not ((Get-PSFConfig -FullName "d365fo.integrations.odata.*.name").Value -contains $Name)) {
Write-PSFMessage -Level Host -Message "An OData configuration with that name <c='em'>doesn't exists</c>."
Stop-PSFFunction -Message "Stopping because an OData configuration with that name doesn't exists."
return
}

$res = (Get-PSFConfig -FullName "d365fo.integrations.active.odata.config.name").Value

if ($res -eq $Name) {
Write-PSFMessage -Level Host -Message "The active OData configuration is the <c='em'>same as the one you're trying to remove</c>. Please set another configuration as active, before removing this one."
Stop-PSFFunction -Message "Stopping because the active OData configuration is the same as the one trying to be removed."
return
}

foreach ($config in Get-PSFConfig -FullName "d365fo.integrations.odata.$Name.*") {
Set-PSFConfig -FullName $config.FullName -Value ""

if (-not $Temporary) { Unregister-PSFConfig -FullName $config.FullName -Scope UserDefault }
}
}
42 changes: 34 additions & 8 deletions d365fo.integrations/tests/functions/Invoke-D365RestEndpoint.Tests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,32 @@
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipelineByPropertyName | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromRemainingArguments | Should -Be $False
}
It 'Should have the expected parameter RetryTimeout' {
$parameter = (Get-Command Invoke-D365RestEndpoint).Parameters['RetryTimeout']
$parameter.Name | Should -Be 'RetryTimeout'
$parameter.ParameterType.ToString() | Should -Be System.TimeSpan
$parameter.IsDynamic | Should -Be $False
$parameter.ParameterSets.Keys | Should -Be '__AllParameterSets'
$parameter.ParameterSets.Keys | Should -Contain '__AllParameterSets'
$parameter.ParameterSets['__AllParameterSets'].IsMandatory | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 3
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipeline | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipelineByPropertyName | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromRemainingArguments | Should -Be $False
}
It 'Should have the expected parameter ThrottleSeed' {
$parameter = (Get-Command Invoke-D365RestEndpoint).Parameters['ThrottleSeed']
$parameter.Name | Should -Be 'ThrottleSeed'
$parameter.ParameterType.ToString() | Should -Be System.Int32
$parameter.IsDynamic | Should -Be $False
$parameter.ParameterSets.Keys | Should -Be '__AllParameterSets'
$parameter.ParameterSets.Keys | Should -Contain '__AllParameterSets'
$parameter.ParameterSets['__AllParameterSets'].IsMandatory | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 4
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipeline | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipelineByPropertyName | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromRemainingArguments | Should -Be $False
}
It 'Should have the expected parameter Tenant' {
$parameter = (Get-Command Invoke-D365RestEndpoint).Parameters['Tenant']
$parameter.Name | Should -Be 'Tenant'
Expand All @@ -58,7 +84,7 @@
$parameter.ParameterSets.Keys | Should -Be '__AllParameterSets'
$parameter.ParameterSets.Keys | Should -Contain '__AllParameterSets'
$parameter.ParameterSets['__AllParameterSets'].IsMandatory | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 3
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 5
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipeline | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipelineByPropertyName | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromRemainingArguments | Should -Be $False
Expand All @@ -71,7 +97,7 @@
$parameter.ParameterSets.Keys | Should -Be '__AllParameterSets'
$parameter.ParameterSets.Keys | Should -Contain '__AllParameterSets'
$parameter.ParameterSets['__AllParameterSets'].IsMandatory | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 4
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 6
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipeline | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipelineByPropertyName | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromRemainingArguments | Should -Be $False
Expand All @@ -84,7 +110,7 @@
$parameter.ParameterSets.Keys | Should -Be '__AllParameterSets'
$parameter.ParameterSets.Keys | Should -Contain '__AllParameterSets'
$parameter.ParameterSets['__AllParameterSets'].IsMandatory | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 5
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 7
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipeline | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipelineByPropertyName | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromRemainingArguments | Should -Be $False
Expand All @@ -97,7 +123,7 @@
$parameter.ParameterSets.Keys | Should -Be '__AllParameterSets'
$parameter.ParameterSets.Keys | Should -Contain '__AllParameterSets'
$parameter.ParameterSets['__AllParameterSets'].IsMandatory | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 6
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 8
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipeline | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipelineByPropertyName | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromRemainingArguments | Should -Be $False
Expand All @@ -110,7 +136,7 @@
$parameter.ParameterSets.Keys | Should -Be '__AllParameterSets'
$parameter.ParameterSets.Keys | Should -Contain '__AllParameterSets'
$parameter.ParameterSets['__AllParameterSets'].IsMandatory | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 7
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 9
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipeline | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipelineByPropertyName | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromRemainingArguments | Should -Be $False
Expand All @@ -123,7 +149,7 @@
$parameter.ParameterSets.Keys | Should -Be '__AllParameterSets'
$parameter.ParameterSets.Keys | Should -Contain '__AllParameterSets'
$parameter.ParameterSets['__AllParameterSets'].IsMandatory | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 8
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 10
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipeline | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipelineByPropertyName | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromRemainingArguments | Should -Be $False
Expand All @@ -149,7 +175,7 @@
$parameter.ParameterSets.Keys | Should -Be '__AllParameterSets'
$parameter.ParameterSets.Keys | Should -Contain '__AllParameterSets'
$parameter.ParameterSets['__AllParameterSets'].IsMandatory | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 9
$parameter.ParameterSets['__AllParameterSets'].Position | Should -Be 11
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipeline | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromPipelineByPropertyName | Should -Be $False
$parameter.ParameterSets['__AllParameterSets'].ValueFromRemainingArguments | Should -Be $False
Expand All @@ -159,7 +185,7 @@
Describe "Testing parameterset __AllParameterSets" {
<#
__AllParameterSets -ServiceName
__AllParameterSets -ServiceName -Payload -PayloadCharset -Tenant -Url -SystemUrl -ClientId -ClientSecret -Token -EnableException -TimeoutSec
__AllParameterSets -ServiceName -Payload -PayloadCharset -RetryTimeout -ThrottleSeed -Tenant -Url -SystemUrl -ClientId -ClientSecret -Token -EnableException -TimeoutSec
#>
}

Expand Down
Loading