Migrated from single portfolio file to separated Products and portfolios causes AlreadyExistsException due to SCF replacing version ID

[jagexjake]

Describe the bug
Trying to swap from structure 4 to structure 3 from https://github.com/awslabs/aws-service-catalog-factory/tree/master/examples/portfolios
As it is unwiedly to manage large portfolio files

SCF then replaced the versions of SC products.

3587 | ERROR MainProcess worker#33 ---- START OF ERROR----
3588 | ERROR MainProcess worker#33 Task ProvisionProductTask:
3589 | ERROR MainProcess worker#33 account_id: '111111111111'
3590 | ERROR MainProcess worker#33 account_parameters: {}
3591 | ERROR MainProcess worker#33 dependencies_by_reference:
3592 | ERROR MainProcess worker#33 - create-policies
3593 | ERROR MainProcess worker#33 - portfolio-get-all-products-and-their-versions-111111111111-eu-west-1-oss-mandatory-catalog-factory-mandatory-product-portfolio
3594 | ERROR MainProcess worker#33 - describe-provisioning-parameters-111111111111-eu-west-1-oss-mandatory-catalog-factory-mandatory-product-portfolio-Guardrails-IAM-v1
... SSM LOGS
3644 | ERROR MainProcess worker#33 describe_provisioning_params_ref: describe-provisioning-parameters-111111111111-eu-west-1-oss-mandatory-catalog-factory-mandatory-product-portfolio-Guardrails-IAM-v1
3645 | ERROR MainProcess worker#33 execution: hub
3646 | ERROR MainProcess worker#33 launch_name: all-guardrails-iam
3647 | ERROR MainProcess worker#33 launch_parameters: {}
3648 | ERROR MainProcess worker#33 manifest_file_path: ./manifest-expanded.yaml
3649 | ERROR MainProcess worker#33 manifest_files_path: .
3650 | ERROR MainProcess worker#33 manifest_parameters: {}
3651 | ERROR MainProcess worker#33 manifest_task_reference_file_path: ./manifest-task-reference.json
3652 | ERROR MainProcess worker#33 need_to_provision: false
3653 | ERROR MainProcess worker#33 portfolio: oss-mandatory-catalog-factory-mandatory-product-portfolio
3654 | ERROR MainProcess worker#33 portfolio_get_all_products_and_their_versions_ref: portfolio-get-all-products-and-their-versions-111111111111-eu-west-1-oss-mandatory-catalog-factory-mandatory-product-portfolio
3655 | ERROR MainProcess worker#33 product: Guardrails-IAM
3656 | ERROR MainProcess worker#33 puppet_account_id: '111111111111'
3657 | ERROR MainProcess worker#33 region: eu-west-1
3658 | ERROR MainProcess worker#33 requested_priority: 0
3659 | ERROR MainProcess worker#33 retry_count: null
3660 | ERROR MainProcess worker#33 ssm_param_inputs: []
3661 | ERROR MainProcess worker#33 ssm_param_outputs: []
3662 | ERROR MainProcess worker#33 tags: []
3663 | ERROR MainProcess worker#33 task_reference: launches_all-guardrails-iam_111111111111_eu_west_1
3664 | ERROR MainProcess worker#33 version: v1
3665 | ERROR MainProcess worker#33 worker_timeout: null
3666 | ERROR MainProcess worker#33
3667 | ERROR MainProcess worker#33 Traceback (most recent call last):
3668 | ERROR MainProcess worker#33
3669 | ERROR MainProcess worker#33   File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/servicecatalog_puppet/waluigi/shared_tasks/workers/worker_requiring_scheduler.py", line 100, in worker_task
3670 | ERROR MainProcess worker#33     task.execute()
3671 | ERROR MainProcess worker#33
3672 | ERROR MainProcess worker#33   File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/servicecatalog_puppet/waluigi/task_mixins/task_executor_mixin.py", line 31, in execute
3673 | ERROR MainProcess worker#33     self.run()
3674 | ERROR MainProcess worker#33
3675 | ERROR MainProcess worker#33   File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/servicecatalog_puppet/workflow/launch/provision_product_task.py", line 180, in run
3676 | ERROR MainProcess worker#33     provisioned_product_id = ***.provision_product_with_plan(
3677 | ERROR MainProcess worker#33
3678 | ERROR MainProcess worker#33   File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/servicecatalog_puppet/***.py", line 326, in provision_product_with_plan
3679 | ERROR MainProcess worker#33     raise Exception(
3680 | ERROR MainProcess worker#33
3681 | ERROR MainProcess worker#33 Exception: [all-guardrails-iam] 111111111111:eu-west-1] :: Plan failed (CREATE_FAILED): AlreadyExistsException  ChangeSet SC-111111111111-pp-viziwocpzf2zk-rec-nq7b2573m4rmo cannot be created due to a mismatch with existing attribute ClientToken (Service: AmazonCloudFormation; Status Code: 400; Error Code: AlreadyExistsException; Request ID: 2911144a-8046-4ae6-95fd-0de86356b8ae; Proxy: null)
3682 | ERROR MainProcess worker#33
3683 | ERROR MainProcess worker#33 ---- END OF ERROR ----
3684 | INFO MainProcess worker#33 executed task [success]: launches_all-guardrails-iam_111111111111_eu_west_1 got lock to unlock resources
3685 | WARNING MainProcess worker#33 launches_all-guardrails-iam_111111111111_eu_west_1 tried to unlock SERVICE_CATALOG_DESCRIBE_PROVISIONED_PRODUCT_eu-west-1_OF_111111111111 but it wasn't present
3686 | INFO MainProcess scheduler receiving: [61]: launches_all-guardrails-iam_111111111111_eu_west_1, ERRORED

when querying the provisioned product:

aws servicecatalog describe-provisioned-product --name all-guardrails-iam
{
    "ProvisionedProductDetail": {
        "Name": "all-guardrails-iam",
        "Arn": "arn:aws:servicecatalog:eu-west-1:111111111111:stack/all-guardrails-iam/pp-viziwocpzf2zk",
        "Type": "CFN_STACK",
        "Id": "pp-viziwocpzf2zk",
        "Status": "AVAILABLE",
        "CreatedTime": "2020-04-15T19:18:25.137000+01:00",
        "IdempotencyToken": "66e24ad4-b212-4334-ac3f-6a8b248aa77f",
        "LastRecordId": "rec-nq7b2573m4rmo",
        "LastProvisioningRecordId": "rec-nq7b2573m4rmo",
        "LastSuccessfulProvisioningRecordId": "rec-nq7b2573m4rmo",
        "ProductId": "prod-nj3hshmqhoi2c",
        "ProvisioningArtifactId": "pa-gblpxt4yuaiki"
    },
    "CloudWatchDashboards": []
}

Can see the ID changed because SCF reprovisioned because it created new pipelines etc:

Therefore, this condition never gets hit:

aws-service-catalog-puppet/servicecatalog_puppet/workflow/launch/provision_product_task.py

Line 123 in abca997

if provisioning_artifact_id == version_id:

and need_to_provision stays as True because it never hits

aws-service-catalog-puppet/servicecatalog_puppet/workflow/launch/provision_product_task.py

Line 142 in abca997

self.need_to_provision = False

So then it tries to provision even though the parameters are the same and the product is already provisioned but the version id has changed

Is there something we have done wrong? or should there be an else statement on

aws-service-catalog-puppet/servicecatalog_puppet/workflow/launch/provision_product_task.py

Line 123 in abca997

if provisioning_artifact_id == version_id:

to check if the provisioned paramters and params to use match, don't need to provision?