[BUG] Error submitting workflow

[nocoli]

What is the bug?

Attempting to follow https://github.com/opensearch-project/opensearch-migrations/wiki/Backfill-Workflow and the workflow submit command is resulting in the following error in Cloudshell

(05:52:06) migration-console (~) -> workflow submit
NOT checking if all secrets have been created.  Run `workflow configure edit` to confirm
Initializing workflow from session: default
Submitting workflow to namespace: ma
ERROR:console_link.workflow.services.script_runner:Script failed with exit code 1
ERROR:console_link.workflow.services.script_runner:stderr: Error: Found 4 errors

Error submitting workflow: Command '['/root/configProcessor/createMigrationWorkflowFromUserConfiguration.sh', '/tmp/tmpcy9mouvd.yaml', '--prefix ma', '--etcd-endpoints http://etcd.ma.svc.cluster.local:2379']' returned non-zero exit status 1.
ERROR:console_link.workflow.commands.submit:Workflow submission failed
Traceback (most recent call last):
  File "/root/lib/console_link/console_link/workflow/commands/submit.py", line 139, in submit_command
    submit_result = runner.submit_workflow(config_yaml, args)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/lib/console_link/console_link/workflow/services/script_runner.py", line 205, in submit_workflow
    output = self.run_script("createMigrationWorkflowFromUserConfiguration.sh", None,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/lib/console_link/console_link/workflow/services/script_runner.py", line 113, in run_script
    return self.run(self.script_dir / script_name, input_data, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/lib/console_link/console_link/workflow/services/script_runner.py", line 75, in run
    result = subprocess.run(
             ^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/root/configProcessor/createMigrationWorkflowFromUserConfiguration.sh', '/tmp/tmpcy9mouvd.yaml', '--prefix ma', '--etcd-endpoints http://etcd.ma.svc.cluster.local:2379']' returned non-zero exit status 1.
ERROR:console_link.workflow.commands.submit:Unexpected error submitting workflow: 1
Traceback (most recent call last):
  File "/root/lib/console_link/console_link/workflow/commands/submit.py", line 139, in submit_command
    submit_result = runner.submit_workflow(config_yaml, args)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/lib/console_link/console_link/workflow/services/script_runner.py", line 205, in submit_workflow
    output = self.run_script("createMigrationWorkflowFromUserConfiguration.sh", None,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/lib/console_link/console_link/workflow/services/script_runner.py", line 113, in run_script
    return self.run(self.script_dir / script_name, input_data, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/lib/console_link/console_link/workflow/services/script_runner.py", line 75, in run
    result = subprocess.run(
             ^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/root/configProcessor/createMigrationWorkflowFromUserConfiguration.sh', '/tmp/tmpcy9mouvd.yaml', '--prefix ma', '--etcd-endpoints http://etcd.ma.svc.cluster.local:2379']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/lib/console_link/console_link/workflow/commands/submit.py", line 162, in submit_command
    ctx.exit(ExitCode.FAILURE.value)
  File "/.venv/lib64/python3.11/site-packages/click/core.py", line 738, in exit
    raise Exit(code)
click.exceptions.Exit: 1
Error: 1

What are your migration environments?

Source: Elasticsearch 6.8
Target: Opensearch 3.3
Migration Assistant: 2.6.1

How can one reproduce the bug?

Here is the template I am using

sourceClusters:
  my-source:
    endpoint: 
	  https://{SOURCE_ENDPOINT}.{REGION}.es.amazonaws.com
    version: "6.8"
    authConfig:
	  sigv4:
	    region: {REGION}
	    service: es

targetClusters:
  my-target:
    endpoint:
	  https://{TARGET_ENDPOINT}.{REGION}.es.amazonaws.com
    version: "3.3"
    authConfig:
	  sigv4:
	    region: {REGION}
	    service: es

migrations:
- sourceCluster: my-source
  targetCluster: my-target
  snapshotMigrations:
  - indices: ["docs-*"]
    metadataMigration:
	  enabled: true
    documentBackfill:
	  enabled: true

What is the expected behavior?

If it's an issue with the configuration I would expect an error that tells me what exactly is wrong with it.

Do you have any additional context?

Config is fairly basic, pretty much a copy and paste from the steps in the link above.

Connections look fine as per below:

(05:56:52) migration-console (~) -> console clusters connection-check
SOURCE CLUSTER
ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='6.8.0')
TARGET CLUSTER
ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='3.3.0')
(05:58:19) migration-console (~) -> console clusters cat-indices

WARNING: Cluster information may be stale. Use --refresh to update.

SOURCE CLUSTER
health status index                          uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_1                      x3Gp5oeJSIKVmB-l3Rs-0Q   1   1          0            0       522b           261b
green  open   docs-2 V3E7z6lWSaauawCWuKzU2w   5   1      58664            0     42.1mb           21mb
green  open   .tasks                         JkrS9ev-T3i5pb4GomnEqA   1   1         12            0     42.7kb         21.3kb
green  open   docs-1            QGrM3J9_QB-buiEurf_vWw   5   1  114663048     17135159    102.7gb         51.3gb

TARGET CLUSTER
health status index                          uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .plugins-ml-config             epvwuItXRyC59f1n3nIniA   1   1          1            0      8.1kb            4kb
green  open   .plugins-ml-jobs               WtfLH8svTse3eBWvWDYHJA   1   1          1            0     14.9kb          6.3kb
green  open   .opendistro-job-scheduler-lock RSJ97ZwhSfaON22el6mlKQ   1   1          1           11     59.6kb         37.8kb
green  open   .kibana_1                      6u3TGew_RXePdsp30B7Q7Q   1   1          1            0     10.5kb          5.2kb

I added an S3 snapshot repo to the config during debugging as per the example in the link above which resulted in the same error output above except for Error: Found 4 errors was changed to Error: Found 6 errors which is what is making me think there could be some trouble parsing the template

[AndreKurait]

Hi @nocoli, thank you for reaching out.

We have addressed this in two places, first, we're updating the wiki documentation to have more up-to-date guidance on sample workflow options.

Second, we fixed a bug where the workflow submit command did not give indication to what the errors were (which will be included in the next release)

You can see an example of the correct template in the latest youtube video: https://youtu.be/2efEdTbQ-wg?t=267

When you used workflow configure edit, did it you see the syntax validation from vim there?

For your case, something like the following should work:

sourceClusters:
  my-source:
    endpoint: "https://{SOURCE_ENDPOINT}.{REGION}.es.amazonaws.com"
    version: "ES 6.8"
    authConfig:
      sigv4:
        region: "{REGION}"
        service: es
    snapshotRepo:
      awsRegion: "{REGION}"
      s3RepoPathUri: "s3://migrations-default-{ACCOUNT_ID}-{STAGE}-{REGION}/snapshots"

targetClusters:
  my-target:
    endpoint: "https://{TARGET_ENDPOINT}.{REGION}.es.amazonaws.com"
    version: "OS 2.19"
    authConfig:
      sigv4:
        region: "{REGION}"
        service: es

migrationConfigs:
  - fromSource: my-source
    toTarget: my-target
    snapshotExtractAndLoadConfigs:
      - snapshotConfig:
          snapshotNameConfig:
            snapshotNamePrefix: "migration"
        createSnapshotConfig: {      s3RoleArn: "arn:aws:iam::{ACCOUNT_ID}:role/migration-eks-cluster-{STAGE}-{REGION}-snapshot-role"}
        migrations:
          - metadataMigrationConfig:
              indexAllowlist: ["regex:docs-.*"]
            documentBackfillConfig:
              indexAllowlist: ["regex:docs-.*"]

**Note: we've recently moved s3RoleArn from createSnapshotConfig to repoConfig from 2.6.2 onwards

[nocoli]

Thanks for the response @AndreKurait! I will try again shortly.

Yes workflow configure edit did have syntax validation - I was a little confused initially as workflow configure sample --load resulted in JSON where the property names didn't quite match the YAML in the documentation. Eg: migrationConfigs in the JSON but migrations in the YAML. This lines up with your comment above.

[nocoli]

Thanks @AndreKurait for your comment above - I was able to get a working configuration by looking at the one in the video.

Is there some documentation on the configuration that I've missed? If not, I think some documentation describing each property, what it does, whether is required, etc would be super valuable to this tool.

I've got some more migrations to run over the next little while which may require different configs to the example in the video.

[AndreKurait]

We have the schema on the migration console, it currently gives this as the openapi schema, but we don't have detailed documentation yet for these values

{
  "type": "object",
  "properties": {
    "skipApprovals": {
      "type": "boolean",
      "default": false
    },
    "sourceClusters": {
      "type": "object",
      "additionalProperties": {
        "type": "object",
        "properties": {
          "endpoint": {
            "type": "string",
            "pattern": "^(?:https?:\\/\\/[^:\\/\\s]+(:\\d+)?(\\/)?)?$",
            "default": ""
          },
          "allowInsecure": {
            "type": "boolean",
            "default": false
          },
          "version": {
            "type": "string",
            "pattern": "^(?:ES [125678]|OS [123])(?:\\.[0-9]+)+$"
          },
          "authConfig": {
            "anyOf": [
              {
                "type": "object",
                "properties": {
                  "basic": {
                    "type": "object",
                    "properties": {
                      "secretName": {
                        "type": "string",
                        "pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$"
                      }
                    },
                    "required": [
                      "secretName"
                    ]
                  }
                },
                "required": [
                  "basic"
                ]
              },
              {
                "type": "object",
                "properties": {
                  "sigv4": {
                    "type": "object",
                    "properties": {
                      "region": {
                        "type": "string"
                      },
                      "service": {
                        "type": "string",
                        "default": "es"
                      }
                    },
                    "required": [
                      "region"
                    ]
                  }
                },
                "required": [
                  "sigv4"
                ]
              },
              {
                "type": "object",
                "properties": {
                  "mtls": {
                    "type": "object",
                    "properties": {
                      "caCert": {
                        "type": "string"
                      },
                      "clientSecretName": {
                        "type": "string"
                      }
                    },
                    "required": [
                      "caCert",
                      "clientSecretName"
                    ]
                  }
                },
                "required": [
                  "mtls"
                ]
              }
            ]
          },
          "snapshotRepo": {
            "type": "object",
            "properties": {
              "awsRegion": {
                "type": "string",
                "description": "The AWS region that the bucket reside in (us-east-2, etc)"
              },
              "endpoint": {
                "type": "string",
                "pattern": "(?:^(http|localstack)s?:\\/\\/[^/]*\\/?$)?",
                "default": "",
                "description": "Override the default S3 endpoint for clients to connect to. Necessary for testing, when S3 isn't used, or when it's only accessible via another endpoint"
              },
              "s3RepoPathUri": {
                "type": "string",
                "pattern": "^s3:\\/\\/[a-z0-9][a-z0-9.-]{1,61}[a-z0-9](\\/[a-zA-Z0-9!\\-_.*'()/]*)?$",
                "description": "s3://BUCKETNAME/PATH"
              },
              "repoName": {
                "type": "string",
                "default": "migration_assistant_repo"
              },
              "s3RoleArn": {
                "type": "string",
                "pattern": "^(arn:aws:iam::\\d{12}:(user|role|group|policy)\\/[a-zA-Z0-9+=,.@_-]+)?$",
                "default": "",
                "description": "IAM role ARN to assume when accessing S3 for snapshot operations"
              }
            },
            "required": [
              "awsRegion",
              "s3RepoPathUri"
            ]
          },
          "proxy": {
            "type": "object",
            "properties": {
              "loggingConfigurationOverrideConfigMap": {
                "type": "string",
                "default": ""
              },
              "otelCollectorEndpoint": {
                "type": "string",
                "default": "http://otel-collector:4317"
              },
              "setHeaders": {
                "type": "array",
                "items": {
                  "type": "string"
                }
              }
            }
          }
        },
        "required": [
          "version"
        ]
      }
    },
    "targetClusters": {
      "type": "object",
      "additionalProperties": {
        "type": "object",
        "properties": {
          "endpoint": {
            "type": "string",
            "pattern": "^https?:\\/\\/[^:\\/\\s]+(:\\d+)?(\\/)?$"
          },
          "allowInsecure": {
            "type": "boolean",
            "default": false
          },
          "version": {
            "type": "string",
            "pattern": "^(?:ES [125678]|OS [123])(?:\\.[0-9]+)+$"
          },
          "authConfig": {
            "anyOf": [
              {
                "type": "object",
                "properties": {
                  "basic": {
                    "type": "object",
                    "properties": {
                      "secretName": {
                        "type": "string",
                        "pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$"
                      }
                    },
                    "required": [
                      "secretName"
                    ]
                  }
                },
                "required": [
                  "basic"
                ]
              },
              {
                "type": "object",
                "properties": {
                  "sigv4": {
                    "type": "object",
                    "properties": {
                      "region": {
                        "type": "string"
                      },
                      "service": {
                        "type": "string",
                        "default": "es"
                      }
                    },
                    "required": [
                      "region"
                    ]
                  }
                },
                "required": [
                  "sigv4"
                ]
              },
              {
                "type": "object",
                "properties": {
                  "mtls": {
                    "type": "object",
                    "properties": {
                      "caCert": {
                        "type": "string"
                      },
                      "clientSecretName": {
                        "type": "string"
                      }
                    },
                    "required": [
                      "caCert",
                      "clientSecretName"
                    ]
                  }
                },
                "required": [
                  "mtls"
                ]
              }
            ]
          }
        },
        "required": [
          "endpoint",
          "version"
        ]
      }
    },
    "migrationConfigs": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "skipApprovals": {
            "type": "boolean",
            "default": false
          },
          "fromSource": {
            "type": "string"
          },
          "toTarget": {
            "type": "string"
          },
          "snapshotExtractAndLoadConfigs": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "name": {
                  "type": "string",
                  "pattern": "^[a-zA-Z][a-zA-Z0-9]*",
                  "default": ""
                },
                "createSnapshotConfig": {
                  "type": "object",
                  "properties": {
                    "indexAllowlist": {
                      "type": "array",
                      "items": {
                        "type": "string"
                      },
                      "default": []
                    },
                    "maxSnapshotRateMbPerNode": {
                      "type": "number",
                      "default": 0
                    },
                    "loggingConfigurationOverrideConfigMap": {
                      "type": "string",
                      "default": ""
                    }
                  }
                },
                "snapshotConfig": {
                  "type": "object",
                  "properties": {
                    "snapshotNameConfig": {
                      "anyOf": [
                        {
                          "type": "object",
                          "properties": {
                            "externallyManagedSnapshot": {
                              "type": "string"
                            }
                          },
                          "required": [
                            "externallyManagedSnapshot"
                          ]
                        },
                        {
                          "type": "object",
                          "properties": {
                            "snapshotNamePrefix": {
                              "type": "string"
                            }
                          },
                          "required": [
                            "snapshotNamePrefix"
                          ]
                        }
                      ]
                    }
                  },
                  "required": [
                    "snapshotNameConfig"
                  ]
                },
                "migrations": {
                  "type": "array",
                  "items": {
                    "type": "object",
                    "properties": {
                      "name": {
                        "type": "string",
                        "pattern": "^[a-zA-Z][a-zA-Z0-9]*",
                        "default": ""
                      },
                      "metadataMigrationConfig": {
                        "type": "object",
                        "properties": {
                          "componentTemplateAllowlist": {
                            "type": "array",
                            "items": {
                              "type": "string"
                            },
                            "default": []
                          },
                          "indexAllowlist": {
                            "type": "array",
                            "items": {
                              "type": "string"
                            },
                            "default": []
                          },
                          "indexTemplateAllowlist": {
                            "type": "array",
                            "items": {
                              "type": "string"
                            },
                            "default": []
                          },
                          "allowLooseVersionMatching": {
                            "type": "boolean",
                            "default": true
                          },
                          "clusterAwarenessAttributes": {
                            "type": "number",
                            "default": 1
                          },
                          "loggingConfigurationOverrideConfigMap": {
                            "type": "string",
                            "default": ""
                          },
                          "multiTypeBehavior": {
                            "anyOf": [
                              {
                                "type": "string",
                                "enum": [
                                  "NONE"
                                ]
                              },
                              {
                                "type": "string",
                                "enum": [
                                  "UNION"
                                ]
                              },
                              {
                                "type": "string",
                                "enum": [
                                  "SPLIT"
                                ]
                              }
                            ],
                            "default": "NONE"
                          },
                          "otelCollectorEndpoint": {
                            "type": "string",
                            "default": "http://otel-collector:4317"
                          },
                          "output": {
                            "anyOf": [
                              {
                                "type": "string",
                                "enum": [
                                  "HUMAN_READABLE"
                                ]
                              },
                              {
                                "type": "string",
                                "enum": [
                                  "JSON"
                                ]
                              }
                            ],
                            "default": "HUMAN_READABLE"
                          },
                          "transformerConfigBase64": {
                            "type": "string",
                            "default": ""
                          },
                          "skipEvaluateApproval": {
                            "type": "boolean",
                            "default": false
                          },
                          "skipMigrateApproval": {
                            "type": "boolean",
                            "default": false
                          }
                        }
                      },
                      "documentBackfillConfig": {
                        "type": "object",
                        "properties": {
                          "indexAllowlist": {
                            "type": "array",
                            "items": {
                              "type": "string"
                            },
                            "default": []
                          },
                          "podReplicas": {
                            "type": "number",
                            "default": 1
                          },
                          "loggingConfigurationOverrideConfigMap": {
                            "type": "string",
                            "default": ""
                          },
                          "allowLooseVersionMatching": {
                            "type": "boolean",
                            "default": true,
                            "description": ""
                          },
                          "docTransformerConfigBase64": {
                            "type": "string",
                            "default": ""
                          },
                          "documentsPerBulkRequest": {
                            "type": "number",
                            "default": 2147483647
                          },
                          "initialLeaseDuration": {
                            "type": "string",
                            "default": "PT10M"
                          },
                          "maxConnections": {
                            "type": "number",
                            "default": 10
                          },
                          "maxShardSizeBytes": {
                            "type": "number",
                            "default": 85899345920
                          },
                          "otelCollectorEndpoint": {
                            "type": "string",
                            "default": "http://otel-collector:4317"
                          },
                          "skipApproval": {
                            "type": "boolean",
                            "default": false
                          },
                          "resources": {
                            "type": "object",
                            "nullable": true,
                            "properties": {
                              "limits": {
                                "type": "object",
                                "properties": {
                                  "cpu": {
                                    "type": "string",
                                    "pattern": "^[0-9]+m$",
                                    "description": "CPU quantity in millicores (e.g., '100m', '500m')"
                                  },
                                  "memory": {
                                    "type": "string",
                                    "pattern": "^[0-9]+((E|P|T|G|M)i?|Ki|k)$",
                                    "description": "Memory quantity with unit (e.g., '512Mi', '2G')"
                                  },
                                  "ephemeral-storage": {
                                    "type": "string",
                                    "pattern": "^[0-9]+((E|P|T|G|M)i?|Ki|k)$",
                                    "description": "Storage quantity with unit (e.g., '10Gi', '5G')"
                                  }
                                },
                                "required": [
                                  "cpu",
                                  "memory"
                                ],
                                "description": "Resource upper bound for a container"
                              },
                              "requests": {
                                "type": "object",
                                "properties": {
                                  "cpu": {
                                    "type": "string",
                                    "pattern": "^[0-9]+m$",
                                    "description": "CPU quantity in millicores (e.g., '100m', '500m')"
                                  },
                                  "memory": {
                                    "type": "string",
                                    "pattern": "^[0-9]+((E|P|T|G|M)i?|Ki|k)$",
                                    "description": "Memory quantity with unit (e.g., '512Mi', '2G')"
                                  },
                                  "ephemeral-storage": {
                                    "type": "string",
                                    "pattern": "^[0-9]+((E|P|T|G|M)i?|Ki|k)$",
                                    "description": "Storage quantity with unit (e.g., '10Gi', '5G')"
                                  }
                                },
                                "required": [
                                  "cpu",
                                  "memory"
                                ],
                                "description": "Resource lower bound for a container"
                              }
                            },
                            "required": [
                              "limits",
                              "requests"
                            ],
                            "description": "Resource limits and requests for RFS container"
                          }
                        }
                      }
                    }
                  },
                  "minItems": 1
                }
              },
              "required": [
                "snapshotConfig",
                "migrations"
              ]
            },
            "minItems": 1
          },
          "replayerConfig": {
            "type": "object",
            "properties": {
              "speedupFactor": {
                "type": "number",
                "default": 1.1
              },
              "podReplicas": {
                "type": "number",
                "default": 1
              },
              "authHeaderOverride": {
                "type": "string",
                "default": ""
              },
              "loggingConfigurationOverrideConfigMap": {
                "type": "string",
                "default": ""
              },
              "resources": {
                "type": "object",
                "properties": {
                  "limits": {
                    "type": "object",
                    "properties": {
                      "cpu": {
                        "type": "string",
                        "pattern": "^[0-9]+m$",
                        "description": "CPU quantity in millicores (e.g., '100m', '500m')"
                      },
                      "memory": {
                        "type": "string",
                        "pattern": "^[0-9]+((E|P|T|G|M)i?|Ki|k)$",
                        "description": "Memory quantity with unit (e.g., '512Mi', '2G')"
                      },
                      "ephemeral-storage": {
                        "type": "string",
                        "pattern": "^[0-9]+((E|P|T|G|M)i?|Ki|k)$",
                        "description": "Storage quantity with unit (e.g., '10Gi', '5G')"
                      }
                    },
                    "required": [
                      "cpu",
                      "memory"
                    ],
                    "description": "Resource upper bound for a container"
                  },
                  "requests": {
                    "type": "object",
                    "properties": {
                      "cpu": {
                        "type": "string",
                        "pattern": "^[0-9]+m$",
                        "description": "CPU quantity in millicores (e.g., '100m', '500m')"
                      },
                      "memory": {
                        "type": "string",
                        "pattern": "^[0-9]+((E|P|T|G|M)i?|Ki|k)$",
                        "description": "Memory quantity with unit (e.g., '512Mi', '2G')"
                      },
                      "ephemeral-storage": {
                        "type": "string",
                        "pattern": "^[0-9]+((E|P|T|G|M)i?|Ki|k)$",
                        "description": "Storage quantity with unit (e.g., '10Gi', '5G')"
                      }
                    },
                    "required": [
                      "cpu",
                      "memory"
                    ],
                    "description": "Resource lower bound for a container"
                  }
                },
                "default": {
                  "limits": {
                    "cpu": "2000m",
                    "memory": "4000Mi"
                  },
                  "requests": {
                    "cpu": "2000m",
                    "memory": "4000Mi"
                  }
                },
                "required": [
                  "limits",
                  "requests"
                ],
                "description": "Resource limits and requests for replayer container."
              }
            }
          }
        },
        "required": [
          "fromSource",
          "toTarget"
        ]
      },
      "minItems": 1
    }
  },
  "required": [
    "sourceClusters",
    "targetClusters",
    "migrationConfigs"
  ]
}

[nocoli]

Thanks for sharing.

I currently have 2 large RFS migrations in progress. workflow status shows 200 ├── ✗ checkHistoricalBackfillCompletion (Failed) entries but the RFS process is still running as I can see the number of searchable documents in the cluster continues to climb. I think I can see on this line that it will no longer continue checking the status of the workflow

Can I please get some guidance on what my options are from here? I want to just leave the current workflows continue to run - is there a way I can force a new status check once I think it's done?

[nocoli]

One of my backfills has now completed, the document count matches the source cluster. From digging into this repo it looks like the only thing that happens after this is some pod cleanup? @AndreKurait Could I please confirm that if the document count match between source/target that the migration is essentially complete?

re pod cleanup - the whole EKS cluster will be cleaned up when I remove the Cloudformation stack

[nocoli]

Sorry for pinging this one - just keen to get it sorted as we're currently paying for double the domains while we're figuring out what needs to be done to complete the migration before we spin down the source clusters

[gregschohn]

One of my backfills has now completed, the document count matches the source cluster. From digging into this repo it looks like the only thing that happens after this is some pod cleanup? @AndreKurait Could I please confirm that if the document count match between source/target that the migration is essentially complete?

re pod cleanup - the whole EKS cluster will be cleaned up when I remove the Cloudformation stack

The line that you referenced earlier tells argo to only retry that step 200 times, which would come out to around a day, which is admittedly probably too short for some migrations.

The migration workflow should be monitoring progress and should wrap up the reindex-from-snapshot (RFS) tasks, which as you correctly identified, will delete the replicaset for the RFS pods. At that point, document counts should be the same for the indices that you've migrated. If there were any errors putting documents on the target, those numbers would be different. It would also be different if the source cluster had additions or deletions after the snapshot was taken. Many customers use document count comparisons for their acceptance test. I'd advise at least spot-checking some of the documents, especially when switching versions to verify that the contents are as-expected.

Deleting the EKS cluster will clean up everything from the migration except for the .migrations_working_state* indices on your target cluster. Deleting the CloudFormation stack will delete the EKS cluster.

[gregschohn]

I've added this bug to lengthen the wait period on the checks.