Tfs Shared Query Processor

[giscus[bot]]

This discussion is for questions, feedback, and community support related to the Tfs Shared Query Processor.

Quick Reference

• Class Name: TfsSharedQueryProcessor
• Friendly Name: Tfs Shared Query
• Documentation: Tfs Shared Query Processor

How to Use This Discussion

• Ask Questions: If you need help configuring or using this component
• Share Examples: Post your configuration examples or use cases
• Report Issues: Describe any problems you encounter (though bugs should be reported as GitHub Issues)
• Request Features: Suggest improvements or new features

Related Documentation

Please refer to the official documentation for detailed configuration options and examples.

[MicE44]

!!ACTION REQUIRED!! we no longer support the v150 config schema...

[MrHinsh]

Can you hep me understand what you are unable to do?

[MicE44]

Sure, the documentation is outdated. You can't use it with version 16.1.

[MrHinsh]

What cant you use? What error are you getting? What are you trying?

Please provide both the config and the log.

[MrHinsh]

I just ran a TfsSharedQueryProcessor with 16.1.

Please provide your config and log.

[MicE44]

(eddited by @MrHinsh for formatting)

I cant use this Processors.
Error:

[09:16:02 INF] [16.1.0] Beginning run of 1 processors
[09:16:02 INF] [16.1.0] Processor: TfsSharedQueryProcessor
[09:16:02 INF] [16.1.0] Migration Context Start: TfsSharedQueryProcessor
[09:16:02 DBG] [16.1.0] TfsLanguageMapOptions::Validate::Success
[09:16:02 DBG] [16.1.0] TfsAuthenticationOptions::Validate::Success
[09:16:02 DBG] [16.1.0] TfsEndpointOptionsValidator::Validate::Success
[09:16:02 FTL] [16.1.0] Error while running TfsSharedQueryProcessor
System.InvalidCastException: Unable to cast object of type 'MigrationTools.Clients.TfsTeamProjectEndpoint' to type 'MigrationTools.Endpoints.TfsEndpoint'.
   at MigrationTools.Processors.TfsSharedQueryProcessor.get_Source() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsSharedQueryProcessor.cs:line 35
   at MigrationTools.Processors.TfsSharedQueryProcessor.InternalExecute() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.TfsObjectModel\Processors\TfsSharedQueryProcessor.cs:line 46
   at MigrationTools.Processors.Infrastructure.Processor.Execute() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools\Processors\Infrastructure\Processor.cs:line 99
Error logged to Elmah.io!
!! Check for latest version - We fix issues constantly - If not, please create a discussion on https://github.com/nkdAgility/azure-devops-migration-tools/discussions so we can get this fixed !!
[09:16:04 INF] [16.1.0] TfsSharedQueryProcessor completed in 00:00:00
[09:16:04 ERR] [16.1.0] TfsSharedQueryProcessor The Processor MigrationEngine entered the failed state...stopping run
[09:16:04 INF] [16.1.0] Command ExecuteMigrationCommand completed in 00:00:02.3883806
[09:16:04 INF] [16.1.0] Check the logs for errors: C:\Tools\MigrationTools\logs\20250401091601

I try to migrate all Shared Queries.

Config:

{
  "Serilog": {
    "MinimumLevel": "Debug"
  },
  "MigrationTools": {
    "Version": "16.1",
    "Endpoints": {
      "Source2": {
        "EndpointType": "TfsTeamProjectEndpoint",
        "Collection": "https://dev.azure.com/shsds",
        "Project": "CS-Cinematic-Anatomy",
        "AllowCrossProjectLinking": false,
        "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemId",
        "Authentication": {
          "AuthenticationMode": "AccessToken",
          "AccessToken": "XXX",
          "NetworkCredentials": {
            "UserName": "",
            "Password": "",
            "Domain": ""
          }
        },
        "LanguageMaps": {
          "AreaPath": "Area",
          "IterationPath": "Iteration"
        }
      },
      "Target2": {
        "EndpointType": "TfsTeamProjectEndpoint",
        "Collection": "https://dev.azure.com/SHS-CS-DDO",
        "Project": "CS-Cinematic-Anatomy",
        "TfsVersion": "AzureDevOps",
        "Authentication": {
          "AuthenticationMode": "AccessToken",
          "AccessToken": "XXX",
          "NetworkCredentials": {
            "UserName": "",
            "Password": "",
            "Domain": ""
          }
        },
        "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemId",
        "AllowCrossProjectLinking": false,
        "LanguageMaps": {
          "AreaPath": "Area",
          "IterationPath": "Iteration"
        }
      }
    },
    "CommonTools": {
      "WorkItemTypeMappingTool": {
        "Enabled": true,
        "Mappings": {
	"ABB": "ABB",
	"Assumption": "Assumption",
    "Bug": "Bug",
	"Claim": "Claim",
	"Disclaimer": "Disclaimer",
	"Epic": "Epic",
	"Feature": "Feature",
	"Headline": "Headline",
	"Issue": "Issue",
	"Market Requirement": "Market Requirement",
	"OTS": "OTS",
	"Product Backlog Item": "Product Backlog Item",
	"Risk": "Risk",
	"Risk Mitigation": "Risk Mitigation",
	"Risk Safety": "Risk Safety",
	"Task": "Task",
	"Software Requirement": "Software Requirement",
	"SOP": "SOP",
	"Test Case": "Test Case",
	"User Story": "Work Package Item"
        }
      },
      "StringManipulatorTool": {
        "Enabled": true,
        "MaxStringLength": 1000000,
		//Fix SingleString "MaxStringLength": 255,
        "Manipulators": [
          {
            "$type": "RegexStringManipulator",
            "Enabled": true,
            "Pattern": "[^( -~)\n\r\t]+",
            "Replacement": "",
            "Description": "Remove invalid characters from the end of the string"
          }
        ]
      },
      "TfsAttachmentTool": {
        "RefName": "TfsAttachmentTool",
        "Enabled": true,
        "ExportBasePath": "c:\\temp\\WorkItemAttachmentExport",
        "MaxRevisions": 480000000
      },
      "TfsChangeSetMappingTool": {
        "Enabled": false,
        "File": "C:\\temp\\ChangeSetMappingFile.json"
      },
      "FieldMappingTool": {
        "Enabled": true,
        "FieldMaps": [
          {
            "FieldMapType": "FieldtoFieldMap",
            "ApplyTo": ["Defect"],
            "sourceField": "Siemens.Defect.Justification",
            "targetField": "Microsoft.VSTS.CMMI.Justification"
          },
			{
            "FieldMapType": "FieldValueMap",
            "ApplyTo": ["*"],
            "defaultValue": "New",
            "sourceField": "System.State",
            "targetField": "System.State",
            "valueMapping": 
			{
        "New": "New",
		"In Work": "In Work",
		"In Play": "In Play",
		"Active": "Active",
		"Closed": "Closed",
		"Resolved": "Resolved",
        "Removed": "Removed",
		"For Analysis": "For Analysis",
		"Qualified": "Qualified",
		"Done": "Done",
		"Blocked": "Blocked",
		"In Planning": "In Planning",
		"Design": "Design",
		"Inactive": "Inactive",
		"Next Up": "Next Up",
		"Ready": "Ready",
		"To Do": "To Do",
		"Completed": "Completed",
		"Proposed": "Proposed",
		"In Progress": "In Progress"
      }
		  }
        ]
      },
      "TfsGitRepositoryTool": {
        "Enabled": true,
        "Mappings": null
		/*{
          "CS-Cinematic-Anatomy": "CS-Cinematic-Anatomy"
        }*/
      },
      "TfsNodeStructureTool": {
        "Enabled": true,
        "Areas": {
		  "Filters": null,
          //"Filters": [ " *\\Team 1,*\\Team 2\\**" ],
          "Mappings": null
		  /*{
            "CS-Cinematic-Anatomy([\\\\]?.*)$": "CS-Cinematic-Anatomy",
			"^1c78a84c-cf99-4d5b-8ba9-e0cb7d6d1f8e([\\\\]?.*)$": "Ultrasound"
          }*/
        },
        "Iterations": {
          "Filters": null,
          "Mappings": null
		  /*{
            "CS-Cinematic-Anatomy([\\\\]?.*)$": "CS-Cinematic-Anatomy",
			"^1c78a84c-cf99-4d5b-8ba9-e0cb7d6d1f8e([\\\\]?.*)$": "Ultrasound"
          } */
        }, 
        "ShouldCreateMissingRevisionPaths": true,
        "ReplicateAllExistingNodes": false
      },
      "TfsRevisionManagerTool": {
        "Enabled": true,
        "ReplayRevisions": true,
        "MaxRevisions": 0
		// Req Fix        "MaxRevisions": 7
		//EPIC/Feature Fix "MaxRevisions": 34
      },
      "TfsTeamSettingsTool": {
        "Enabled": true,
        "MigrateTeamSettings": true,
        "UpdateTeamSettings": true,
        "MigrateTeamCapacities": true,
        "Teams": null
      }
    },
    "Processors": [
      {
        "ProcessorType": "TfsWorkItemMigrationProcessor",
        "Enabled": false,
        "UpdateCreatedDate": true,
        "UpdateCreatedBy": true,
        "WIQLQuery": "SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = @TeamProject  AND [System.ID] >= '1' AND [System.WorkItemType] NOT IN ('Test Plan','Test Suite') ORDER BY [System.ID] asc",
        // AND [System.ID] IN ('389','415','416','1032','4519','9574','9575','10520','10521','10813','61701','67214','67326','67325','67324','40618','40870','40906','40907','40921','40922','40871','40908','40913','40923','40924','40872','40909','40911','40914','40915','40925','40926','40928','40929','40877','40930','40911','40931','40878','40919','40933','40934','40879','40920')
		// AND [System.ID] NOT IN ('389','61700')
		// AND [System.ChangedDate] >= '2023-12-17T00:00:00.00Z'
		// AND [System.ID] NOT IN ('563419','674128')
	  // AND [Siemens.Defect.Product] = 'VSIM'
	  // AND [System.ID] >= '440407' 
	  // AND [System.AreaPath] UNDER 'XRay\\Hepton'		
	  // AND [System.Tags] Contains 'migrated'
	  // AND [System.IterationPath] UNDER 'XRay\\Hepton' OR [System.IterationPath] IN ('XRay\\Hepton\\AT2306','XRay\\Hepton\\AT2309','XRay\\Hepton\\AT2312','XRay\\Hepton\\AT2403','XRay\\Hepton\\AT2406')
	  // AND [System.WorkItemType] IN ('Epic','Task','User Story','Feature')
	  // AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan','Shared Steps','Shared Parameter','Feedback Request')
	  // AND [System.State] NOT IN ('Closed') 
	  // AND [System.ID]  IN ('63','64','65','68')
		"FixHtmlAttachmentLinks": false,
        "WorkItemCreateRetryLimit": 5,
        "FilterWorkItemsThatAlreadyExistInTarget": true,
        "PauseAfterEachWorkItem": false,
        "AttachRevisionHistory": false,
        "GenerateMigrationComment": false,
        "SourceName": "Source2",
        "TargetName": "Target2",
        //"WorkItemIDs": [ 12 ],
		"WorkItemIDs": null,
        "MaxGracefulFailures": 0,
        "SkipRevisionWithInvalidIterationPath": false,
        "SkipRevisionWithInvalidAreaPath": false
      },
	  {
      "ProcessorType": "TfsTestPlansAndSuitesMigrationProcessorOptions",
	  "Enabled": false,
      "ReplayRevisions": true,
      "PrefixProjectToNodes": false,
	  //This Part is for Test Plans and Suites (run 2nd)!!!
      //"TestPlanQueryBit": "Select '*' From TestPlan Where [System.ID] >= '58348' AND [System.AreaPath] UNDER 'SENSE\\Magellan Team'",
	  "TestPlanQueryBit": "AreaPath UNDER 'CS-Cinematic-Anatomy' AND [System.ID] >= '303000' AND [System.ID] NOT IN ('290801','303131')",
	  // AND [System.ID] NOT IN ('77828')
	  //'Test Case','Test Plan','Test Suite' AND [System.State] NOT IN ('Closed') AND [System.WorkItemType] IN ('Test Plan','Test Suite') AND [System.AreaPath] UNDER 'SENSE\\Magellan Team',
	  "WIQLOrderBit": "[System.ID] asc",
	  "OnlyElementsWithTag": "Migrate",
	  "SourceName": "Source2",
      "TargetName": "Target2",
	  "RemoveInvalidTestSuiteLinks": false,
	  "UseCommonNodeStructureEnricherConfig": false,
	  "NodeBasePaths": ["123456"],
	  "AreaMaps": {},
      "IterationMaps": {}
    },
	 {
        "ProcessorType": "TfsSharedQueryProcessor",
        "Enabled": true,
        "PrefixProjectToNodes": false,
        "SharedFolderName": "Shared Queries",
        "SourceToTargetFieldMappings": null,
        "SourceName": "Source2",
        "TargetName": "Target2"
     },
	{
		"ProcessorType": "TfsWorkItemDeleteProcessorOptions",
		"Enabled": false,
		"WIQLQuery": "SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = @TeamProject AND [System.WorkItemType] IN ('Defect','Test Case','Test Plan','Test Suite') AND [System.AreaPath] UNDER 'syngo Application Software\\Migration_DefectsXRay'  ORDER BY [System.ChangedDate] desc",
		// DANGER!, but good for deleting Test Cases!
		// AND [System.WorkItemType] IN ('Test Case') AND [System.AreaPath] = 'BranchDa\\BranchDA'
		"WorkItemIDs": null,
		"SourceName": "Source2",
        "TargetName": "Target2",
		"FilterWorkItemsThatAlreadyExistInTarget": false,
		"PauseAfterEachWorkItem": false,
		"WorkItemCreateRetryLimit": 0
	}
    ]
  }
}

Thanks for your support.

[MicE44]

If you want share your working v16.1 config with me/us.
Maybe I can compare it, thanks.

[MrHinsh]

Your config is a mess of unessesary mappings and

Here is mine:

{
  "Serilog": {
    "MinimumLevel": "Debug"
  },
  "MigrationTools": {
    "Version": "16.0",
    "Endpoints": {
      "Source": {
        "EndpointType": "TfsTeamProjectEndpoint",
        "Collection": "https://dev.azure.com/nkdagility-preview/",
        "Project": "migrationSource1",
        "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemId",
        "Authentication": {
            "AuthenticationMode": "Prompt"
        }
      },
      "Target": {
        "EndpointType": "TfsTeamProjectEndpoint",
        "Collection": "https://dev.azure.com/nkdagility-preview/",
        "Project": "migrationTest55",
        "TfsVersion": "AzureDevOps",
        "Authentication": {
          "AuthenticationMode": "Prompt"
        },
        "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemId"
      }
    },
    "Processors": [
        {
            "ProcessorType": "TfsSharedQueryProcessor",
            "Enabled": true,
            "SourceName": "Source",
            "TargetName": "Target"
        }
    ]
  }
}

[vifor2]

Hello @MicE44, I had a similar issue, try changing your EndpointType to TfsEndpoint for both the source and the target like so:

    "Endpoints": {
      "Source2": {
        "EndpointType": "TfsEndpoint",
        [...]
      },
      "Target2": {
        "EndpointType": "TfsEndpoint",
        [...]
      },

[MrHinsh]

Its interesting that I did not need to do that! But hey! Whatever works right!

[potatojaisiladki]

@vifor2 - Thank you. Your comment saved my time :)

[MicE44]

Thanks, Ok the EndpointType is the problem.

I need "EndpointType": "TfsEndpoint", for "ProcessorType": "TfsSharedQueryProcessor",

and

"EndpointType": "TfsTeamProjectEndpoint", for "ProcessorType": "TfsWorkItemMigrationProcessor",

[12:58:27 FTL] [16.1.0] Validation of your configuration failed:
The configuration entry at 'MigrationTools:Processors' did not pass validation!
Please check the following failures:
-The Endpoint 'Target2' specified for TfsWorkItemMigrationProcessor is of the wrong type! TfsTeamProjectEndpoint was expected.

[satano]

@MrHinsh Guys here are right. It really throws that exception and I do not understand how could yout configuration work. The problem is this:

azure-devops-migration-tools/src/MigrationTools.Clients.TfsObjectModel/Processors/TfsSharedQueryProcessor.cs

Line 35 in 9c3eb05

public new TfsEndpoint Source => (TfsEndpoint)base.Source;

The Source and Target properties are casted to TfsEndpoint, which is not possible. I wanted to fix it, but it is not simple, because TfsSharedQueryProcessor is using things from TfsEndpoint which are not in TfsTeamProjectEndpoint.

[MrHinsh]

Ive not run Shared Queries in a while but TfsTeamProjectEndpoint inherts from TfsEndpoint

[satano]

No, that is not true. The inheritance is this:

• TfsTeamProjectEndpoint → Endpoint<TfsTeamProjectEndpointOptions>
• TfsEndpoint → GenericTfsEndpoint<TfsEndpointOptions> → Endpoint<TfsEndpointOptions>

So they have common ancestor Endpoint. But the can not be casted from one to another. And the TfsSharedQueryProcessor uses methods and properties from GenericTfsEndpoint so even cast to base Endpoint will not help.

[satano]

Not to mention, that even the base is different, because they have different options. So now to use TfsSharedQueryProcessor, we must use TfsEndpoint. If someone uses work item migration and also shared queries, she must define two sources and two targets. One set as TfsEndpoint for shared queries and the other set as TfsTeamProjectEndpoint for work items.

[ankitsharma21]

How can we move Shared Queries from one project to a child folder of shared Queries in another project ?

[jaymichaelancheta-cell]

changing to TfsEndpoint allows the program to continue but i keep getting this error and not sure why? 23:47:56 FTL] [16.2.11] Error while running TfsSharedQueryProcessor
System.InvalidOperationException: TF237012: Root node may not be modified.
at Microsoft.TeamFoundation.WorkItemTracking.Client.QueryFolder.Add(QueryItem item)
at MigrationTools.Processors.TfsSharedQueryProcessor.MigrateFolder(QueryHierarchy targetHierarchy, QueryFolder sourceFolder, Please help