Getting "Multiple actions matched" if adding refrence to 2 different Web API projects to test project

[lisamariet]

AspNetCore.TestHost version 2.2.0
AspNetCore version 2.2
Swagger version 4.0.1

I have 2 web api projects (inside a Service Fabric app) that both has a controller with the same name. But they do different things. One api is Admin api and the other is client consumer - read only api.
Both has a controller with the same name and same method for GetAll() and the route is the same. But because the APIs spin up on different ports (configured by service fabric) they are in reality not the same route.
It works fine to run these APIs in Service Fabric cluster and calling the APIs.

This however is not something the TestHost seems to notice and I don't know how to configure the TestHost for this to work either.
I first setup TestHost with the AdminApi and created test methods, and this works like a charm. Then when I added the ClientApi project that has the same GetAll method with the same route, my tests for that endpoint in AdminAPi started failing with the errormessage "Multiple actions matched".
I mean I have not done anything else in test project but add a reference to the ClientApi project, The Startup that is used is still from AdminApi project. So simply just adding a reference to ClientApi... makes test fail.

I assume the problem is that TestHost scans for all /controller/ folders in any project that is referenced, regardless of the code in Startup. And since I don't know how to configure the Test Startup to have different ports or a different base path I cannot get this to work.
I would like to write integration tests that calls AdminApi and ClientApi in the same test. This seems impossible.

public class AdminApiTestContext : TestContextBase
	{
		public AdminApiTestContext()
		{
			ServiceFabricMocks = new ServiceFabricMocks();

			var builder = new WebHostBuilder()
					.UseStartup<AdminApiTestStartupCopy>()
					.UseEnvironment("Local");

			TestServer = new TestServer(builder);
			ApiClient = TestServer.CreateClient();
		}
        }

This inhertis from the AdminApit projects Startup.cs file and overrides methods because a need to have a different setup in test.

public class AdminApiTestStartupCopy : Startup
	{
		public AdminApiTestStartupCopy(IConfiguration configuration) : base(configuration)
		{
		}

		public override void ConfigureServices(IServiceCollection services)
		{
			services
				.AddSwaggerGen(c =>
				{
					c.SwaggerDoc("api", getInfo);
					c.OperationFilter<SwaggerDefaultsOperationFilter>();
					c.CustomSchemaIds(DefaultSchemaIdSelector);
					c.EnableAnnotations();
				})
				.AddMvc()
				.AddJsonOptions(options => { options.SerializerSettings.Formatting = Formatting.Indented; });


			services.Configure<MvcOptions>(options =>
			{
				options.Filters.Add(new CorsAuthorizationFilterFactory("AllowSpecificOrigin"));
			});

			services.AddMvcCore();
                }

                public static IList<Action<SwaggerUIOptions>> SwaggerUiOptionDelegates { get; } = new List<Action<SwaggerUIOptions>>();
             
                public override void Configure(IApplicationBuilder app, IHostingEnvironment env)
		{
			app.UseAuthentication();
			var swaggerBasePath = new TestConfigHelper().Swagger.BaseUrl;
			
                        app.UseSwagger(c =>
				{
					c.PreSerializeFilters.Add((swaggerDoc, httpReq) => swaggerDoc.BasePath = swaggerBasePath);
				});

                        SwaggerUiOptionDelegates.Add(options =>
			{
				options.SwaggerEndpoint(swaggerBasePath + "/api/swagger.json", apiInfo.Title);
			});
			var swaggerUiOptionsAction = Delegate.Combine(SwaggerUiOptionDelegates.ToArray());
			app.UseSwaggerUI((Action<SwaggerUIOptions>)swaggerUiOptionsAction);

			app.UseMvcWithDefaultRoute();
		}
        }