Kafka behaving differently depending on whether its run with ctrl+f5 or DistributedApplicationTestingBuilder

[Kylemiller995]

Hello all,

I have the following app host config for kafka,

var builder = DistributedApplication.CreateBuilder(args);

var kafka = builder
    .AddContainer("kafka", "confluentinc/cp-kafka:7.6.1")
    .WaitFor(zookeeper)
    .WithHttpEndpoint(port: 29092, targetPort: 29092, name: "kafka")
    .WithHttpEndpoint(port: 9092, targetPort: 9092, name: "bootstrap")
    .WithHttpEndpoint(port: 9997, targetPort: 9997, name: "metrics")
    .WithEnvironment("KAFKA_ZOOKEEPER_CONNECT", "zookeeper:2181")
    .WithEnvironment("KAFKA_LISTENER_SECURITY_PROTOCOL_MAP", "PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT")
    .WithEnvironment("KAFKA_ADVERTISED_LISTENERS", "PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092")
    .WithEnvironment("KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR", "1")
    .WithEnvironment("KAFKA_INTER_BROKER_LISTENER_NAME", "PLAINTEXT")
    .WithEnvironment("KAFKA_MIN_INSYNC_REPLICAS", "1")
    .WithEnvironment("KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL", "http://schema-registry:8081");

var schemaRegistry = builder
    .AddContainer("schema-registry", "confluentinc/cp-schema-registry:latest")
    .WaitFor(kafka)
    .WaitFor(zookeeper)
    .WithHttpEndpoint(8081, 8081)
    .WithEnvironment("SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS", "kafka:29092")
    .WithEnvironment("SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL", "zookeeper:2181")
    .WithEnvironment("SCHEMA_REGISTRY_HOST_NAME", "schema-registry")
    .WithEnvironment("SCHEMA_REGISTRY_LISTENERS", "http://schema-registry:8081,http://localhost:8081");

var seedData = builder.AddProject<Projects.LocalSetup>("seed-data")
    .WithEnvironment("SchemaRegistryUrl", schemaRegistry.Resource.GetEndpoint("http"))
    .WithEnvironment("KafkaUrl", kafka.Resource.GetEndpoint("kafka"))
    .WaitFor(kafka)
    .WaitFor(schemaRegistry);
    
    var app = builder.Build();
    app.Run();

When running the app host from ide, my seed-data project is able to connect with kafka and seed my data, however when running from my integration test the seed data project is unable to connect:

%3|1744038961.018|ERROR|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: 1/1 brokers are down
%3|1744038961.018|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Unknown error (after 2036ms in state CONNECT)
%3|1744038964.005|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv6#[::1]:9092 failed: Unknown error (after 2013ms in state CONNECT)
%3|1744038964.005|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: localhost:9092/bootstrap: Connect to ipv6#[::1]:9092 failed: Unknown error (after 2013ms in state CONNECT)

Test code handling the DistributedApplicationTestingBuilder:

var builder = await DistributedApplicationTestingBuilder
                .CreateAsync<Projects.AppHost>();
            using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(180));

            await using var app = await builder.BuildAsync(CancellationToken.None);

            await app.StartAsync(CancellationToken.None);
            // Act
            await app.ResourceNotifications.WaitForResourceAsync(
                "seed-data",
                KnownResourceStates.Finished,
                CancellationToken.None);
            var httpClient = app.CreateHttpClient("sut");

            await app.ResourceNotifications.WaitForResourceHealthyAsync(
                "sut",
                cts.Token);

When running in the ctrl+f5 I notice that kafka.Resource.GetEndpoint("kafka") always resolves to localhost:9092, whereas when running with the DistributedApplicationTestingBuilder, the proxy port seems to be used instead. Oddly enough hard coding localhost:9092 when running DistributedApplicationTestingBuilder doesn't work and neither did dynamically registering KAFKA_ADVERTISED_LISTENERS to be the proxy endpoint e.g:

var selfEndpoint = kafka.GetEndpoint("bootstrap");
    context.EnvironmentVariables["KAFKA_ADVERTISED_LISTENERS"] =  $"PLAINTEXT://kafka:29092,PLAINTEXT_HOST://{selfEndpoint.Host}:{selfEndpoint.Port.ToString()}";;

Any ideas on how I would get my DistributedApplicationTestingBuilder to behave the same way as running the app host with f5?