Skip to content

v1.4.0: new conformance tests are too slow #4101

@howardjohn

Description

@howardjohn

The v1.4.0 conformance tests adds new 'Weight' tests. These are very very slow on a fast cluster:

--- PASS: TestGatewayConformance (38.44s)
    --- PASS: TestGatewayConformance/MeshGRPCRouteWeight (18.11s)
        --- PASS: TestGatewayConformance/MeshGRPCRouteWeight/Requests_should_have_a_distribution_that_matches_the_weight (18.10s)
    --- PASS: TestGatewayConformance/MeshHTTPRouteWeight (19.35s)
        --- PASS: TestGatewayConformance/MeshHTTPRouteWeight/Requests_should_have_a_distribution_that_matches_the_weight (19.35s)

In v1.3 the entire suite took 135.81s for us. So these 2 new tests add 27% additional time.

There is no need for this test to take so long.

Running 500 simulations of the test logic: echo {0..500} | xargs -P 10 -n1 kubectl exec echo-v1-67d4b56bd4-vdjc2 -- client grpc://echo:7070 --qps takes 7s. Note the test intentionally adds 'sleeps' for some reason to the calls, hence why its 17s I imagine.

Doing 500 calls without port-forward takes 1s.

Using 1 call with --count 500 takes 0.037s.

I would highly recommend we switch to --count

Metadata

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions