Integration tests fail too much

[mfisher87]

I've seen integration test failures on more than 50% of PRs I've worked on in the last couple weeks, many of which only touch docs or otherwise should not cause test breakage. The source of the breakage is usually collections moving around in CMR, e.g. migrating to the cloud. To fix we have to update our "expected" values.

Is this providing value that's worth the cost? I feel like we're testing CMR hasn't changed more than testing earthaccess.

Are there ways we could improve the resilience and usefulness of the tests?

[chuckwondo]

I have more than once expressed distaste for the volume of integration tests we have in this category, precisely because of this brittleness. We can look more specifically at how we can pare things down, but in general, I don't think we should be "testing" the ability to reach a broad swath of collections and/or granules. That seems to be the responsibility of the CMR team, not ours.

Without revisiting our large suite of tests, at this point I don't have more specific advice other than we need to stop "testing" against our laundry list of "popular" collections that are moving targets, thus subjecting ourselves to frequently failing integration tests.

[mfisher87]

💯

I think the breadth of the integration tests comes from our distrust of CMR to communicate breaking changes before making them. I think we don't need this much breadth to give us the comfort level we need though.