Stop using sources.csv in acceptance tests since it will be discontinued

[jcpitre]

Describe the problem

Currently we used the old csv to run acceptance tests. See harvest_latest_versions.py.

This file will be discontinued soon (MobilityData/mobility-database-catalogs#862

We have to start using feeds_v2.csv instead.
Here is the schema for that file: here

Note that the feeds_v2.csv contains quite a bit more feeds than the original sources.csv. This means the first time we change the file will result in large differences in the acceptance tests.

References:
MobilityData/mobility-database-catalogs#767

• Filter out deprecated feeds

Ideal state would be to have a coverage tool to assess what feeds have all fields covered by the spec to ensure we're including all cases necessary to test the validator.

Automatically update deprecated feeds to point to redirect version in acceptance test feed list.

[emmambd]

@skalexch We want to use a sample of the total number of feeds in the Mobility Database for acceptance tests (about 1000) so it doesn't take forever to run. We want to have a good representation of Mobility database data to evaluate the new validation rules on. It would be helpful to get your thoughts on this.

Cohorts that could be critical:

• At least 50-100 feeds that have each feature
• Feeds that are very large file size
• Feeds that come from high-ridership operators and feeds that have very small counts

We should filter out deprecated feeds.

[skalexch]

@emmambd those cohorts make sense. I also suggest leaving 100-200 at the of those 1000 as a random sample every time, but that might cause unpredictable behaviour.

I created a dashboard in the data monitoring section containing sorted feeds by size. We can use that for the 2nd cohort.

For the first cohort, the number of feeds that contain all features is probably low if not 0. Maybe we can sample from feeds that have base add-ons, then add at least 50 Flex feeds and 50 fare feeds. At the same time, I keep thinking of potential cases where we introduce a validation rule for Fares v2 and it does not run on all fare feeds. E.g. If we had introduced Rider Categories rules and none of the fare feeds used contain Rider Categories.

Another issue is that if we adopt a new feature, the acceptance tests will not run on every feed. therefore to be 100% sure we didn't make a mistake, we have to retroactively search for feeds containing the new feature and open them up.

It depends on the purpose of acceptance tests. If they are used to check for false positives and unintended changing behaviour, then a sample might risk removing that.

[emmambd]

@skalexch This is a good start to defining cohorts!

I agree that having samples would help. Here's what I initially meant in "50-100 feeds that have each feature": 50-100 feeds that cover the full spectrum of features available across them. I was not clear but I did not mean all 50-100 have every feature.

The core purpose of the acceptance tests is to check if the notice logic we implement is wrong. You've seen this from reviewing the acceptance test summaries and cross checking with real world data to see if there were edge cases we missed.

[jcpitre]

The core purpose of the acceptance tests is to check if the notice logic we implement is wrong.

Nitpick: I would say the core purpose of the acceptance tests is to detect regressions, i.e. do changes made to the code alter the validation results in unforeseen ways.

[emmambd]

I wish Github would let me react with the 💯 emoji. 😊