v6.0.0

[emmambd]

Highlights

Partial validation support for Flex

GTFS Flex is a proposal that ensures riders can discover demand responsive services and how to book them. Flex was officially adopted into GTFS this March. With this validator release, we offer partial validation support for feeds that include Flex data. This includes:

• Flex features in summary. If a feed has Flex data, you'll see its Flexible Services in the GTFS Features section of the validation report summary. Flex Features include Booking Rules, Predefined Routes with Deviation, Zone-Based Demand Responsive Services, and Fixed-Stops Demand Responsive Services.

• No more false positives! Before, feeds with Flex features would trigger certain notices when the data was actually valid, such as missing_required_field for a missing stop_id in stop_times.txt when location_group_id or location_id was provided. Now, any error or warning generated for your Flex feed is legitimate and should be fixed.

• Validation for booking_rules.txt. 10 new validation notices have been added to check data from booking_rules.txt

• Validation for polygons in locations.geojson. Checks that each polygon in locations.geojson conforms to the OpenGIS Simple Features Specification, section 6.1.11, as required by GTFS.

We'll continue to add more Flex validation rules in future releases. You can see the full list of validation rules here.

Notice changes

• 🔴 missing_required_file: feed_info.txt is now conditionally required when translations.txt exists. Clarification adopted in Update requirement for feed_info.txt google/transit#460.
• 🔴 missing_required_field for transfers.txt: from_stop_id and to_stop_id are no longer required for a transfer_type of 4 or 5. Clarification adopted in [Clarification] from/to_stop_id & from/to_trip_id description in transfers.txt google/transit#455.
• 🔴 foreign_key_violation: Now checks if network_id exists in fare_leg_rules.txt but not in routes.txt OR networks.txt. Before, there was a bug that meant network_id had to be included in both routes.txt AND networks.txt.
• 🟡 missing_timepoint_value: Modified to clarify that timepoint values should be set explicitly to 0 or 1, not empty. Clarification adopted into GTFS in Clarify intended use for timepoint in stop_times.txt google/transit#474. 🟡 missing_recommended_column was also removed as a result of this clarification.

Flex notice changes

• 🔴 missing_required_file: stops.txt is now conditionally required. If locations.geojson is not present, stops.txt is required.
• 🔴 missing_required_field: stop_id is conditionally required in stop_times.txt, in cases where the record does not include location_id or location_group_id.
• 🔴 missing_trip_edge: Only checks stop times records if they do not include start_pickup_dropoff_window or end_pickup_drop_off_window.
• 🔴 decreasing_or_equal_stop_time_distance: Skips stop times records with a location_id or location_group_id instead of a stop_id. Cases where shape_dist_traveled is provided with a location_id or location_group_id are triggered in the new 🔴 forbidden_shape_dist_traveled error.
• 🟡 stop_without_stop_time: Skips stops that are a part of a location group referenced in stop_times.txt.
• 🟡 stop_too_far_from_shape: Only checks stop times records with a stop_id.

New notices

• 🔴 bidirectional_exit_gate: Triggered when exit gates (pathway_mode=7 in pathways.txt) are bidirectional.
• 🔴 invalid_character: Triggered when text contains invalid characters, such as the replacement character ("�"). Check that text was properly encoded in UTF-8 as required by GTFS.
• 🟡 single_shape_point: Triggered when a shape in shapes.txt only includes one shape point.

Flex new notices

• 🔴 forbidden_shape_dist_traveled: Triggered when shape_dist_traveled is provided with a location_id or location_group_id in stop_times.txt.
• 🔴 forbidden_geography_id: Triggered when a stop_times.txt includes more than one geography id. Only one of stop_id, location_group_id or location_id can be defined for a given entry.

locations.geojson notices

• 🔴 malformed_json: Triggered when locations.geojson is malformed.
• 🔴 invalid_geometry: Triggered when polygon in locations.geojson is unparsable or invalid. Each polygon must be valid by the definition of the OpenGIS Simple Features Specification, section 6.1.11.
• 🔴 missing_required_element: Triggered when an element that's required in the geoJSON file is missing.
• 🔴 unsupported_geo_json_type: Triggered when geoJSON type is a value other than "FeatureCollection".
• 🔴 unsupported_feature_type: Triggered when feature type is a value other than "Feature".
• 🔴 unsupported_geometry_type: Triggered when the geometry type is a value other than "Polygon" or "MultiPolygon".

booking_rules.txt notices

• 🔴 forbidden_prior_day_booking_field_value: Triggered when a forbidden field value is present for a prior-day booking rule.
• 🔴 forbidden_prior_notice_start_day: prior_notice_start_day value is forbidden when prior_notice_duration_max is set.
• 🔴 forbidden_prior_notice_start_time: prior_notice_start_time value is forbidden when prior_notice_start_day value is not set
• 🔴 forbidden_real_time_booking_field_value: A forbidden field value is present for a real-time booking rule.
• 🔴 forbidden_same_day_booking_field_value: A forbidden field value is present for a same-day booking rule.
• 🔴 invalid_prior_notice_duration_min: The prior_notice_duration_max field value needs to be greater or equal to the prior_notice_duration_min field value.
• 🔴 missing_prior_notice_duration_min: prior_notice_duration_min value is required for same day booking_type
• 🔴 missing_prior_day_booking_field_value: prior_notice_last_day and prior_notice_last_time values are required for prior day booking_type.
• 🔴 missing_prior_notice_start_time: prior_notice_start_time value is required when prior_notice_start_day value is set
• 🔴 prior_notice_last_day_after_start_day: prior_notice_last_day should not be greater than the prior_notice_start_day.

Impacts on existing data

You can find the impact of this release on all existing datasets we have in the Mobility Database in this spreadsheet.

Other notable improvements

• Bug Fix: Validation report summary shows Unix time 0 for feed_start_date and feed_end_date when feed_info.txt exists with empty date values
• Bug Fix: Count appears as 1 for block even when no blocks exist
• Optimisation: Do not run validators on columns that are not present
• feat: Include service window in Summary report

New Contributors

• @praneethd7 made their first contribution in feat: New rule - SingleShapePointValidator for flagging shapes with a single shape point #1753

PR List

• feat: Changed notice migration generation. Added zip file to release. by @jcpitre in feat: Changed notice migration generation. Added zip file to release. #1736
• feat: documented the changes made in PR 1633 by @qcdyx in feat: documented the changes made in PR 1633 #1748
• Feat: 1639 release process for minor versions by @jcpitre in Feat: 1639 release process for minor versions #1657
• feat: notify mobility-feeds-api when the web validator is updated by @cka-y in feat: notify mobility-feeds-api when the web validator is updated #1760
• fix: set transfer from and to stop ids as conditional required fields by @davidgamez in fix: set transfer from and to stop ids as conditional required fields #1772
• feat: add performance assessment to acceptance tests by @cka-y in feat: add performance assessment to acceptance tests #1771
• feat: New rule - SingleShapePointValidator for flagging shapes with a single shape point by @praneethd7 in feat: New rule - SingleShapePointValidator for flagging shapes with a single shape point #1753
• Docs: Fix typo in FEATURES.md by @isabelle-dr in Docs: Fix typo in FEATURES.md #1777
• fix: Display empty values for feed_start_date and feed_end_date if there's… by @qcdyx in fix: Display empty values for feed_start_date and feed_end_date if there's… #1798
• fix: Count appears as 1 for block even when no blocks exist by @qcdyx in fix: Count appears as 1 for block even when no blocks exist #1802
• fix: Added a validator for bad networkId foreign key in fareLegRules by @jcpitre in fix: Added a validator for bad networkId foreign key in fareLegRules #1804
• feat: added new field for rule priority in GtfsFareLegRuleSchema by @qcdyx in feat: added new field for rule priority in GtfsFareLegRuleSchema #1806
• feat: 1791 feed infotxt should be added to missing required file when translationstxt exists by @qcdyx in feat: 1791 feed infotxt should be added to missing required file when translationstxt exists #1803
• feat: 1786 exit gates pathway mode7 must not be bidirectional by @qcdyx in feat: 1786 exit gates pathway mode7 must not be bidirectional #1808
• feat: 1776 add flex features to list by @qcdyx in feat: 1776 add flex features to list #1780
• feat: Flex - Removed required field false positive for stop_id + added Foreign key violation for location_groups_id by @jcpitre in feat: Flex - Removed required field false positive for stop_id + added Foreign key violation for location_groups_id #1834
• feat: add feed contact email to json and html report by @davidgamez in feat: add feed contact email to json and html report #1836
• feat: update missing_trip_edge for flex feed by @cka-y in feat: update missing_trip_edge for flex feed #1843
• feat: flex - forbidden_real_time_booking_field_value validation notice by @cka-y in feat: flex - forbidden_real_time_booking_field_value validation notice #1845
• feat: 1534 service window in summary report by @qcdyx in feat: 1534 service window in summary report #1837
• feat: add feedServiceWindow to JSON report by @qcdyx in feat: add feedServiceWindow to JSON report #1852
• fix: only display service window on HTML report, not start/end date by @qcdyx in fix: only display service window on HTML report, not start/end date #1855
• feat: flex - added forbidden_same_day_booking_field_value notice by @cka-y in feat: flex - added forbidden_same_day_booking_field_value notice #1847
• feat: changes + added documentation to features by @cka-y in feat: changes + added documentation to features #1844
• feat: 1829 flex invalid prior notice day range by @qcdyx in feat: 1829 flex invalid prior notice day range #1858
• fix: keep the notice description as is by @qcdyx in fix: keep the notice description as is #1863
• feat: added forbidden_prior_day_booking_field_value, invalid_prior_notice_duration_min and forbidden_prior_notice_start_day notices by @cka-y in feat: added forbidden_prior_day_booking_field_value, invalid_prior_notice_duration_min and forbidden_prior_notice_start_day notices #1860
• locations.geojson POC phase2: End-to-end partial support of json data by @jcpitre in locations.geojson POC phase2: End-to-end partial support of json data #1810
• feat: added support notice for flex feeds by @cka-y in feat: added support notice for flex feeds #1861
• feat: new booking_rules.txt validation notices by @cka-y in feat: new booking_rules.txt validation notices #1866
• fix: Predefined Routes with Deviation trigger + test by @cka-y in fix: Predefined Routes with Deviation trigger + test #1862
• feat: modify stops.txt to be conditionally required by @cka-y in feat: modify stops.txt to be conditionally required #1868
• docs: Update FEATURES.md by @emmambd in docs: Update FEATURES.md #1812
• docs: Update doc links in features.md by @emmambd in docs: Update doc links in features.md #1877
• docs: fix color typos in FEATURES.md by @emmambd in docs: fix color typos in FEATURES.md #1878
• feat: 1795 timepoint clarification by @qcdyx in feat: 1795 timepoint clarification #1867
• chore: add memory usage records to the JSON report by @davidgamez in chore: add memory usage records to the JSON report #1857
• feat: modified stop_too_far_from_shape so it only checks stop times record with a stop_id by @qcdyx in feat: modified stop_too_far_from_shape so it only checks stop times record with a stop_id #1884
• docs: Clarified below threshold language in notices by @emmambd in docs: Clarified below threshold language in notices #1886
• feat: Do not call validators for missing columns by @jcpitre in feat: Do not call validators for missing columns #1875
• feat: add memory metrics to acceptance tests by @davidgamez in feat: add memory metrics to acceptance tests #1874
• feat: Changed Fares Transfers to Fare Transfers by @qcdyx in feat: Changed Fares Transfers to Fare Transfers #1880
• feat: 1883 flex modify stop without stop time to exclude stops associated with a used location group by @qcdyx in feat: 1883 flex modify stop without stop time to exclude stops associated with a used location group #1887
• feat: Added a validator for forbidden shape_distance by @jcpitre in feat: Added a validator for forbidden shape_distance #1896
• feat:locations.geojson file parsing and geometry validation by @cka-y in feat:locations.geojson file parsing and geometry validation #1879
• fix: add exception for decreasing_or_equal_stop_time_distance by @davidgamez in fix: add exception for decreasing_or_equal_stop_time_distance #1895
• feat: 1840 invalid characters by @qcdyx in feat: 1840 invalid characters #1892
• feat: included network_id as part of the primary key for fare_leg_rules.txt by @qcdyx in feat: included network_id as part of the primary key for fare_leg_rules.txt #1894
• chore(deps): bump urllib3 from 1.26.6 to 1.26.19 in /scripts/mobility-database-harvester in the pip group across 1 directory by @dependabot in chore(deps): bump urllib3 from 1.26.6 to 1.26.19 in /scripts/mobility-database-harvester in the pip group across 1 directory #1899
• chore(deps): bump the pip group across 2 directories with 4 updates by @dependabot in chore(deps): bump the pip group across 2 directories with 4 updates #1900
• ci: aling matrix dimensions to max concurrent jobs by @davidgamez in ci: aling matrix dimensions to max concurrent jobs #1904
• chore(deps): bump requests from 2.32.2 to 2.32.3 in /scripts/mobility-database-harvester in the pip group across 1 directory by @dependabot in chore(deps): bump requests from 2.32.2 to 2.32.3 in /scripts/mobility-database-harvester in the pip group across 1 directory #1903

Full Changelog: v5.0.1...v6.0.0

---

This discussion was created from the release v6.0.0.