GTFS-ServiceChanges vs extending GTFS-TripUpdates

[LeoFrachet]

Hi GTFS Community,

A decision has to be made regarding service changes, and the feedback of everybody is heavily needed.

Currently, many things cannot be done in real-time, including:

• Case A: Updating time of scheduled arrival/departure
• Case B: Updating trip_headsign, trip_short_name, wheelchair_accessible or bikes_allowed
• Case C: Adding a new stop, a new shape, a new route or a new transfer.
• Case D: Have unactivated trips in the GTFS, that we trigger when needed.

What we currently have on the table are:

• Existing GTFS-TripUpdates (allowing to provide real-time update for arrival & departure).
• A proposal to extend GTFS-TripUpdates (Add prediction certainty #111 , including UPDATE_SCHEDULE) by @TransitApp, handling case A.
• A proposal to define a new feed, called GTFS-ServiceChanges, using the CSV GTFS structure, handling cases A, B, C & D (bit.ly/gtfs-service-changes).

So what do we do?

Option 0: Use only the existing formats. Update static GTFS more often.

Will require overhaul of GTFS consumer GTFS pipeline to be able to process GTFS every hour or more.

Option 1: We beef up GTFS-TripUpdates to support the cases A, B, C & D (Cc @Stefan)

We already have a proposal for case A (#111 ), we could easily extend the TripUpdate object to handle case B, but no proposal so far to handle case C.

Advantages:

• Small change to existing pipeline for case A & B.

Disadvantages:

• Handling cases C & D forces us to redefine almost every GTFS concepts slightly differently, therefore duplicating the spec.

Option 2: We keep GTFS-TripUpdates for real-time update only, and use GTFS-ServiceChanges to change schedule data (aka cases A, B, C & D)

This is what I had in mind when I drafted GTFS-ServiceChanges, and this is the current state of the GTFS-ServiceChanges proposal.

Advantage:

• It makes a clear distinction between schedule updates and real-time updates
• It keep TripUpdate implementation as they are
• As branding, it pushes agency to adopt this new feed as a new feature for their rider
• It doesn’t increase the size of the TripUpdates feed (but it may be not an issue @Stefan @gcamp)

Disadvantage

• Some overlapping for adding feed with GTFS-TripUpdates
• It’s a distinct feed, and force to open a new endpoint, with a new URL

Option 3: Middle ground proposed by Transit (Cc @gcamp & @juanborre)

• Extend GTFS-TripUpdate for case A
• Use GTFS-ServiceChanges for cases C & D.

=> What about case B (trip headsigns, short names…)? Should we extend TripUpdate also for this or not?

(Link to the GTFS-ServiceChanges proposal: bit.ly/gtfs-service-changes)

[juanborre]

Hi GTFS Community,

What agencies need more often is to update the schedules or the predictions of a trip and choosing how to display those times to the user, either saying it is real time and the vehicle is being tracked or saying it is an updated scheduled and the vehicle is not being tracked.

That information can be provided easily in the TripUpdates feed. The ServiceChanges can be used less often to convey bigger changes in the data.

Changing headsigns or other trip specific data would add too many things in the TripUpdates feed. Although it is conceivable, it can be better to split that to a different feed that aims at bigger modifications of the data.

It is true that the three options are valid, but the option 3 solves the problem of updating schedules and deciding which predictions are real time or not relatively quickly while being scalable and easier to implement for all the agencies/AVL that already have a TripUpdates feed.

[abyrd]

I approach this proposal with caution, because it will increase the complexity of the GTFS specification, creating multiple ways to define the same objects, and therefore demand an increase in the complexity of software consuming the GTFS format. I acknowledge the point of view that this is necessary to provide up-to-date information to passengers. But time should be taken to consider the impact carefully, because once this change happens and some people adopt it there will be no going back to the simpler world where a single file contains a complete snapshot of a transit system's schedules, stops, etc.

The proposal linked to above (
http://bit.ly/gtfs-service-changes) begins with the following statements:

1. GTFS cannot be processed more than once an hour, hardy not more than once a day.
2. GTFS-RT feeds cannot contain huge amount of information since they are fetched every 30s (and also they do not allow all kind of service changes).

GTFS Service Changes are then presented as a solution to these issues, but I do not see them as self-evident axioms.

1. Is there a fundamental reason why GTFS cannot be processed multiple times in a day? For many applications, and for many feeds, it takes at most a few minutes to completely handle a feed. I acknowledge that for very large feeds like all of the Netherlands or the New York City region, fully processing and integrating a whole new data set can be more on the hour-long time scale. You probably wouldn't want to re-publish feeds every hour of every day, but for the occasional schedule revision I don't see an inherent problem with just re-publishing a new feed.

2. As I posted on the service changes proposal, this second statement assumes that a polling (pull) method is used to receive the GTFS-RT messages. In bigger systems I believe only a streaming (differential push) method makes sense. This is done in the Netherlands, Norway, and Finland and consumed by OpenTripPlanner. A very large number of service patches could be pushed out using such a system without encountering any bandwidth constraints.

I don't rule out the possibility that a third time scale is needed between the static and real-time feeds, but it seems questionable. We should be absolutely sure that effective operation can't be achieved with two time scales before sacrificing simplicity.

[abyrd]

I initially lean toward Option 1: revise GTFS-RT TripUpdates to support all cases A, B, C, and D.

You list as disadvantages the fact that handling cases C & D "forces us to redefine almost every GTFS concepts slightly differently, therefore duplicating the spec."

Can you clarify this? Why would all GTFS concepts need to be redefined, and why would the new definitions be different?

It seems quite healthy to me to continue work on GTFS-RT to make it capable of a wide range of revisions to the scheduled data.

[barbeau]

What I like about Option 2 (as @LeoFrachet says) is that it's a clear separation of functionality between the feeds, where TripUpdates would be used only for real-time predictions, and ServiceChanges would be used for any static modifications to the network. There are backwards compatibility issues for consumers that aren't aware of new enums when you start piling new functionality on top of TripUpdates. Separating these network/schedule updates into a new channel with a clear purpose ensures that unaware consumers don't start interpreting schedule updates as real-time predictions.

Additionally, if consumers want to consume just VehiclePositions and Service Changes and generate their own predictions, they can do this without needing to parse the TripUpdates as well.

[abyrd]

On 31 Oct 2018, at 03:18, Sean Barbeau ***@***.***> wrote: What I like about Option 2 (as @LeoFrachet <https://github.com/LeoFrachet> says) is that it's a clear separation of functionality between the feeds, where TripUpdates would be used only for real-time predictions, and ServiceChanges would be used for any static modifications to the network.

I like the terminology @LeoFrachet used here: "a clear distinction between schedule updates and real-time updates” (or predictions), i.e. GTFS-RT is for real-time only, GTFS is for schedules only, the hypothetical service updates are also for schedules only. This line of reasoning does have merit. But it’s not entirely clear to me why it’s important to have a one-to-one correspondence between data formats and categories of data. Already the concept of real-time includes two different kinds of information: empirical observations of vehicle location and lateness, and predictions of the downstream effect of that empirical information. We want to distinguish clearly between a) schedules (planned service), b) observations of actual position and delay, and c) predictions of future service. But I don’t see any reason why GTFS-RT couldn’t contain all of these. To my mind the core distinction between GTFS and RT is the time scale over which they’re valid. GTFS provides a slowly changing baseline that is patched in real-time by a stream of RT messages. Even this division into two time scales is essentially an optimization. The number of layers of patching at different time scales is somewhat arbitrary. I’m not sure it’s a good idea to introduce additional layers (time-scales) of patching as optimization responding to current operational details (bandwidth, push vs. pull RT etc.) because we’ll then be stuck with the additional complexity forever.

There are backwards compatibility issues for consumers that aren't aware of new enums when you start piling new functionality on top of TripUpdates. Separating these network/schedule updates into a new channel with a clear purpose ensures that unaware consumers don't start interpreting schedule updates as real-time predictions.

All of these approaches have backward compatibility issues. They all require significant changes to all GTFS consumer software. But the introduction of ServiceChanges is arguably more disruptive. GTFS-RT is currently interpreted as patches on top of a specific version of a GTFS-static feed. If it’s redefined as patches on top of a set of ServiceChanges on top of a GTFS-static feed, any software that doesn’t know about ServiceChanges can/will have an incorrect view of the system. The new challenge also emerges of ensuring that the correct versions of each layer are combined. On the other hand, if schedule updates are sent as GTFS-RT, older software interpreting those schedule updates as predictions seems comparatively harmless. It still results in the end user seeing the service that the operator expects to provide them. I’m not entirely opposed to the concept of ServiceChanges, but I do think the irreversible impact on the unity and complexity of the GTFS ecosystem should be considered carefully.

…

-Andrew

[harringtonp]

Having read both this and proposal #111 the following questions arise:

1. How often would an operator need to update a schedule ?
2. How long in advance are they aware of the new schedule ?

The answer to both these questions is presumably along the lines of "as long as a piece of string" but if the general feeling was that the answer for 1) were "Not too often" and 2) "Usually more than a day"
then would the existing mechanism (as mentioned above) of just publishing a new schedule be best.

If this works in over 90% of cases is there really a need to introduce a new layer. I've seen cases where the schedule is updated on several consecutive nights and it may well be to cater for cases like this.

As regards user experience, I think if you can show reliably where a bus or train is on a map at a given time AND the user can see the scheduled or predicted times of arrival for the preceding and following vehicle stop, they will form an opinion themselves as to when the vehicle will arrive at their stop regardless of the schedule or predictions. The point I am getting at is if extra complexity is introduced, confidence would be needed that it can improve the user experience.

[abyrd]

Changing headsigns or other trip specific data would add too many things in the TripUpdates feed. Although it is conceivable, it can be better to split that to a different feed that aims at bigger modifications of the data.

I acknowledge that this opinion is shared by several people, but before making a spec change it will be important to justify that opinion. When you say this is "too many things" what is the threshold for "too many"? What specific technical or conceptual restrictions make it excessive to include this information in real-time updates?

Why would it be considered excessive to augment an existing spec, but not excessive to define another separate spec containing the same information, requiring significant software development and additional complexity in every GTFS consumer and data pipeline?

[abyrd]

As someone working on producing and consuming large GTFS static and realtime feeds, @skinkie I think your opinion would be valuable here. Do you find it advantageous to add another time-scale of patches with a new format between GTFS-static and GTFS-RT? I see that option 1 above says CC @Stefan and I wonder if that is supposed to be you, because the Github user by that name seems inactive.

[skinkie]

I discussed this with Leo and did some software development for exchanging a day worth of GTFS data inside tripUpdates. In my opinion tripUpdates should be extended with functionality for adding all possible GTFS static fields, opposed to define a new format. Because no matter what the outcome is with ServiceChanges, there are cases we must update some stuff of existing trips in realtime, which is currently not supported.

But in general my opinion is: this can be exchanged with SIRI-PT, what does justify to make a GTFS-RT alternative?

[abyrd]

@skinkie my sense is: many people are bothered by the huge catch-all nature of the Transmodel/SIRI ontology and the verbosity of its fetch/subscription mechanism, and GTFS-RT is a chance to complete a more compact spec that covers the 95% of common passenger information cases.

SIRI is more of an alternative to using GTFS-RT at all. As long as people see benefit to staying in the GTFS-static + GTFS-RT world, I would say it makes sense to complete the functionality available in that pair to cover some high percentage of common use cases.

[LeoFrachet]

I'm thrilled to see this conversation moving forward! Thanks @harringtonp, @skinkie & @abyrd for your contributions!

A few answers here.

How often would an operator need to update a schedule ?

For years now we have producers updating their schedules every day (e.g. WMATA in US-DC). We started to speak about ServiceChanges when some big producers started to speak about updating their (big) GTFS every hours. From what I've heard from the GTFS consumers side of the industry, nobody is ready for that. But I fully agree that "The existing pipeline cannot take it" doesn't not imply "We need a new format". It's an open question. What I want to stress out is that the industry is moving from a seasonal update of their GTFS (aka 4 times per year) to a every-day and even every-hour update (which is great!), but knowing how to address that is an unresolved question.

Do you find it advantageous to add another time-scale of patches with a new format between GTFS-static and GTFS-RT?

@abyrd I assume you haven't actually read the GTFS-ServiceChanges proposal, which is fair. The goal of ServiceChanges is to stick to the CSV GTFS format. It's working by declaring what type of change you want to do (deletion, addition, modification), then selecting a row (table name + id), then if needed providing the field names and values you want to change. So the whole goal is to not add another format, but to stick to CSV GTFS. The only exception is that you're allowed to specify a day for your changes, because otherwise editing the service_id by hand is a huge mess. So GTFS-ServiceChanges aims to be a kind of GTFS-delta if you want. Not another format.

The reason why I said extending TripUpdates "forces us to redefine almost every GTFS concepts slightly differently, therefore duplicating the spec" is that e.g. GTFS-rt StopTimeUpdate object is pretty different than CSV GTFS stop_time object. It's arrival value can be given either by a "delay" or by a "time", with time defined as an absolute POSIX time... which is completely different from the HH:MM:DD format used in CSV GTFS, which allow hours above 24 and which defines 01 as noon minus eleven hours.

Another example: the TripUpdate object doesn't contains the trip_id. It's its child, the TripDescriptor which does. So if we want to add the feature to add or alter a route, should we replicated the same pattern and define a RouteUpdate containing a RouteDescriptor containing the route_id? Or should we simplify it and define a RouteUpdate which will do both? Whichever you pick, you'll define specific object that people will have to memorize. With ServiceChange, you have nothing new to memorize. Routes are in the routes table and contains route_id, route_stop_name, etc...

====

That being said, I agree with what you guys said: in all cases, we will have the backward compatibly that we have new data to be ingested. Either because GTFS will have to updated every hours, or because TripUpdates will have changed, or because there will be a new feed.

The decision which has to be made here is an practical decision. On a theoretical point of view, there is no problem, you can just output a new CSV GTFS as often as you want. But with today implementations, it's not practically possible to consume such update. Regarding expanding TripUpdates, Stefan & Guillaume both think we won't have size issue. So the decision is really an industrial decision, of whether this industry:

• A. Is ready do change their pipeline to produce & ingest GTFS every 1h, and even every 10min if there is a e.g. huge snow storm.
• B. Says they won't ever do that change, and that they want another way to get updates on the CSV GTFS.
  • B1. Either by changing the existing TripUpdates formats, which might requires to merge data from different stream it it's hosted by different tools, but maybe not,
  • B2. Either adding another feed, with the extra cost that it will also generate.

I would be nice that everybody gives his point of view, and then we'll do whatever the industry agreed on. @slai & @dbabramov among others.

But I agree with Andrew: there is a need which requires to be addressed.