Experimenting with VLMs for Auto-validations and AI-assisted Validations

[jonfroehlich]

In the past ~decade+, we've experimented with all sorts of CV approaches to classifying sidewalk streetscapes images (for auto-labeling, auto-validation, etc.). In the past ~2 years, our lab has started to heavily experiment with VLMs and multimodal LLMs (StreetReaderAI, BikeButler, etc.). They seem to be performing extraordinarily well.

Even though we only recently shipped AI-assisted validation (thanks @jsomeara), it may already be time to investigate a new solution: VLMs.

The following example about whether a curb ramp or tactile surface indicator is required here is incredibly nuanced but comes up from time-to-time in Project Sidewalk. The first screenshot shows my take. I then asked Gemini for its take (although perhaps it was "biased" by seeing my response; I should try it again without biasing it)

Gemini Interaction 1

Prompt

Is a tactile surface indicator required here (where the red dot is) at the sidewalk-to-alleyway transition

Gemini Pro 3.1 Response

Here's Gemini's response: https://gemini.google.com/share/3c3be7eeab7d

Based on standard accessibility guidelines (like PROWAG in the US), a tactile surface indicator (often called detectable warning surfaces or truncated domes) is generally not required—and often explicitly discouraged—at this specific type of transition.

Here is the breakdown of why the selection in your validation interface ("I don't think a curb ramp is required here") is the correct approach for this label:

The Role of Tactile Indicators

Tactile warning surfaces are designed to warn pedestrians with vision disabilities that they are leaving a safe pedestrian zone and entering a hazardous vehicular roadway (typically a street crossing where the pedestrian must stop and assess traffic).

Alleyways and Driveways vs. Street Intersections

• Continuous Pedestrian Route: The transition in the image appears to be a residential driveway or minor, uncontrolled alleyway. In these scenarios, the sidewalk effectively continues across the vehicular path.
• Right-of-Way: At these minor crossings, the pedestrian retains the right-of-way, and vehicles entering or exiting the alley are expected to yield to sidewalk traffic.
• Risk of Confusion: Placing tactile domes at a flush driveway or minor alleyway crossing can provide misleading sensory feedback to a white cane user. It falsely signals a standard street intersection, potentially causing them to stop unnecessarily in the middle of their path while waiting for non-existent traffic controls.

The Exception

The only time a tactile indicator would typically be required at an alleyway is if the alley functions as a full street intersection—for example, if it has a stop sign or traffic signal controlling the pedestrian crossing, or if the pedestrian route is fully broken by a standard street-level curb drop rather than a continuous concrete apron.

For the location marked by the pink dot, treating it as a continuous pathway without a curb ramp or tactile indicator is the standard accessible design choice.

Gemini Interaction 2

Gemini's Second Response

I tried again but with a diff input image (where it doesn't see my answer). Used same text prompt. Here's the image.

Gemini Pro 3.1 Response

The Short Answer

No, a tactile surface indicator (also known as a detectable warning surface or truncated domes) is not required at this specific location.

Why They Aren't Required Here

Based on the guidelines set by the Americans with Disabilities Act (ADA) and the Public Right-of-Way Accessibility Guidelines (PROWAG), here is why they are omitted at this type of transition:

• Continuous Pedestrian Route: In residential alleyways or driveways like the one pictured, the pedestrian generally maintains the right-of-way. The sidewalk is considered a continuous pedestrian access route, even though it crosses a path used by vehicles.
• Lack of Traffic Controls: Detectable warnings are meant to warn pedestrians that they are entering a hazardous vehicular area. They are required at curb ramps and blended transitions at standard street crossings. For alleys and driveways, they are generally only required if the crossing is governed by a traffic control device (like a stop sign or a traffic light). This alley has no such controls.
• Preventing False Cues: Installing tactile indicators at an unsignalized alley could actually be hazardous. For visually impaired pedestrians, truncated domes provide a specific tactile cue indicating they are at a street intersection and need to stop, listen, and assess traffic before crossing. Placing them here would give a false signal, potentially causing the pedestrian to stop unnecessarily or mistakenly believe they are at a full street crossing rather than a continuous sidewalk.

To maintain compliance and safety, the sidewalk should remain a smooth, continuous path across the alleyway apron.

Gemini Interaction 3

I then tried to send it only an image and describe access issue. Unfortunately, the image does include a label artifact (when you click "hide label", there is an outline still shown).

Prompt: Does this sidewalk have any access issues for people with disabilities (according to modern PROWAG standards)?

Interestingly, here, the response is a bit contradictory with the above--perhaps due to the aforementioned label artifact or perhaps due to the analysis being far harder (it's an open-ended question vs. a directed question, no salient label locator, etc.)

Gemini Pro 3.1 Response

Based on the visual evidence in the image and modern PROWAG (Public Right-of-Way Accessibility Guidelines) standards, this intersection of the sidewalk and the alleyway presents several significant access issues.

Key Accessibility Violations

• Lack of Detectable Warnings (Truncated Domes):
  PROWAG requires detectable warning surfaces (the yellow or contrasting bumpy tiles) at any point where a pedestrian path crosses a vehicular way. This alley entrance lacks them entirely, which is a major safety hazard for pedestrians with vision impairments who rely on tactile cues to know they are entering a "danger zone" with vehicle traffic.
• Abrupt Changes in Level (Vertical Discontinuities):
  There is a visible vertical lip where the sidewalk concrete meets the asphalt of the road/alley. PROWAG specifies that vertical changes in level must be no greater than 1/4 inch (6.4 mm). Changes between 1/4 and 1/2 inch must be beveled. The gap here appears to exceed these limits, creating a tripping hazard and a barrier for wheelchair users.
• Cross Slope Issues:
  The "flare" or apron of the alley (the triangular concrete sections) appears to cut into the sidewalk path. PROWAG requires a Pedestrian Access Route (PAR) with a maximum cross slope of 2% ($1:50$). The transition from the flat sidewalk into the slanted alley apron likely exceeds this, which can cause manual wheelchairs to veer toward the street or tip.
• Surface Condition and Stability:
  The joint where the sidewalk meets the alley is cracked and deteriorating. PROWAG requires surfaces to be "firm, stable, and slip-resistant." The loose gravel and uneven pavement at this transition point fail this requirement.