Skip to content

Conversation

@elkoled
Copy link
Contributor

@elkoled elkoled commented Jan 7, 2026

Car diff - CarState regression testing

This PR implements a diff method to catch behavioral changes in CarState by replaying a subset of carCommaSegments through CarInterface and comparing their CarState output against master refs in ci-artifacts.

Example:
elkoled#245

CI

  • PR: Detects changed platforms, downloads master refs as zst from ci-artifacts, replays segments from commaCarSegments, reports any behavior differences as bot comment
  • master: Generates and uploads new references for changed platforms

Usage

auto-detect platform
source setup.sh && python opendbc/car/tests/car_diff.py

test specific platform
source setup.sh && python opendbc/car/tests/car_diff.py --platform TOYOTA_COROLLA

CI Benchmark

Measured on standard gh actions runner

Brand Platforms Segs CI time
hyundai 56 (Hyundai/Kia/Genesis) 260 8:41
toyota 36 (Toyota/Lexus) 190 7:09
honda 21 (Honda/Acura) 154 5:30
volkswagen 21 (VW/Audi/Seat/Skoda) 130 4:41
gm 12 (Chevy/Buick/Cadillac/GMC) 97 4:11
ford 10 91 4:00
chrysler 9 (Chrysler/Jeep/Dodge/Ram) 90 2:32
subaru 9 69 3:56
nissan 5 50 3:16
tesla 2 20 2:16
mazda 3 17 2:12

@github-actions github-actions bot added CI / testing car related to opendbc/car/ tesla car safety vehicle-specific safety code labels Jan 7, 2026
@elkoled elkoled changed the title Opendbc replay car replay Jan 8, 2026
@elkoled elkoled changed the title car replay Car replay Jan 8, 2026
* add replay POC

* compare HEAD to master

* split in compare and worker

* copy worker.py

* cleanup

* fix

* apply tesla brake PR

* fix

* fix mutation test

* add panda

* add timestamps

* simplify

* add falling edge

* add remaining carstate

* simple lines

* bump

* rename

* use zst as ref

* ref commit

* cleanup

* formatting

* no failing

* fix git

* clean

* use pool.map

* fix docker build

* clean

* Revert "fix docker build"

This reverts commit da35774.

* use flat paths

* format

* use azure test instance

* print full diff

* no stderr output

* use GIT_REF

* Revert "apply tesla brake PR"

This reverts commit f377e21.

* allow overwrite blobs

* supress stderr
@github-actions github-actions bot removed the car safety vehicle-specific safety code label Jan 9, 2026
elkoled added 18 commits January 9, 2026 14:39
* add replay POC

* compare HEAD to master

* split in compare and worker

* copy worker.py

* cleanup

* fix

* apply tesla brake PR

* fix

* fix mutation test

* add panda

* add timestamps

* simplify

* add falling edge

* add remaining carstate

* simple lines

* bump

* rename

* use zst as ref

* ref commit

* cleanup

* formatting

* no failing

* fix git

* clean

* use pool.map

* fix docker build

* clean

* Revert "fix docker build"

This reverts commit da35774.

* use flat paths

* format

* use azure test instance

* print full diff

* no stderr output

* use GIT_REF

* Revert "apply tesla brake PR"

This reverts commit f377e21.

* allow overwrite blobs

* supress stderr

* print upload refs

* add error on failed generation

* Revert "use azure test instance"

This reverts commit ba969cc.

* Revert "fix mutation test"

This reverts commit 8f5e01d.

* detect safety changes

* use logReader

* only use valid platforms

* use local database

* Revert "use local database"

This reverts commit 4ebe9fe.

* Reapply "use azure test instance"

This reverts commit e9968a0.
@github-actions github-actions bot added subaru tesla toyota car safety vehicle-specific safety code labels Jan 17, 2026
@github-actions github-actions bot added honda and removed tesla car safety vehicle-specific safety code labels Jan 17, 2026
@github-actions

This comment was marked as outdated.

@github-actions github-actions bot removed the honda label Jan 17, 2026
Copy link
Contributor

@adeebshihadeh adeebshihadeh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, aside from those minor comments. Feel free to merge once those are addressed.

@elkoled elkoled merged commit 80f030a into master Jan 19, 2026
11 checks passed
@elkoled elkoled deleted the opendbc_replay branch January 19, 2026 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

car related to opendbc/car/ CI / testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants