You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/source/docs/contributing/design-descriptions/sql-database.md
+16-14Lines changed: 16 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,14 +10,16 @@ This design was a carry-over from ChameleonVision, which had used a similar dire
10
10
11
11
In summer YEAR????, I found a [Pi forum post](a;sldfja;sdflj) that seemed to indicate there was perhaps some funny business happening with file caching. The Pi OS at the time tried its best to avoid writing updates to the physical SD card (SD cards are not designed to run an OS off of. Don't get me started.), which means that if you don't force flush/sync, changes can be lost on power cycle as they're never written to the SD card. In response we added flushes and [FileDescriptor::syncs](https://docs.oracle.com/javase/8/docs/api/java/io/FileDescriptor.html#sync--) to the code, and called it good.
12
12
13
-
And yet we kept getting reports of settings being lost. In particular, a Raspberry Pi base OS update that occurred during the summer of 2022 seemed to make this report more frequent. Reports tracked in GitHub included:
14
-
- 22 Febuary 2022: https://www.chiefdelphi.com/t/photon-vision-lost-config-file/403692 -- "After looking at the logs, I found that half of the config files were gone. It lost the configs for the pipelines and one camera."
- Every time we saved to disk, we had to delete a directory recursively, then re-create its contents. We do this whenever settings are changed, as well as [immediately on startup](https://github.com/PhotonVision/photonvision/blob/f81304846251b82a128013ce0800a2cc1b71e744/photon-server/src/main/java/org/photonvision/Main.java#L348). Hopefully you don't turn off your robot at the wrong time.
15
+
- Even on the Pi's NTFS filesystem, even syncing the world on every write, we were still getting reports of settings being lost
16
+
- In retrospect, some of these could be attributed to poor multi-camera matching code behavior (TODO link this GH issue), which was [updated in Summer 2024](camera-matching.md)
18
17
19
-
As of February 2023, we were making [every attempt available to us from Java](https://github.com/PhotonVision/photonvision/blob/f81304846251b82a128013ce0800a2cc1b71e744/photon-core/src/main/java/org/photonvision/common/util/file/JacksonUtils.java#L113C1-L118C34) to close, flush, and force a sync to the physical SD card for config files we serialized:
18
+
## Timeline
20
19
20
+
And yet we kept getting reports of settings being lost. In particular, a Raspberry Pi base OS update that occurred during the summer of 2022 seemed to make this report more frequent. For a rough timeline of events:
21
+
22
+
- Since [at least 26 March 2020](https://github.com/PhotonVision/photonvision/blame/b236d20fd6216ee3d5ef3b9e8511db8f27567262/chameleon-server/src/main/java/com/chameleonvision/common/util/file/JacksonUtils.java), we were making [every attempt available to us from Java](https://github.com/PhotonVision/photonvision/blob/f81304846251b82a128013ce0800a2cc1b71e744/photon-core/src/main/java/org/photonvision/common/util/file/JacksonUtils.java#L113C1-L118C34) to close, flush, and force a sync to the physical SD card for config files we serialized:
@@ -32,11 +34,13 @@ As of February 2023, we were making [every attempt available to us from Java](ht
32
34
fileOutputStream.close();
33
35
}
34
36
```
35
-
36
-
So in summary:
37
-
- Every time we saved to disk, we had to delete a directory recursively, then re-create its contents. We do this whenever settings are changed, as well as [immediately on startup](https://github.com/PhotonVision/photonvision/blob/f81304846251b82a128013ce0800a2cc1b71e744/photon-server/src/main/java/org/photonvision/Main.java#L348). Hopefully you don't turn off your robot at the wrong time.
38
-
- Even on the Pi's NTFS filesystem, even syncing the world on every write, we were still getting reports of settings being lost
39
-
- In retrospect, some of these could be attributed to poor multi-camera matching code behavior (TODO link this GH issue), which was [updated in Summer 2024](camera-matching.md)
37
+
- 22 Febuary 2022: https://www.chiefdelphi.com/t/photon-vision-lost-config-file/403692 -- "After looking at the logs, I found that half of the config files were gone. It lost the configs for the pipelines and one camera."
- 3 March 2023 -- @mdurrani834[creates "PSA for PhotonVision Teams (Settings Loss Bug)""](https://www.chiefdelphi.com/t/psa-for-photonvision-teams-settings-loss-bug/428823)
40
+
- Matt: "The failure mode shows up as Photon being unable to parse a specific camera’s config JSON file (not malformed, just straight up missing based on logs)"
41
+
- 13 April 2023: https://www.chiefdelphi.com/t/photon-vision-resets-configuration-to-default/434259 -- ""
42
+
- 17 June 2023: [SQLite pull request merged](https://github.com/PhotonVision/photonvision/pull/818). All further config loss reports were root-caused to camera-matching bugs.
43
+
- TODO: more Discord and CD link archeology
40
44
41
45
## What to do?
42
46
@@ -46,9 +50,7 @@ Databases like SQLite are also designed to be robust utilizing [atomic database
46
50
- We threw together a [SQLite proof-of-concept](https://github.com/PhotonVision/photonvision/pull/822/files)
47
51
- We also made an attempt at putting the same database into a ZIP file and [just atomically renaming it](https://github.com/PhotonVision/photonvision/pull/822/files) to replace the old config ZIP, using Java `File.move`'s [ATOMIC_MOVE](https://docs.oracle.com/javase/8/docs/api/java/nio/file/StandardCopyOption.html#ATOMIC_MOVE)
48
52
49
-
Both were tested, and we decided to go with SQLite. It was a rather spicy mid-season merge, but we felt that losing config data was already a game-over scenario, so anything was better than nothing.
50
-
51
-
The SQL config change landed [in !818 in March 2023](https://github.com/PhotonVision/photonvision/pull/818). With it, we just shove the exact same set of config files into a SQLite database instead of a
53
+
Both were tested, and post-champs, we decided to go with SQLite. The SQL config change landed [in !818 in June 2023](https://github.com/PhotonVision/photonvision/pull/818). With it, we just shove the exact same set of config files into a SQLite database instead of a
0 commit comments