Skip to content

Commit 4a524ed

Browse files
committed
Add more links
1 parent f55ecfe commit 4a524ed

File tree

1 file changed

+16
-14
lines changed

1 file changed

+16
-14
lines changed

docs/source/docs/contributing/design-descriptions/sql-database.md

Lines changed: 16 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,16 @@ This design was a carry-over from ChameleonVision, which had used a similar dire
1010

1111
In summer YEAR????, I found a [Pi forum post](a;sldfja;sdflj) that seemed to indicate there was perhaps some funny business happening with file caching. The Pi OS at the time tried its best to avoid writing updates to the physical SD card (SD cards are not designed to run an OS off of. Don't get me started.), which means that if you don't force flush/sync, changes can be lost on power cycle as they're never written to the SD card. In response we added flushes and [FileDescriptor::syncs](https://docs.oracle.com/javase/8/docs/api/java/io/FileDescriptor.html#sync--) to the code, and called it good.
1212

13-
And yet we kept getting reports of settings being lost. In particular, a Raspberry Pi base OS update that occurred during the summer of 2022 seemed to make this report more frequent. Reports tracked in GitHub included:
14-
- 22 Febuary 2022: https://www.chiefdelphi.com/t/photon-vision-lost-config-file/403692 -- "After looking at the logs, I found that half of the config files were gone. It lost the configs for the pipelines and one camera."
15-
- 29 October, 2022: https://github.com/PhotonVision/photonvision/issues/552
16-
- User reports from CD (todo link)
17-
- User reports from Discord (todo link)
13+
So in summary:
14+
- Every time we saved to disk, we had to delete a directory recursively, then re-create its contents. We do this whenever settings are changed, as well as [immediately on startup](https://github.com/PhotonVision/photonvision/blob/f81304846251b82a128013ce0800a2cc1b71e744/photon-server/src/main/java/org/photonvision/Main.java#L348). Hopefully you don't turn off your robot at the wrong time.
15+
- Even on the Pi's NTFS filesystem, even syncing the world on every write, we were still getting reports of settings being lost
16+
- In retrospect, some of these could be attributed to poor multi-camera matching code behavior (TODO link this GH issue), which was [updated in Summer 2024](camera-matching.md)
1817

19-
As of February 2023, we were making [every attempt available to us from Java](https://github.com/PhotonVision/photonvision/blob/f81304846251b82a128013ce0800a2cc1b71e744/photon-core/src/main/java/org/photonvision/common/util/file/JacksonUtils.java#L113C1-L118C34) to close, flush, and force a sync to the physical SD card for config files we serialized:
18+
## Timeline
2019

20+
And yet we kept getting reports of settings being lost. In particular, a Raspberry Pi base OS update that occurred during the summer of 2022 seemed to make this report more frequent. For a rough timeline of events:
21+
22+
- Since [at least 26 March 2020](https://github.com/PhotonVision/photonvision/blame/b236d20fd6216ee3d5ef3b9e8511db8f27567262/chameleon-server/src/main/java/com/chameleonvision/common/util/file/JacksonUtils.java), we were making [every attempt available to us from Java](https://github.com/PhotonVision/photonvision/blob/f81304846251b82a128013ce0800a2cc1b71e744/photon-core/src/main/java/org/photonvision/common/util/file/JacksonUtils.java#L113C1-L118C34) to close, flush, and force a sync to the physical SD card for config files we serialized:
2123
```java
2224
private static void saveJsonString(String json, Path path, boolean forceSync) throws IOException {
2325
var file = path.toFile();
@@ -32,11 +34,13 @@ As of February 2023, we were making [every attempt available to us from Java](ht
3234
fileOutputStream.close();
3335
}
3436
```
35-
36-
So in summary:
37-
- Every time we saved to disk, we had to delete a directory recursively, then re-create its contents. We do this whenever settings are changed, as well as [immediately on startup](https://github.com/PhotonVision/photonvision/blob/f81304846251b82a128013ce0800a2cc1b71e744/photon-server/src/main/java/org/photonvision/Main.java#L348). Hopefully you don't turn off your robot at the wrong time.
38-
- Even on the Pi's NTFS filesystem, even syncing the world on every write, we were still getting reports of settings being lost
39-
- In retrospect, some of these could be attributed to poor multi-camera matching code behavior (TODO link this GH issue), which was [updated in Summer 2024](camera-matching.md)
37+
- 22 Febuary 2022: https://www.chiefdelphi.com/t/photon-vision-lost-config-file/403692 -- "After looking at the logs, I found that half of the config files were gone. It lost the configs for the pipelines and one camera."
38+
- 29 October, 2022: https://github.com/PhotonVision/photonvision/issues/552
39+
- 3 March 2023 -- @mdurrani834 [creates "PSA for PhotonVision Teams (Settings Loss Bug)""](https://www.chiefdelphi.com/t/psa-for-photonvision-teams-settings-loss-bug/428823)
40+
- Matt: "The failure mode shows up as Photon being unable to parse a specific camera’s config JSON file (not malformed, just straight up missing based on logs)"
41+
- 13 April 2023: https://www.chiefdelphi.com/t/photon-vision-resets-configuration-to-default/434259 -- ""
42+
- 17 June 2023: [SQLite pull request merged](https://github.com/PhotonVision/photonvision/pull/818). All further config loss reports were root-caused to camera-matching bugs.
43+
- TODO: more Discord and CD link archeology
4044

4145
## What to do?
4246

@@ -46,9 +50,7 @@ Databases like SQLite are also designed to be robust utilizing [atomic database
4650
- We threw together a [SQLite proof-of-concept](https://github.com/PhotonVision/photonvision/pull/822/files)
4751
- We also made an attempt at putting the same database into a ZIP file and [just atomically renaming it](https://github.com/PhotonVision/photonvision/pull/822/files) to replace the old config ZIP, using Java `File.move`'s [ATOMIC_MOVE](https://docs.oracle.com/javase/8/docs/api/java/nio/file/StandardCopyOption.html#ATOMIC_MOVE)
4852

49-
Both were tested, and we decided to go with SQLite. It was a rather spicy mid-season merge, but we felt that losing config data was already a game-over scenario, so anything was better than nothing.
50-
51-
The SQL config change landed [in !818 in March 2023](https://github.com/PhotonVision/photonvision/pull/818). With it, we just shove the exact same set of config files into a SQLite database instead of a
53+
Both were tested, and post-champs, we decided to go with SQLite. The SQL config change landed [in !818 in June 2023](https://github.com/PhotonVision/photonvision/pull/818). With it, we just shove the exact same set of config files into a SQLite database instead of a
5254

5355
<img style="display: block; margin: auto; width:500px" src="images/sqlite_format.png">
5456

0 commit comments

Comments
 (0)