Cache global metadata in separate meta.sqlite3 database#80
Cache global metadata in separate meta.sqlite3 database#80c-cube merged 2 commits intosneeuwballen:masterfrom
Conversation
This makes it faster to access the prover metadata for all runs in the `compare2` view because we don't have to open a bajillion different databases (goes from several minutes to a few seconds for me when restarting the server). Note that the database cache is not currently garbage collected and might need to be manually cleaned periodically.
c-cube
left a comment
c-cube
left a comment
There was a problem hiding this comment.
Very useful! I'm not super fond of the key/value table, but it's flexible. I'm open to merging after we discuss my few questions :)
src/server/Meta_cache.ml
Outdated
| CREATE TABLE IF NOT EXISTS test_database ( | ||
| id INTEGER PRIMARY KEY, | ||
| path BLOB NOT NULL UNIQUE | ||
| ); |
There was a problem hiding this comment.
I think all these tables could be strict now :)
There was a problem hiding this comment.
Interesting, did not know sqlite supported strict typing now!
src/server/Meta_cache.ml
Outdated
| key TEXT NOT NULL, | ||
| value BLOB, | ||
| CONSTRAINT meta_key_unique UNIQUE (database_id, key) | ||
| ); |
There was a problem hiding this comment.
I think you're going to want to build an index on (database_id, key) for faster access.
There was a problem hiding this comment.
also: it could be easier, maybe, to use a JSON field (sqlite supports json). I'm not sure.
There was a problem hiding this comment.
I believe that the UNIQUE constraint creates such an index already.
https://www.sqlite.org/lang_createtable.html#unique_constraints
Also did not know sqlite had json support, all this new stuff! It might have been easier to use json from the get-go but now that the code is written I am a bit reluctant to rewrite it if it's not necessary. It's also consistent with the way data is stored elsewhere (which I think pre-dates json support).
There was a problem hiding this comment.
By "elsewhere" I mean: in Test_metadata.
There was a problem hiding this comment.
that's fair! It's just good to know for next time :). The JSON support is very close to PG's afaik, and it's useful for flexible columns. But no need to rewrite your code.
There was a problem hiding this comment.
Actually I was more concerned with the OCaml side of things than the SQLite side of things re: json, but now that I think about it there's already a json printer in the Misc module and it's probably just strings at the Sqlite3 level so I can actually change this quickly I think, let me have a go.
There was a problem hiding this comment.
Actually there's no json parsing so forget it.
2 participants
This makes it faster to access the prover metadata for all runs in the
compare2view because we don't have to open a bajillion different databases (goes from several minutes to a few seconds for me when restarting the server).Note that the database cache is not currently garbage collected and might need to be manually cleaned periodically.