bench | locli: new DB persistence backend for analysis data #6054
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a new database persistence backend to the benchmark analysis tool
locli.Persisting log objects in a serverless SQLite DB has proven several advantages over the current file based solution (FS). Considering a perf cluster run with abundant raw log data:
Furthermore, the DB backend opens up a way to carry out custom quick queries into the analysis input - either on the application level or in SQL. A suitable abstraction for those will be a future endeavor; this PR adds a new testbed application
locli-quickfor development (which, for that reason, needs not be aworkbenchshell input as of now).Last not least, default RTS GC settings for
loclihave been revisited. This PR changes the default such that on the perf cluster deployer, a standard analysis (almost) fits into 64GiB, and as a result there's (almost) no swapping going on. This is translates into a significant speed improvement, regardless of which persistence backend is used.The PR implements all changes listed in
locli's changelog, bumping it to a new major version2.0:prepare-dbandunlog-dbto create and read from DB persistence backend respectivelyLogObjectSourceto represent input from different backends (file or DB)locli-quickwhich aims to be a development testbed for (upcoming) DB-backed quick queriesIt also integrates the new
loclifeature intoworkbenchsuch that:$WB_LOCLI_DBis introduced, defaulting to0--locli-db1results in using the DB for both preparation and analysis stepsThis means the changes of this PR are currently "opt-in". All existing functionality is retained, unmodified, and as such, this PR is non-breaking wrt. to existing workflows.
Checklist
See Runnings tests for more details
CHANGELOG.mdfor affected package.cabalfiles are updatedhlint. See.github/workflows/check-hlint.ymlto get thehlintversionstylish-haskell. See.github/workflows/stylish-haskell.ymlto get thestylish-haskellversionghc-8.10.7andghc-9.2.7