ASV Benchmarks: Real storage tests

General

Having a real storage test would require a single test to be able to be executed against any supported by ArcticDB storage - Amazon S3, LMDB, GCP etc, without any change of test logic but only changing the parameters - like number of rows per dataframe etc. That requires creation of small abstraction over Arctic class that will allow us to take advantages of persistancy of storages like S3, and also emulate that with LMDB. Such abstraction is TestLibraryManager class.

TestLibraryManager Class

The class main purpose is to provide ASV test developer with a way to manage lifecycle of persistent/or shared/ libraries and modifiable/or private/ libraries that only the particular test can access on any of the supported storages.

• A modifiable /or private/ library is such that is isolated and reachable only for the instance of the test that is currently working on one's machine. Any instances of the same tests on other machines will create their onw modifiable/or private/ libraries. This way each instance of the runs in isolation. The main important rule here is that those libraries should be cleaned either before the test is executed as part of the setup or as part of the cleanup. As recomendation it would be make sure to clean all libraries that could remain from previous execution in the setup of the test, as this makes sure the test runs in clean/controlled environment in any test execution
• A persistent/or shared/ library is such that can be created once and read many times by a particular ASV test. Its benefit is to have creation of the needed for test infrastructure of library and symbols done once and then reused many times. Additional benefit of such storage is the ability to simulate aging over time - if each run of test only adds data to libraries symbols without destroing them that simulates the typical aging process which is natural for all actual usage scenarios.

Another important characteristic of this class is that it provides a way to resuse that logic outside of ASV in other type of tests or tests of the class functionality itsilf. In other way it is not coupled at all with ASV.

The class provides minimal set of needed methods to achieve any user scenario for any of those storages. It does not implement all methods that Arctic class does have thus when needed can be exteneded. Here are practical examples of this class usage.