Querying, unit testing with SQLite

[chisholm]

This PR includes fixes to the relational DB query side to make it work with SQLite. It also does some revamp to the relational DB datastore unit test suite to run tests on both postgres and sqlite. It won't work under tox/github runner until PR #625 is merged (or we decide on another way to resolve that problem). But you can still run the tests locally outside of tox.

I think there are still problems on the create/insert side for sqlite which you will need to look into. We may need to hand off to each other for awhile to get problems resolved on both sides, until all the unit tests pass.

Regarding the unit test suite, some unit tests run much more slowly now. That is because fixtures are added to get proper test setup and teardown. That is the normal pytest way to manage things, as opposed the module global variable that was there before. You get better test independence, have more options to manage scope, etc. So, the slower unit tests invoke the store fixture, which takes the place of the old global, which creates all the tables. But that happens per unit test now, as part of the setup. It operates naively, creating all tables for all STIX types, even though a given unit test doesn't need all that. That repeated table creating/dropping is slow. It would be better to only create the tables you need, but the fixture doesn't know which ones are needed. But that's how it works for now. The test_property* unit tests only create the tables they need, so they run much faster.

Because the datastore is no longer a module global variable, that error associated with creating the store no longer occurs on module import. That means that unit test collection can occur normally, which actually improves the ability of the unit tests to run in github/tox. So you can see a normal test suite run, but when it comes to the relational datastore test suite, there are lots of "E"s, indicating errors. It is the same multiply-defined-table error, but occurs repeatedly per-test. It also means that the relational datastore unit tests which are not affected by it, can run normally, so you see a lot of successes too.

As a side effect of the above changes, that unexpected DB connection drop problem seems to have gone away. I can get the entire postgres test suite to pass successfully now.

Running all tests for all DBs every time is probably way more than you normally want to do as you are developing. If you want to run just the sqlite tests for example, you can edit the db_backend fixture. There is a params argument, and you can just delete the "postgresql" item if you want. I don't know of an easy commandline way to select one vs the other. It may be possible with more sophisticated pytest config tricks. But for now, you can do simple edits to source code.