0.3.0

Breaking Change

This release changes how directories are handled. Previously, wherever you ran the doltgres binary was where your databases would be located. This was causing a lot of confusion, as attempting to run the doltgres binary from within the doltgres folder (on non-Windows machines) would result in a name conflict, where the program wanted to create a folder named doltgres, but that conflicted with the binary's name.

This has been changed, so that we now use the ~/doltgres/databases folder by default. This means that it is now irrelevant where you run doltgres from, as it will always access that folder. If you want to use a different folder then you can use the --data-dir argument. If a relative path is given, then that is relative to the invocation directory. If an absolute path is given, then that path is used. In addition, there's now a DOLTGRES_DATA_DIR environment variable, which accomplishes the same thing as the --data-dir argument, except that you don't have to include the argument every time you run the binary. To mimic the previous behavior, you can set the argument or environment variable to ., which will use the invocation directory.

Merged PRs

doltgresql

• 79: Added static analyzer & linter
  This adds a static analyzer and linter. This is just to help enforce a bit more code quality.
• 78: Changed default database creation behavior
  Previously, when running doltgres, it would look at the current directory to determine if it was a valid DoltgreSQL directory. If it wasn't, then it would attempt to create a doltgres database within the directory. This behavior was chosen since "it just works", which is a selling point of the product. This was causing issues though, as users would run the tool from within the same directory that they downloaded it in. This caused an error message to appear stating that the file was conflicting with the attempt to create the database, however users were still confused by the behavior even with the error message.
  This default behavior has been changed. We now use the ~/doltgres/databases directory as the default directory. It no longer matters where you run doltgres from, it will always point to that directory. This directory can be changed by changing the environment variable DOLTGRES_DATA_DIR to a different directory.
  In addition, there's another environment variable named DOLTGRES_DATA_DIR_CWD. When this is set to true, then it causes DoltgreSQL to operate with the previous behavior, in that it will use the current directory rather than the global data directory.
• 77: End to end tests of all postgres types (many skipped)
  Behavior and syntax verified against postgres, but many of them may require custom types instead of strings when implemented.
• 74: Test generation framework additions
  I ran into too many issues while trying to generate a set of SELECT tests to add to sqllogictests, but I think this code here could still be useful in the future. The SELECT statements were too complex for random generation to happen upon a valid combination of queries, aliases, values, etc., but this could still have use for simpler statements. Since our synopses are written in a small DSL, you could technically generate anything as long as random generation is viable. It's basically what I've done for a lot of the generated tests throughout GMS and Dolt, but more formalized here.
• 73: Correct issue test
  #25 was incorrect in its assumptions, but still proved valuable for learning. As a result, I've modified the skipped test so that we're now checking the same relative thing.
• 72: Fix release notes generator
• 71: Release v0.2.0
  Created by the Release workflow to update DoltgreSQL's version
• 66: Contribution Guide
  Contribution guide. Primarily intended for Dolt devs transitioning to DoltgreSQL, but also written for anyone who wants to work on the repository. Now I can link to the document for some feedback, rather than leaving a paragraph on why it's important to change something lol.
• 65: Add usage metrics to doltgres
  Usage metrics are now reported to the dolthub metrics servers for the sql-server command, tagged with the doltgres application ID.
  Also refactors application and server start to put top-level concerns in the main method.

Closed Issues

• 43: No databases on macOS arm64?
• 44: Document backup and replication needs/compability
• 25: call dolt_branch("br1") errors

0.2.0

Merged PRs

doltgresql

• 67: Potentially fix release workflow
  GH Actions cannot bypass branch rules, so this is an attempt to have them play by the same rules.
• 63: Fixes for issues: 40, 42, 43
  This contains fixes for:
  • #40
  • #42
  • #43
    In particular, the Getting Started example was a good test, as it actually uncovered quite a few issues separate from the example. These have all been fixed. I know you're not going to like how we're handling TableFuncExprs (we weren't before), but it's way easier than going to the GMS side and adding functionality for now. With some of these bugs fixed, I'd actually expect a decent bump in the sqllogictests TBH.
• 62: Added parser tests
  This PR adds a lot of parser tests (all validated against Postgres 15), in addition to fixing a ton of issues with the synopses. Never again will I rely on the Postgres documentation being correct, in some ways it's even worse than MySQL's documentation. They should have just posted the grammar, because it looks handwritten and it's actually wrong in a lot of places. Rant aside, there's actually a bit of non-test stuff here, but it's all to support the test generation, so this probably doesn't need more than a glance over.
  Some fun statistics:

  Test	Percentage	Number	Description
  Total	100%	332547	The total number of tests that were added
  Unimplemented	96.86%	322092	The number of tests that do not parse at all
  Parses	2.09%	6936	The number of tests that parse, but cannot be converted to a Vitess AST
  Converts To AST	1.05%	3524	The number of tests that can successfully create a Vitess AST (accuracy not tested)

• 61: Added the ability to get an exact permutation from parser tests
  This allows you to pass in a number and get that exact permutation. So that means that, even though SELECT has 144 septillion permutations, if we give it 25,757,684,286,468,754, then it will give us exactly that permutation. So to generate a set of tests, we just generate some amount of random numbers between [0, permutations), and we have our tests!
• 60: Implemented support for various expressions needed by sqllogictest
• 59: Added permutation counting, variable definitions to parser test generation
  This adds the ability to count the number of permutations without having to actually loop through them all and increment one by one. This also adds variable declaration support (the where VAR is one of: in the synopsis). With those two, I can now verify that with some of the variables defined in SELECT, we're up to 144 septillion variations lol. I also removed the repetition strategy, which cut out a ton of tests that probably weren't necessary (exponential on top of combinatorial).
  This is a prelude to the next PR. I've found a strategy to be able to grab a deterministic set of tests (we can randomly decide the set, but given the proper input it's deterministic), but it requires knowing the number of possible permutations ahead of time, hence this PR.
• 58: Parser Tests Tool
  This adds a tool that can download synopses from the Postgres documentation page, save those synopses under the testing/generation/command_docs/synopses folder, iterate over each synopsis to create a StatementGenerator that can create all permutations of a query based on the synopsis, check that the queries are valid against a Postgres installation, and then create a test file verifying the level of implementation for the query in DoltgreSQL (whether the query parses or can be fully transformed). The idea here is that we run this once (or whenever we upgrade to a newer Postgres version, like version 16), and then as we continue developing, we'll naturally add support for more statements. This tracks regressions too, so if something used to successfully parse and now it doesn't, then we can see that we've broken something. At least, that's the idea.
  The major issue is that the permutation count is massive. The tool even supports variables as defined by each synopsis, and without adding those, I got to over 50 billion permutations for just SELECT before I stopped it. We need to find a way to get some reasonable subset for the massive statements, so we can discuss that later.
  Regarding this PR though, the tool is mostly complete. There are a few "issues"
  • Doesn't currently check with an actual Postgres installation yet. This is easy to add, but I didn't bother since we need to find a way to get the total number of tests down first.
  • Doesn't actually do anything with variables yet. Permutation counts exhibit exponential behavior, so I wouldn't be surprised if the final numbers are in the quintillions.
  • Downloaded synopses require manual fiddling. This is mostly due to the Postgres documentation being inconsistent with whitespace, and writing a tool to deal with these inconsistencies was too much. Not all files require fiddling (variable substitution is handled by the tool), and it's not a lot of fiddling either, but it's there.
    Besides those things, this is fully complete. The single example file in testing/generation/command_docs/output/abort_test.go works from start to finish (downloading synopsis, generating queries, creating test file), so you can see what it'll look like once we figure out how to get a subset.
    Regarding how to review this, I'd mainly say to just like at that test file. You can also check out some of the synopses to see how some of them differ from what's on the website. If you really want to, you can look at the parser and generator, but it's not required.
• 57: Better type handling in harness, support COUNT(*) expressions and CAST
• 56: Null literal handling
  Also adds unit tests for error handling during extended query execution, and some debug logging for messages received.
• 55: Fixed INT8 handling, support for testing statements, proper connection error handling
  This primarily implements four things:
  1. We now discard all messages until we reach a Sync message when an error occurs, which is consistent with the documentation.
  2. Proper support for boolean results. The sqllogictests make use of them, and GMS doesn't have a native boolean type, so we treat all booleans as Int32 types. You can read more in the comments.
  3. Proper error handling is the multi-message test case. Our conn.Query calls in tests only return specific kinds of errors, and all other errors are put into the returned rows. This differs from conn.Exec, which returns all errors. Previously, we never checked for an error in rows (since I never knew an error would be put there), so it was getting swallowed. We now properly handle those errors.
  4. Expanded statement support for Describe messages. I noticed in the zachmu/err-handling-repro branch that the test file testing/go/prepared_statement_test.go is inconsistent with the general test framework, as it runs the setup statements using conn.Query rather than conn.Execute. We don't support Describe at all right now, as GMS currently requires analyzing a statement to determine the output schema, and analyzing some statements produces execution side effects. As a workaround, we start a transaction, execute the query, record the schema, and rollback the transaction. This does not work for any commands that implicitly commit the transaction, so this adds another hack to fake it by running an equivalent DROP to select CREATE statements. This does cause an implicit commit, but it's better than failing for now.
• 53: Support arbitrary message size from clients
  This change implements support for arbitrary messages sizes from clients. Previously we were limited to 2k messages.
  Now that we are reading a message of known size, there's no need for Receive() to return more than a single message: we always fetch one at a time from the connection.
  Also refactors the main listening loop into manageable chunks.
• 52: Added Format CI and Release CD
  The formatter does not auto-format and push to the repo, it just fails for now. The release scripts are untested, but they seem like they should work. They're built off of Dolt's scripts, but since there are quite a few differences, I may have missed something that's required. I'm fairly confident that it's all included and supposed to work though. We'll know for sure when we attempt to use it.
• 51: Warn on non-ascending index
  This is to bootstrap compatibility until we can implement descending order indexes. Since GMS assumes all indexes are ordered ascending, this will not impact correctness, just produce slightly different schemas.
• 50: sqllogictest runner
  First attempt at sqllogictest runner
  Also fixes several small bugs related to prepared statement caching, types, and tuples, and updates to the latest gms / vite...

0.1.0

This is the first release of DoltgreSQL! This is a very early release that should not be used for any serious applications, however it will allow you to preview the future capabilities of the product!