diff --git a/README.md b/README.md index 476241d..82d23dc 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,8 @@ Store encrypted data alongside your existing data: - [How is data encrypted in the database?](#how-is-data-encrypted-in-the-database) - [Helper packages](#helper-packages-and-examples) - [Releasing](#releasing) +- [Developing](#developing) +- [Testing](#testing) --- @@ -393,20 +395,14 @@ To cut a [release](https://github.com/cipherstash/encrypt-query-language/release This will trigger the [Release EQL](https://github.com/cipherstash/encrypt-query-language/actions/workflows/release-eql.yml) workflow, which will build and attach artifacts to [the release](https://github.com/cipherstash/encrypt-query-language/releases/). -## Testing - -There are tests for EQL for PostgreSQL versions 14–17. - -They easiest way to run them is in [GitHub Actions](https://github.com/cipherstash/encrypt-query-language/actions/workflows/test-eql.yml). - -### Running tests locally +## Developing > [!IMPORTANT] -> **Before you run the tests** you need to have this software installed: +> **Before you follow the quickstart* you need to have this software installed: > - [mise](https://mise.jdx.dev/) — see the [installing mise](#installing-mise) instructions > - [Docker](https://www.docker.com/) — see Docker's [documentation for installing](https://docs.docker.com/get-started/get-docker/) -To run tests locally: +Local development quickstart: ``` shell # Clone the repo @@ -416,17 +412,190 @@ cd encrypt-query-language # Install dependencies mise trust --yes -# Start a postgres instance -mise run postgres:up postgres-17 --extra-args "--detach --wait" +# Build EQL installer and uninstaller, outputting to release/ +mise run build + +# Start a postgres instance (defaults to PostgreSQL 17) +mise run postgres:up --extra-args "--detach --wait" + +# Run the tests (defaults to PostgreSQL 17) +mise run test + +# Stop and remove all containers and networks +mise run postgres:down +``` + +### Installing mise + +> [!IMPORTANT] +> You must complete this step to set up a local development environment. + +Local development and task running in CI is managed through [mise](https://mise.jdx.dev/). + +To install mise: + +- If you're on macOS, run `brew install mise` +- If you're on another platform, check out the mise [installation methods documentation](https://mise.jdx.dev/installing-mise.html#installation-methods) + +Then add mise to your shell: + +```shell +# If you're running Bash +echo 'eval "$(mise activate bash)"' >> ~/.bashrc + +# If you're running Zsh +echo 'eval "$(mise activate zsh)"' >> ~/.zshrc +``` + +We use [`cargo-binstall`](https://github.com/cargo-bins/cargo-binstall) for faster installation of tools installed via `mise` and Cargo. +We install `cargo-binstall` via `mise` when installing development and testing dependencies. + +> [!TIP] +> We provide abbreviations for most of the commands that follow. +> For example, `mise run postgres:setup` can be abbreviated to `mise r s`. +> Run `mise tasks --extended` to see the task shortcuts. + +### How this project is organised + +Development is managed through [mise](https://mise.jdx.dev/), both locally and [in CI](https://github.com/cipherstash/encrypt-query-language/actions). + +mise has tasks for: + +- Building EQL install and uninstall scripts (`build`) +- Starting and stopping PostgreSQL containers (`postgres:up`, `postgres:down`) +- Running unit and integration tests (`test`, `reset`) -# Run the tests -mise run test --postgres 17 +These are the important files in the repo: + +``` +. +├── mise.toml <-- the main config file for mise +├── tasks/ <-- mise tasks +├── sql/ <-- The individual SQL components that make up EQL +├── docs/ <-- Tutorial, reference, and concept documentation +├── tests/ <-- Unit and integration tests +│ ├── docker-compose.yml <-- Docker configuration for running PostgreSQL instances +│ └── *.sql <-- Individual unit and integration tests +├── release/ <-- Build artifacts produced by the `build` task +├── examples/ <-- Example uses of EQL in different languages +└── playground/ <-- Playground enviroment for experimenting with EQL and CipherStash Proxy +``` + +## Testing + +There are tests for checking EQL against PostgreSQL versions 14–17, that verify: + +- Adding, removing, and modifying encrypted data and indexes +- Validating, applying, and removing configuration for encrypted data and encrypted indexes +- Validating schemas for EQL configuration, encrypted data, and encrypted indexes +- Using PostgreSQL operators on encrypted data and indexes (`=`, `<>`, `@>`) + +The easiest way to run the tests [is in GitHub Actions](./.github/workflows/test-eql.yml): + +- Automatically whenever there are changes in the `sql/`, `tests/`, or `tasks/` directories +- By manually running [the workflow](https://github.com/cipherstash/encrypt-query-language/actions/workflows/test-eql.yml) + +This is how the `test-eql.yml` workflow functions: + +```mermaid +--- +title: Testing EQL +--- +stateDiagram-v2 + direction LR + classDef code font-family:monospace; + + + state "🧍 Human makes changes to EQL sources" as changes + state sources_fork <> + state sources_join <> + state "sql/*.sql" as source_sql + state "tasks/**/*" as source_tasks + state "tests/**/*" as source_tests + state sources_changed <> + + state "🛠️ Trigger GitHub Actions workflow test-eql.yml" as build_triggered + state "Matrix: Test EQL SQL components" as matrix + state "Test with Postgres 14" as pg14 + state "Test with Postgres 15" as pg15 + state "Test with Postgres 16" as pg16 + state "Test with Postgres 17" as pg17 + state "Check build results" as check + state if_state <> + + changes --> sources_fork + sources_fork --> source_sql:::code + sources_fork --> source_tests:::code + sources_fork --> source_tasks:::code + source_sql --> sources_join + source_tests --> sources_join + source_tasks --> sources_join + sources_join --> source_changed_check + source_changed_check --> sources_changed + sources_changed --> build_triggered : Some changes + sources_changed --> [*]: No changes + + state "Check source changes" as source_changed_check + + [*] --> changes + + build_triggered --> matrix + + state fork_state <> + matrix --> fork_state + fork_state --> pg14 + fork_state --> pg15 + fork_state --> pg16 + fork_state --> pg17 + + state join_state <> + pg14 --> join_state + pg15 --> join_state + pg16 --> join_state + pg17 --> join_state + + state "✅ Pass build" as build_pass + state "❌ Fail build" as build_fail + join_state --> check + check --> if_state + if_state --> build_pass: All success + if_state --> build_fail : Any failures + build_pass --> [*] + build_fail --> [*] +``` + +You can also [run the tests locally](#running-tests-locally) when doing local development. + +### Running tests locally + +> [!IMPORTANT] +> **Before you run the tests locally** you need to [set up a local dev environment](#developing). + +To run tests locally with PostgreSQL 17: + +``` shell +# Start a postgres instance (defaults to PostgreSQL 17) +mise run postgres:up --extra-args "--detach --wait" + +# Run the tests (defaults to PostgreSQL 17) +mise run test # Stop and remove all containers and networks mise run postgres:down ``` -You can run the same tasks for Postgres 14, 15, 16, and 17. +You can run the same tasks for Postgres 14, 15, 16, and 17 by specifying arguments: + +```shell +# Start a postgres 14 instance +mise run postgres:up postgres-14 --extra-args "--detach --wait" + +# Run the tests against postgres 14 +mise run test --postgres 14 + +# Stop postgres and remove all containers and networks +mise run postgres:down +``` The configuration for the Postgres containers in `tests/docker-compose.yml`. @@ -437,4 +606,5 @@ Limitations: `tests/docker-compose.yml` - **You can't run multiple Postgres containers at the same time.** All the containers bind to the same port (`7543`). If you want to run - multiple containers at the same time, you'll have to change the ports. + multiple containers at the same time, you have to change the ports by + editing `tests/docker-compose.yml` diff --git a/tasks/build.sh b/tasks/build.sh index 62a9ded..a8c5975 100755 --- a/tasks/build.sh +++ b/tasks/build.sh @@ -41,3 +41,12 @@ cat sql/666-rename_configuration_table.sql >> release/cipherstash-encrypt-uninst # remove the drop file rm release/cipherstash-encrypt-tmp-drop-install.sql rm release/cipherstash-encrypt-tmp-drop-uninstall.sql + +set +x +echo +echo '###############################################' +echo "# ✅Build succeeded" +echo '###############################################' +echo +echo 'Installer: release/cipherstash-encrypt.sql' +echo 'Uninstaller: release/cipherstash-encrypt-uninstall.sql'