|
| 1 | +# Dgraph Testing Framework |
| 2 | + |
| 3 | +Dgraph employs a ~~complex~~ sophisticated testing framework that includes extensive test coverage. |
| 4 | +Due to the comprehensive nature of these tests, a complete test run can take several hours, |
| 5 | +depending on your hardware. To manage this complex testing process efficiently, we've developed a |
| 6 | +custom test framework implemented in Go: [t.go](t.go). This specialized runner provides enhanced |
| 7 | +control and flexibility beyond what's available through the standard Go testing framework. |
| 8 | + |
| 9 | +**Note:** This testing framework was built with Linux in mind. Non-Linux testing _can_ be |
| 10 | +achieved—see [Running tests on OSX](#running-tests-on-osx) below. |
| 11 | + |
| 12 | +## Requirements |
| 13 | + |
| 14 | +The framework requires several external dependencies. You can check your system for the required |
| 15 | +dependencies by running `make check`. |
| 16 | + |
| 17 | +### Go |
| 18 | + |
| 19 | +Version 1.22.12 or higher. |
| 20 | + |
| 21 | +### Docker |
| 22 | + |
| 23 | +The framework uses Docker extensively for integration testing. Tests will often "fail" if Docker is |
| 24 | +not given enough resources, specifically memory. If you experience testing that seems to hang |
| 25 | +waiting for connections to other cluster members, it's probably a memory issue. |
| 26 | + |
| 27 | +The test runner takes a concurrency flag (./t -j=N) which will attempt to create _N_ number of |
| 28 | +Dgraph clusters and run tests concurrently in those clusters. If you are testing on a machine with |
| 29 | +limited resources, we advise you to not set this above _1_ (which is the default). |
| 30 | + |
| 31 | +You can preserve the test Docker containers for failure analysis with the `--keep` flag. |
| 32 | + |
| 33 | +### gotestsum |
| 34 | + |
| 35 | +The framework uses [gotestsum](https://github.com/gotestyourself/gotestsum#install) for collating |
| 36 | +test output and other advanced functions. |
| 37 | + |
| 38 | +### protoc |
| 39 | + |
| 40 | +On non-Linux systems, protocol buffer tests are skipped. On Linux systems, instructions for |
| 41 | +installing and configuring protoc can be found [here](https://github.com/protocolbuffers/protobuf). |
| 42 | +Or, `sudo apt update && sudo apt install -y protobuf-compiler`. |
| 43 | + |
| 44 | +## Running Tests |
| 45 | + |
| 46 | +The tests use the Dgraph binary found at $(GOPATH)/bin/dgraph. To check your current $GOPATH: |
| 47 | +`go env GOPATH`. |
| 48 | + |
| 49 | +--- |
| 50 | + |
| 51 | +Use the `make install` target in the top-level Makefile to build a binary with your changes that |
| 52 | +need testing. Note for non-Linux users: because the binary is run in the Docker environment, it |
| 53 | +needs to be a valid Linux executable. See the section below on |
| 54 | +[Running tests on OSX](#running-tests-on-osx). |
| 55 | + |
| 56 | +First, build the `t` program if you haven't already: |
| 57 | + |
| 58 | +```sh |
| 59 | +make check && go build . |
| 60 | +``` |
| 61 | + |
| 62 | +This will produce a `t` executable, which is the testing framework _cli_. |
| 63 | + |
| 64 | +To see a list of available flags: |
| 65 | + |
| 66 | +```sh |
| 67 | +./t --help |
| 68 | +``` |
| 69 | + |
| 70 | +### Testing packages |
| 71 | + |
| 72 | +One popular use of the framework is to target a specific package for integration testing. For |
| 73 | +instance, to test the GraphQL system: |
| 74 | + |
| 75 | +```sh |
| 76 | +./t --pkg=graphql/e2e/normal |
| 77 | +``` |
| 78 | + |
| 79 | +Multiple packages can be specified by separated them with a comma. |
| 80 | + |
| 81 | +### Testing suites |
| 82 | + |
| 83 | +You can test one or more "suites" of functionality using the `--suite` flag. For instance: |
| 84 | + |
| 85 | +```sh |
| 86 | +./t --suite=core,vector |
| 87 | +``` |
| 88 | + |
| 89 | +The `--help` flag lists the available suites. |
| 90 | + |
| 91 | +### Testing single test functions |
| 92 | + |
| 93 | +```sh |
| 94 | +./t --test=TestParseCountValError |
| 95 | +``` |
| 96 | + |
| 97 | +This flag uses `ack` to find all tests matching the specified name(s). |
| 98 | + |
| 99 | +### Other useful flags |
| 100 | + |
| 101 | +The `--dry` (dry-run) flag can be used to list the packages that will be included for testing |
| 102 | +without actually invoking the tests. |
| 103 | + |
| 104 | +The `--skip-slow` flag will skip tests known to be slow to complete. |
| 105 | + |
| 106 | +## Docker Compose Conventions |
| 107 | + |
| 108 | +The Docker-based integration tests follow a hierarchical file discovery system. When executing a |
| 109 | +test, the framework first searches for a docker-compose.yml file in the test's immediate package |
| 110 | +directory. If no file is found, it progressively checks parent directories until it locates one. The |
| 111 | +root-level test configuration file is stored at |
| 112 | +[../dgraph/docker-compose.yml](../dgraph/docker-compose.yml). When implementing tests that require |
| 113 | +unique Docker configurations not covered by existing compose files, you should create a new |
| 114 | +directory with your tests also containing a custom docker-compose.yml file tailored to your specific |
| 115 | +testing requirements. |
| 116 | + |
| 117 | +## Running tests on OSX |
| 118 | + |
| 119 | +The testing framework works well on Linux systems. Some additional steps need to be taken for the |
| 120 | +tests to run on OSX. |
| 121 | + |
| 122 | +### Install location |
| 123 | + |
| 124 | +The Docker environment used to perform integration testing uses the dgraph binary found in |
| 125 | +$GOPATH/bin. This binary is required to be a GOOS=linux image. The following commands need to be run |
| 126 | +prior to starting tests to ensure the appropriate images are in place. Note, if your GOPATH variable |
| 127 | +is not set, run ``export GOPATH=`go env GOPATH` `` |
| 128 | + |
| 129 | +```sh |
| 130 | +cd .. |
| 131 | +# builds the OSX version |
| 132 | +make install |
| 133 | +mv $GOPATH/bin/dgraph $GOPATH/bin/dgraph_osx |
| 134 | +# builds the linux version, take note of where the target reports it has written the dgraph executable |
| 135 | +GOOS=linux make install |
| 136 | +mv $GOPATH/bin/linux_arm64/dgraph $GOPATH/bin/dgraph |
| 137 | +cd t |
| 138 | +make check |
| 139 | +``` |
| 140 | + |
| 141 | +The following environment variables are needed when tests are executed: |
| 142 | + |
| 143 | +- GOPATH - needed to map the Dgraph image in Docker environments |
| 144 | +- DGRAPH_BINARY - the system-native (OSX) dgraph image, used by some tests not in the Docker |
| 145 | + environment |
| 146 | +- DOCKER_HOST - on newer Docker Desktop versions, the Docker communications socket was moved to your |
| 147 | + home folder |
| 148 | + |
| 149 | +Example: |
| 150 | + |
| 151 | +```sh |
| 152 | +export GOPATH=`go env GOPATH` |
| 153 | +export DGRAPH_BINARY=$GOPATH/bin/dgraph_osx |
| 154 | +export DOCKER_HOST=unix://${HOME}/.docker/run/docker.sock |
| 155 | +``` |
| 156 | + |
| 157 | +At this point, the `t` executable can be run as described above. |
| 158 | + |
| 159 | +### Common Pitfalls |
| 160 | + |
| 161 | +If you see `exec format error` output from test runs, it is most likely because some tests attempt |
| 162 | +to run the Dgraph image copied from the filesystem in the Docker environment. This is a known issue |
| 163 | +with some integration tests. |
0 commit comments