Skip to content

Commit e1b85a9

Browse files
benjiebenjie
authored
Overhaul "contributing" documentation (#517)
2 parents 6684e2a + 6ae0d21 commit e1b85a9

File tree

4 files changed

+236
-35
lines changed

4 files changed

+236
-35
lines changed

‎Dockerfile.dev

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
FROM node:18-alpine
22
RUN apk add --no-cache bash
3+
RUN apk add --no-cache postgresql-client
34

45
ENTRYPOINT ["yarn"]
56
CMD ["watch"]
@@ -14,8 +15,11 @@ COPY ./sql ./sql
1415
COPY ./src ./src
1516
COPY ./scripts ./scripts
1617
COPY ./perfTest ./perfTest
18+
COPY ./website ./website
1719
COPY ./tsconfig.json ./
1820
COPY ./jest.config.js ./
19-
21+
COPY ./.eslintrc.js ./
22+
COPY ./.prettierrc.js ./
23+
COPY ./.eslintignore ./
2024

2125
RUN yarn prepack

‎docker-compose.yml

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
version: "3.4"
2-
31
services:
42
db:
53
image: postgres:12-alpine
@@ -22,5 +20,9 @@ services:
2220
- ./sql:/app/sql
2321
- ./src:/app/src
2422
- ./perfTest:/app/perfTest
23+
- ./website:/app/website
2524
environment:
26-
TEST_CONNECTION_STRING: "postgres://postgres:workerdev@db:5432/graphile_worker_test"
25+
PGUSER: "postgres"
26+
PGPASSWORD: "workerdev"
27+
PGHOST: "db"
28+
PGPORT: "5432"

‎package.json

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,9 @@
2020
"perfTest": "cd perfTest && node ./run.js",
2121
"preversion": "grep '^### Pending' RELEASE_NOTES.md && echo \"⚠️ Cannot publish with 'Pending' in RELEASE_NOTES ⚠️\" && exit 1 || true",
2222
"version": "node scripts/postversion.mjs && git add src/version.ts",
23-
"website": "cd website && yarn run"
23+
"website": "cd website && yarn run",
24+
"website:start": "yarn website start",
25+
"website:deploy": "yarn website deploy"
2426
},
2527
"bin": {
2628
"graphile-worker": "./dist/cli.js"
@@ -104,7 +106,8 @@
104106
"sql"
105107
],
106108
"engines": {
107-
"node": ">=14.0.0"
109+
"node": ">=14.0.0",
110+
"yarn": "^1.22.22"
108111
},
109112
"browserslist": {
110113
"production": [

‎website/docs/contributing.md

Lines changed: 221 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -3,71 +3,263 @@ title: "Contributing"
33
sidebar_position: 170
44
---
55

6-
We love contributions from the community; but please: if you're planning to
7-
do something big, talk to us first. Graphile Worker is quite opinionated and
8-
prioritizes performance over many other things, so there's a risk that we
9-
may not want your feature in core, and we don't want you to waste your
10-
time!
6+
We love contributions from the community; but please: if you are planning to do
7+
something big, talk to us first. Graphile Worker is quite opinionated and
8+
prioritizes performance over many other things, so there is a risk that we may
9+
not want your feature in core, and we do not want you to waste your time!
1110

1211
## Development
1312

13+
### Setup
14+
15+
1. Ensure `yarn` is installed (e.g. `npm install -g yarn`).
16+
2. Fork and clone the (Graphile Worker git
17+
repository)[https://github.com/graphile/worker]
18+
3. From the root of your local Graphile Worker repository, run `yarn install`
19+
20+
### Ensure PostgreSQL is running
21+
22+
We assume you have a local PostgreSQL server running in "trust" authentication
23+
mode. Other options may or may not work - you may need to set `PGHOST`,
24+
`PGPORT`, `PGUSER`, `PGPASSWORD` and/or similar config variables.
25+
26+
If you don't have such a server, you can use docker to run it locally:
27+
28+
```bash
29+
# Run a temporary postgres instance on port 6432
30+
docker run --rm -it -e POSTGRES_HOST_AUTH_METHOD=trust -p 6432:5432 postgres:17
31+
```
32+
33+
Note that this Docker will keep running until you kill it (e.g. with `Ctrl-C`)
34+
and thus you will need to continue with a different terminal window.
35+
36+
Be sure to set the required environmental variables for this setup before you
37+
attempt to run the tests; you will need these for each terminal window that you
38+
attempt to run the tests from:
39+
40+
```bash
41+
export PGUSER=postgres
42+
export PGHOST=127.0.0.1
43+
export PGPORT=6432
44+
```
45+
46+
The command `psql postgres` should now work (exit with `Ctrl-D`). We require
47+
`psql` to install the test fixtures; if you don't have `psql` installed, install
48+
it using your operating system's package manager or from the
49+
[PostgreSQL website](https://www.postgresql.org/download/), for example:
50+
51+
```bash
52+
sudo apt update && sudo apt install postgresql-client
53+
```
54+
55+
### Automated Functional Testing
56+
57+
Graphile Worker leans on its automated tests to prevent regressions in
58+
functionality and performance. After making any change to the source code, you
59+
should run the test suite to ensure that you did not introduce any regressions.
60+
Any edit to the expected behavior should also include an accompanying additon to
61+
the test suite to prevent future regressions.
62+
63+
You must have a running Postgres database to run the tests. The test framework
64+
creates a template database. Each test clones the template database on demand.
65+
This allows the tests to run in parallel.
66+
67+
Run `yarn test` to run the tests, this will also set up the database.
68+
69+
:::tip Debugging
70+
71+
If you're having some trouble, you can run the tests in stages.
72+
73+
1. Compile the code: `yarn prepack`
74+
2. Setup the test DB: `yarn test:setupdb`
75+
3. Run the tests: `yarn test:only`
76+
77+
:::
78+
79+
:::warning Do not create a 'tasks' folder at the root!
80+
81+
If you have any files in `./tasks`, some tests will fail.
82+
83+
:::
84+
85+
### Running in CLI Mode
86+
87+
When users run the `graphile-worker` command they actually execute the script
88+
defined in `package.json` under `bin.graphile-worker`, which is `dist/cli.js`
89+
(corresponding with the `src/cli.ts` source file).
90+
91+
To run your local version of Graphile Worker similarly, run the `dist/cli.js`
92+
file with `node` directly. It will fail to start if you don't have any tasks, so
93+
you should create a tasks folder first (but not in the root!):
94+
95+
```sh
96+
yarn prepack
97+
mkdir -p _LOCAL/tasks
98+
echo 'module.exports = () => {}' > _LOCAL/tasks/hello.js
99+
cd _LOCAL
100+
node ../dist/cli.js -c "postgres:///my_db"
101+
```
102+
103+
:::tip Keep `dist` up to date with `yarn watch`
104+
105+
In development it's generally annoying to have to remember to run `yarn prepack`
106+
before each action. Instead, run `yarn watch` in a different terminal and the
107+
`dist` folder will stay up to date as you edit the source code.
108+
109+
:::
110+
111+
See the [CLI documentation](./cli/run.md) for more information about CLI mode.
112+
113+
### Running in Library Mode
114+
115+
When Graphile Worker users run in library mode, they use the functions exported
116+
from `src/index.ts`. The scrappiest thing you can do to run your local version
117+
of Graphile Worker similarly is to create a Typescript file that runs functions
118+
imported from `.`.
119+
120+
```ts title="src/temp.ts"
121+
import { run, WorkerPreset } from ".";
122+
123+
async function main() {
124+
const runner = await run({
125+
taskList: {
126+
hello: async (_, helpers) => {
127+
helpers.logger.info("Hello, world!");
128+
},
129+
},
130+
preset: {
131+
extends: [WorkerPreset],
132+
worker: {
133+
connectionString: "postgres:///my_db",
134+
},
135+
},
136+
});
137+
138+
await runner.promise;
139+
}
140+
141+
main().catch((err) => {
142+
console.error(err);
143+
process.exit(1);
144+
});
145+
```
146+
147+
Then you can run `temp.ts` with `ts-node`:
148+
14149
```sh
15-
yarn install
16-
yarn run watch
150+
yarn run ts-node src/temp.ts
17151
```
18152

19-
In another terminal:
153+
You have to remember not to commit `src/temp.ts`, so a cleaner way to achieve
154+
this would be using `yarn link`. In the root of your local Graphile Worker
155+
repository run the following:
20156

21157
```sh
22-
createdb graphile_worker_test
23-
yarn test
158+
yarn link
24159
```
25160

26-
### Using Docker to develop this module
161+
Create another node.js project with yarn that imports from `graphile-worker`
162+
like it would if it was using the published package. In that directory, run the
163+
following:
164+
165+
```sh
166+
yarn link graphile-worker
167+
```
27168

28-
Start the dev db and app in the background
169+
Note that once you link, you still need to compile your local graphile-worker
170+
package any time you make a change in the package that you want to test. You can
171+
compile with the following command:
29172

30173
```sh
31-
docker-compose up -d
174+
yarn prepack
32175
```
33176

34-
Run the tests
177+
If you're making frequent changes, you may want to automatically recompile any
178+
time there is a change. You can do so with the following command:
35179

36180
```sh
37-
docker-compose exec app yarn jest -i
181+
yarn watch
38182
```
39183

40-
Reset the test db
184+
See the [yarn link](https://classic.yarnpkg.com/lang/en/docs/cli/link/) docs for
185+
more information about how linking works, including instructions for unlinking.
186+
187+
### Docker Compose
188+
189+
Some people run their Graphile Worker development environments in Docker
190+
Compose. If this is you, please contribute back fixes to the setup, because our
191+
lead maintainer does not use it.
192+
193+
The `docker-compose.yml` file starts a minimal setup with a `db` container
194+
containing a Postgres database and an `app` container that is similar to running
195+
in CLI mode.
196+
197+
To rebuild the docker containers, run:
41198

42199
```sh
43-
cat __tests__/reset-db.sql | docker-compose exec -T db psql -U postgres -v GRAPHILE_WORKER_SCHEMA=graphile_worker graphile_worker_test
200+
docker compose build
44201
```
45202

46-
Run the perf tests
203+
To run the `db` and `app` containers in the backround, run the following:
47204

48205
```sh
49-
docker-compose exec app node ./perfTest/run.js
206+
docker compose up -d
50207
```
51208

52-
monitor the container logs
209+
You can run the tests via:
53210

54211
```sh
55-
docker-compose logs -f db
56-
docker-compose logs -f app
212+
docker compose exec app yarn test
57213
```
58214

59-
### Database migrations
215+
Tail the containers' logs the with the following command:
60216

61-
New database migrations must be accompanied by an updated db dump. This can be
62-
generated using the command `yarn db:dump`, and requires a running postgres 12
63-
server. Using docker:
217+
```sh
218+
docker compose logs -f
219+
```
220+
221+
### Authoring Database Migrations
222+
223+
New database migrations must be accompanied by an updated db dump. Before
224+
generating a new dump, ensure the following:
225+
226+
1. You have a Postgres running as described above.
227+
2. You have `pg_dump`, and the version of `pg_dump` is the same major version of
228+
your Postgres database.
229+
230+
To check your `pg_dump` version, run the following:
231+
232+
```sh
233+
pg_dump --version
234+
```
235+
236+
To check your Postgres version, run the following:
64237

65238
```sh
66-
docker run -e POSTGRES_HOST_AUTH_METHOD=trust -d -p 5432:5432 postgres:12
239+
psql postgres:///template1 -c "SELECT version();"
67240
```
68241

69-
then run
242+
To update the db dump, run the following command:
70243

71244
```sh
72-
PGUSER=postgres PGHOST=localhost yarn run db:dump
245+
yarn db:dump
73246
```
247+
248+
### Developing With Windows Machines
249+
250+
The maintainer does not have access to a Windows development machine, so he
251+
cannot ensure that the development environment works.
252+
253+
[This comment](https://github.com/graphile/worker/pull/316#issuecomment-1427173046)
254+
suggests that at least one change needs to be made to support contributing from
255+
a Windows machine. If you use Windows and want to help here, please do!
256+
257+
One option is to try using the docker-compose setup detailed above.
258+
259+
## Contributing to the Documentation
260+
261+
The docs are maintained in the
262+
[main Graphile Worker Repository](https://github.com/graphile/worker/tree/main/website/docs).
263+
See the
264+
[Website README](https://github.com/graphile/worker/blob/main/website/README.md)
265+
for more info on the website.

0 commit comments

Comments
 (0)