diff --git a/contents/handbook/engineering/developing-locally.md b/contents/handbook/engineering/developing-locally.md index c4bbb5389b54..3c81ffa6d813 100644 --- a/contents/handbook/engineering/developing-locally.md +++ b/contents/handbook/engineering/developing-locally.md @@ -35,14 +35,14 @@ These components rely on a few external services: - Redis – for caching and inter-service communication - Zookeeper – for coordinating Kafka and ClickHouse clusters -When spinning up an instance of PostHog for development, we recommend the following configuration: +When spinning up an instance of PostHog for development, we recommend the following hybrid configuration: -- the external services run in Docker over `docker compose` -- PostHog itself runs on the host (your system) +- External services (ClickHouse, Kafka, PostgreSQL, Redis, etc.) run in Docker via `docker compose` +- PostHog apps (Django, frontend, plugin-server, Celery) run on the host using `hogli start` (which uses mprocs, a terminal UI, to manage and display logs from all processes simultaneously) -This is what we'll be using in the guide below. +This approach gives you fast iteration on the code you're developing while keeping infrastructure isolated. -> It is also technically possible to run PostHog in Docker completely, but syncing changes is then much slower, and for development you need PostHog dependencies installed on the host anyway (such as formatting or typechecking tools). +> It is also technically possible to run PostHog in Docker completely, but syncing code changes is then much slower, and for development you need PostHog dependencies installed on the host anyway (such as formatting or typechecking tools). > The other way around – everything on the host, is not practical due to significant complexities involved in instantiating Kafka or ClickHouse from scratch. The instructions here assume you're running macOS or the current Ubuntu Linux LTS (24.04). @@ -53,27 +53,9 @@ Windows isn't supported natively. But, Windows users can run a Linux virtual mac In case some steps here have fallen out of date, please tell us about it – feel free to [submit a patch](https://github.com/PostHog/posthog.com/blob/master/contents/handbook/engineering/developing-locally.md)! -## Option 1: Developing with Codespaces +## Option 1: Developing locally -This is a faster option to get up and running. If you don't want to or can't use Codespaces, continue from the next section. - -1. Create your codespace. -![](https://user-images.githubusercontent.com/890921/231489405-cb2010b4-d9e3-4837-bfdf-b2d4ef5c5d0b.png) -2. Update it to 8-core machine type (the smallest is probably too small to get PostHog running properly). -![](https://user-images.githubusercontent.com/890921/231490278-140f814e-e77b-46d5-9a4f-31c1b1d6956a.png) -3. Open the codespace, using one of the "Open in" options from the list. -4. In the codespace, open a terminal window and run `docker compose -f docker-compose.dev.yml up`. -5. Ensure that you are using the right Node version (`nvm install 22 && nvm use 22`) then, in another terminal, run `pnpm i` (and use the same terminal for the following commands). -6. Then run `uv sync` - - If this doesn't activate your python virtual environment, run `uv venv` (install `uv` following the [uv standalone installer guide](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer) if needed) -7. Install `sqlx-cli` with `cargo install sqlx-cli` (install Cargo following the [Cargo getting started guide](https://doc.rust-lang.org/cargo/getting-started/installation.html) if needed) -8. Now run `DEBUG=1 ./bin/migrate` -9. Install [mprocs](https://github.com/pvolok/mprocs#installation) (`cargo install mprocs`) -10. Run `./bin/start`. -11. Open browser to . -12. To get some practical test data into your brand-new instance of PostHog, run `DEBUG=1 ./manage.py generate_demo_data`. - -## Option 2: Developing locally +This is the recommended option for most developers. ### Prerequisites @@ -111,9 +93,11 @@ This is a faster option to get up and running. If you don't want to or can't use Clone the [PostHog repo](https://github.com/posthog/posthog). All future commands assume you're inside the `posthog/` folder. ```bash -git clone https://github.com/PostHog/posthog && cd posthog/ +git clone --filter=blob:none https://github.com/PostHog/posthog && cd posthog/ ``` +**Performance tip:** The `--filter=blob:none` flag downloads all commit history and tree structure, but defers file contents (blobs) until needed. This reduces the clone from ~3 GB to a few hundred MB and makes the initial clone **15-17x faster**. You still get full git history for commands like `git log` and `git diff` – blobs are fetched on demand as you use them. + > The `feature-flags` container relies on the presence of the GeoLite cities > database in the `/share` directory. If you haven't run `./bin/start` this database may not exist. > You can explicitly download it by running `./bin/download-mmdb`. You may also need to modify the @@ -121,18 +105,18 @@ git clone https://github.com/PostHog/posthog && cd posthog/ > > `chmod 0755 ./share/GeoLite2-City.mmdb` -### Instant setup +### Setup with Flox (recommended) -You can set your development environment up instantly using [Flox](https://flox.dev/). +Set up your development environment instantly using [Flox](https://flox.dev/). -Flox is a development environment manager – it ensures we all have the same right system-level dependencies when developing PostHog. It's pretty much an npm for runtimes and libraries: `.flox/env/manifest.toml` is like `package.json`, `.flox/env/manifest.lock` is akin to `package-lock.json`, and `.flox/cache/` resembles `node_modules/`. +Flox manages your development environment. The `manifest.toml` file declares all dependencies (similar to `package.json`), and Flox automatically provides the correct versions for your system. To get PostHog running in a dev environment: -1. Once you have cloned the repo and installed OrbStack, now install Flox (plus `ruff` and `rustup` for pre-commit checks outside the Flox env). +1. Once you have cloned the repo and installed OrbStack, install Flox: ```bash - brew install flox ruff rustup && rustup-init && rustup default stable + brew install flox ``` 2. From the root of the repository, activate the environment. (On first activation, you'll be prompted if you'd like the environment to be activated automatically using `direnv`.) @@ -151,299 +135,71 @@ This is it – you should be seeing the PostHog app at **Friendly tip 1:** If you see `Error while fetching server API version: 500 Server Error for http+docker://localhost/version:`, it's likely that Docker Engine isn't running. - -> **Friendly tip 2:** If you see "Exit Code 137" anywhere, it means that the container has run out of memory. In this case you need to allocate more RAM in OrbStack settings. - -> **Friendly tip 3:** On Linux, you might need `sudo` – see [Docker docs on managing Docker as a non-root user](https://docs.docker.com/engine/install/linux-postinstall). Or look into [Podman](https://podman.io/getting-started/installation) as an alternative that supports rootless containers. - ->**Friendly tip 4:** If you see `Error: (HTTP code 500) server error - Ports are not available: exposing port TCP 0.0.0.0:5432 -> 0.0.0.0:0: listen tcp 0.0.0.0:5432: bind: address already in use`, you have Postgres already running somewhere. Try `docker compose -f docker-compose.dev.yml` first, alternatively run `lsof -i :5432` to see what process is using this port. - -```bash -sudo service postgresql stop -``` - -Second, verify via `docker ps` and `docker logs` (or via the OrbStack dashboard) that all these services are up and running. They should display something like this in their logs: - -```shell -# docker ps NAMES -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -5a38d4e55447 temporalio/ui:2.10.3 "./start-ui-server.sh" 51 seconds ago Up 44 seconds 0.0.0.0:8081->8080/tcp posthog-temporal-ui-1 -89b969801426 temporalio/admin-tools:1.20.0 "tail -f /dev/null" 51 seconds ago Up 44 seconds posthog-temporal-admin-tools-1 -81fd1b6d7b1b clickhouse/clickhouse-server:23.6.1.1524 "/entrypoint.sh" 51 seconds ago Up 50 seconds 0.0.0.0:8123->8123/tcp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:9009->9009/tcp, 0.0.0.0:9440->9440/tcp posthog-clickhouse-1 -f876f8bff35f bitnami/kafka:2.8.1-debian-10-r99 "/opt/bitnami/script…" 51 seconds ago Up 50 seconds 0.0.0.0:9092->9092/tcp posthog-kafka-1 -d22559261575 temporalio/auto-setup:1.20.0 "/etc/temporal/entry…" 51 seconds ago Up 45 seconds 6933-6935/tcp, 6939/tcp, 7234-7235/tcp, 7239/tcp, 0.0.0.0:7233->7233/tcp posthog-temporal-1 -5313fc278a70 postgres:12-alpine "docker-entrypoint.s…" 51 seconds ago Up 50 seconds (healthy) 0.0.0.0:5432->5432/tcp posthog-db-1 -c04358d8309f zookeeper:3.7.0 "/docker-entrypoint.…" 51 seconds ago Up 50 seconds 2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp posthog-zookeeper-1 -09add699866e maildev/maildev:2.0.5 "bin/maildev" 51 seconds ago Up 50 seconds (healthy) 0.0.0.0:1025->1025/tcp, 0.0.0.0:1080->1080/tcp posthog-maildev-1 -61a44c094753 elasticsearch:7.16.2 "/bin/tini -- /usr/l…" 51 seconds ago Up 50 seconds 9200/tcp, 9300/tcp posthog-elasticsearch-1 -a478cadf6911 minio/minio:RELEASE.2022-06-25T15-50-16Z "sh -c 'mkdir -p /da…" 51 seconds ago Up 50 seconds 9000/tcp, 0.0.0.0:19000-19001->19000-19001/tcp posthog-object_storage-1 -91f838afe40e redis:6.2.7-alpine "docker-entrypoint.s…" 51 seconds ago Up 50 seconds 0.0.0.0:6379->6379/tcp posthog-redis-1 - -# docker logs posthog-db-1 -n 1 -2021-12-06 13:47:08.325 UTC [1] LOG: database system is ready to accept connections - -# docker logs posthog-redis-1 -n 1 -1:M 06 Dec 2021 13:47:08.435 * Ready to accept connections - -# docker logs posthog-clickhouse-1 -n 1 -Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'. - -# ClickHouse writes logs to `/var/log/clickhouse-server/clickhouse-server.log` and error logs to `/var/log/clickhouse-server/clickhouse-server.err.log` instead of stdout/stsderr. It can be useful to `cat` these files if there are any issues: -# docker exec posthog-clickhouse-1 cat /var/log/clickhouse-server/clickhouse-server.log -# docker exec posthog-clickhouse-1 cat /var/log/clickhouse-server/clickhouse-server.err.log - -# docker logs posthog-kafka-1 -[2021-12-06 13:47:23,814] INFO [KafkaServer id=1001] started (kafka.server.KafkaServer) - -# docker logs posthog-zookeeper-1 -# Because ClickHouse and Kafka connect to Zookeeper, there will be a lot of noise here. That's good. -``` +### Common gotchas -> **Friendly tip 1:** Kafka is currently the only x86 container used, and might segfault randomly when running on ARM. Restart it when that happens. +These issues can occur regardless of whether you're using Flox or manual setup. -> **Friendly tip 2:** Checking the last Clickhouse log could show a `get_mempolicy: Operation not permitted` message. However, it shouldn't affect the app startup - checking the whole log should clarify that Clickhouse started properly. To double-check you can get into the container and run a basic query. -> -> ```bash -> # docker logs posthog-clickhouse-1 -> # docker exec -it posthog-clickhouse-1 bash -> # clickhouse-client --query "SELECT 1" +**Docker/OrbStack resource limits** +If you see "Exit Code 137" or out-of-memory errors, your Docker container doesn't have enough resources. In OrbStack settings, allocate **at least 4 GB RAM** (8 GB recommended) and **at least 4 CPU cores** (400%). -Finally, install Postgres locally. Even if you are planning to run Postgres inside Docker, we need a local copy of Postgres (version 11+) for its CLI tools and development libraries/headers. These are required by `pip` to install `psycopg2`. +**Docker not running** +If you see `Error while fetching server API version: 500 Server Error for http+docker://localhost/version`, make sure Docker (or OrbStack) is actually running. -- On macOS: - - ```bash - brew install postgresql - ``` +**Port conflicts** +If you see a port binding error for 5432, you have Postgres running locally. Use `lsof -i :5432` to find the process, then `sudo service postgresql stop` to stop it. -This installs both the Postgres server and its tools. DO NOT start the server after running this. +**GeoLite database missing** +The feature-flags container needs the GeoLite database in `/share`. If it's missing, run `./bin/download-mmdb` and then `chmod 0755 ./share/GeoLite2-City.mmdb`. -- On Debian-based Linux: +**ClickHouse "get_mempolicy" warning** +You might see `get_mempolicy: Operation not permitted` in the ClickHouse logs. This is harmless and can be ignored. To verify ClickHouse started properly, run `docker exec -it posthog-clickhouse-1 bash` then `clickhouse-client --query "SELECT 1"`. - ```bash - sudo apt install -y postgresql-client postgresql-contrib libpq-dev - ``` +**Database migration errors** +If you see `fe_sendauth: no password supplied`, set `DATABASE_URL=postgres://posthog:posthog@localhost:5432/posthog` and ensure containers are running. On ARM machines, you may also hit `psycopg2` errors – see [this comment](https://github.com/psycopg/psycopg2/issues/1216#issuecomment-820556849) for fixes. -This intentionally only installs the Postgres client and drivers, and not the server. If you wish to install the server, or have it installed already, you will want to stop it, because the TCP port it uses conflicts with the one used by the Postgres Docker container. +**Frontend typegen stuck in loop** +The first time you run typegen, it may get stuck. Cancel it (`Ctrl+C`), run `git reset --hard`, then try again. You may need to discard changes once more when the second round completes. -On Linux, it's recommended to disable Postgres service by default, to ensure no port conflict arises. If `postgres` is already running on the port `5432`, you can confirm it by checking the port, and then kill it manually. +**"layout.html is not defined" error** +This happens on first startup. Wait for the frontend to finish compiling and try accessing the app again. -```bash -sudo systemctl disable postgresql.service -sudo lsof -i :5432 -sudo kill -9 `sudo lsof -t -i :5432` -``` +**Kafka segfaults on ARM** +Kafka is an x86 container and may segfault randomly on ARM machines. Simply restart it when that happens. -On Linux you often have separate packages: `postgres` for the tools, `postgres-server` for the server, and `libpostgres-dev` for the `psycopg2` dependencies. Consult your distro's list for an up-to-date list of packages. +**Apple Silicon OpenSSL issues** +On Apple Silicon Macs, you may get build errors related to OpenSSL. For plugin-server: set `CPPFLAGS=-I/opt/homebrew/opt/openssl/include` and `LDFLAGS=-L/opt/homebrew/opt/openssl/lib` before installing. For Python packages, you may need custom OpenSSL headers – consult the [xmlsec issue](https://github.com/xmlsec/python-xmlsec/issues/254) for details. -#### 2. Prepare the frontend +**Plugin server rebuild** +If the plugin server won't start, try `cd plugin-server && pnpm rebuild && pnpm i`. -1. Install nvm, with `brew install nvm` or by following the instructions at . If using fish, you may instead prefer . +**Python setuptools error** +If you see `import gyp # noqa: E402` during plugin-server install, run `brew install python-setuptools`. -
- After installation, make sure to follow the instructions printed in your terminal to add NVM to your{' '} - $PATH. Otherwise the command line will use your system Node.js version instead. -
+**OpenSSL certificate verification error** +If you get `Configuration property "enable.ssl.certificate.verification" not supported in this build: OpenSSL not available at build time` when running `./bin/start`, set the right OpenSSL environment variables as described in [this issue](https://github.com/xmlsec/python-xmlsec/issues/261#issuecomment-1630889826) and try again. -2. Install the latest Node.js 22 (the version used by PostHog in production) with `nvm install 22`. You can start using it in the current shell with `nvm use 22`. - -3. Install pnpm by running `corepack enable` and then running `corepack prepare pnpm@9 --activate`. Validate the installation with `pnpm --version`. - -4. Install Node packages by running `pnpm i`. - -5. Run `pnpm --filter=@posthog/frontend typegen:write` to generate types for [Kea](https://keajs.org/) state management logics used all over the frontend. - -> The first time you run typegen, it may get stuck in a loop. If so, cancel the process (`Ctrl+C`), discard all changes in the working directory (`git reset --hard`), and run `pnpm typegen:write` again. You may need to discard all changes once more when the second round of type generation completes. - -#### 3. Prepare plugin server - -1. Install the `brotli` compression library and `rust` stable via `rustup`: - -- On macOS: - - ```bash - brew install brotli rustup - rustup default stable - rustup-init - # Select 1 to proceed with default installation - ``` - -- On Debian-based Linux: - - ```bash - sudo apt install -y brotli - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh - # Select 1 to proceed with default installation - ``` - -2. Run `pnpm --filter=@posthog/plugin-server install` to install all required packages. We'll actually run the plugin server in a later step. - -> **Note:** If you face an error like `ld: symbol(s) not found for architecture arm64`, most probably your openssl build flags are coming from the wrong place. To fix this, run: - -```bash -export CPPFLAGS=-I/opt/homebrew/opt/openssl/include -export LDFLAGS=-L/opt/homebrew/opt/openssl/lib -pnpm --filter=@posthog/plugin-server install -``` - -> **Note:** If you face an error like `import gyp # noqa: E402`, most probably need to install `python-setuptools`. To fix this, run: - -```bash -brew install python-setuptools -``` - -> **Troubleshooting plugin server issues:** If you encounter problems starting up the plugin server, try these debugging steps: - -```bash -cd plugin-server -pnpm rebuild -pnpm i -``` +**pyproject.toml parse warnings** +When running `uv sync`, you may see a `Failed to parse` warning related to `pyproject.toml`. This is usually harmless – if you see the `Activate with:` line at the end, your environment was created successfully. -#### 4. Prepare the Django server +## Option 2: Developing with Codespaces -1. Install a few dependencies for SAML to work. If you're on macOS, run the command below, otherwise check the official [xmlsec repo](https://github.com/mehcode/python-xmlsec) for more details. +This is a faster option to get up and running if you can't or don't want to set up locally. - - On macOS: - - ```bash - brew install libxml2 libxmlsec1 pkg-config - ``` - - > If installing `xmlsec` doesn't work, try updating macOS to the latest version (Sonoma). - - - On Debian-based Linux: - - ```bash - sudo apt install -y libxml2 libxmlsec1-dev libffi-dev pkg-config - ``` - -1. Install Python 3.11. - - - On macOS, you can do so with Homebrew: `brew install python@3.11`. - - - On Debian-based Linux: - - ```bash - sudo add-apt-repository ppa:deadsnakes/ppa -y - sudo apt update - sudo apt install python3.11 python3.11-venv python3.11-dev -y - ``` - -Make sure when outside the venv to always use `python3` instead of `python`, as the latter may point to Python 2.x on some systems. If installing multiple versions of Python 3, such as by using the `deadsnakes` PPA, use `python3.11` instead of `python3`. - -You can also use [pyenv](https://github.com/pyenv/pyenv) if you wish to manage multiple versions of Python 3 on the same machine. - -1. Install `uv` - -`uv` is a very fast tool you can use for python virtual env and dependency management. See [https://docs.astral.sh/uv/](https://docs.astral.sh/uv/). Once installed you can prefix any `pip` command with `uv` to get the speed boost. - -1. Create the virtual environment with the right Python version, and install dependencies - all in one with this command: - - ```bash - uv sync - ``` - - > **Friendly tip:** Creating an env could raise a `Failed to parse` warning related to `pyproject.toml`. However, you should still see the `Activate with:` line at the very end, which means that your env was created successfully. - -1. Activate the virtual environment: - - ```bash - # For bash/zsh/etc. - source .venv/bin/activate - - # For fish - source .venv/bin/activate.fish - ``` - -1. Install requirements with uv - - If your workstation is an Apple Silicon Mac, the first time you install Python packages, you must set custom OpenSSL headers: - - ```bash - brew install openssl - CFLAGS="-I /opt/homebrew/opt/openssl/include $(python3.11-config --includes)" LDFLAGS="-L /opt/homebrew/opt/openssl/lib" GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 uv sync - ``` - - > **Friendly tip:** If you see `ERROR: Could not build wheels for xmlsec`, refer to this [issue](https://github.com/xmlsec/python-xmlsec/issues/254). - - These will be used when installing `grpcio` and `psycopg2`. After doing this once, and assuming nothing changed with these two packages, next time simply run: - - ```bash - uv sync - ``` - -#### 5. Prepare databases - -We now have the backend ready, and Postgres and ClickHouse running – these databases are blank slates at the moment however, so we need to run _migrations_ to e.g. create all the tables: - -```bash -cargo install sqlx-cli # If you haven't already -DEBUG=1 ./bin/migrate -``` - -> **Friendly tip 1:** The error `fe_sendauth: no password supplied` connecting to Postgres happens when the database is set up with a password and the user:pass isn't specified in `DATABASE_URL`. Try `export DATABASE_URL=postgres://posthog:posthog@localhost:5432/posthog`. - -> **Friendly tip 2:** You may run into `psycopg2` errors while migrating on an ARM machine. Try out the steps in this [comment](https://github.com/psycopg/psycopg2/issues/1216#issuecomment-820556849) to resolve this. - -> **Friendly tip 3:** When migrating, make sure the containers are running (detached or in a separate terminal tab). - -#### 6. Start PostHog - -Now start all of PostHog (backend, worker, plugin server, and frontend – simultaneously) with one of: - -```bash -./bin/start - -# only services strictly required to run posthog -./bin/start --minimal - -# if you want to log additionally each process to a /tmp/posthog-.log file for AI code editors to be able to grep -./bin/start --custom bin/mprocs-with-logging.yaml -``` - -> **Note:** This command uses [mprocs](https://github.com/pvolok/mprocs) to run all development processes in a single terminal window. It will be installed automatically for macOS, while for Linux you can install it manually (`cargo` or `npm`) using the official repo guide. - -> **Friendly tip:** If you get the error `Configuration property "enable.ssl.certificate.verification" not supported in this build: OpenSSL not available at build time`, make sure your environment is using the right `openssl` version by setting [those](https://github.com/xmlsec/python-xmlsec/issues/261#issuecomment-1630889826) environment variables, and then run `./bin/start` again. - -Open [http://localhost:8010](http://localhost:8010) to see the app. - -> **Note:** The first time you run this command you might get an error that says "layout.html is not defined". Make sure you wait until the frontend is finished compiling and try again. - -To get some practical test data into your brand-new instance of PostHog, run `DEBUG=1 ./manage.py generate_demo_data`. For a list of useful arguments of the command, run `DEBUG=1 ./manage.py generate_demo_data --help`. - -> **Friendly Tip** The first time you run the app, you can log in with a test account: _user_:`test@posthog.com` _pwd_:`12345678`. - -#### 7. Develop - -This is it – you should be seeing the PostHog app at http://localhost:8010. - -You can now change PostHog in any way you want. See [Project structure](/handbook/engineering/project-structure) for an intro to the repository's contents. To commit changes, create a new branch based on `master` for your intended change, and develop away. +1. Create your codespace. +![](https://user-images.githubusercontent.com/890921/231489405-cb2010b4-d9e3-4837-bfdf-b2d4ef5c5d0b.png) +2. Update it to 8-core machine type (the smallest is probably too small to get PostHog running properly). +![](https://user-images.githubusercontent.com/890921/231490278-140f814e-e77b-46d5-9a4f-31c1b1d6956a.png) +3. Open the codespace, using one of the "Open in" options from the list. +4. In the codespace, open a terminal window and run `docker compose -f docker-compose.dev.yml up`. +5. Ensure that you are using the right Node version (`nvm install 22 && nvm use 22`) then, in another terminal, run `pnpm i` (and use the same terminal for the following commands). +6. Then run `uv sync` + - If this doesn't activate your python virtual environment, run `uv venv` (install `uv` following the [uv standalone installer guide](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer) if needed) +7. Install `sqlx-cli` with `cargo install sqlx-cli` (install Cargo following the [Cargo getting started guide](https://doc.rust-lang.org/cargo/getting-started/installation.html) if needed) +8. Now run `DEBUG=1 ./bin/migrate` +9. Install [mprocs](https://github.com/pvolok/mprocs#installation) (`cargo install mprocs`) +10. Run `./bin/start`. +11. Open browser to . +12. To get some practical test data into your brand-new instance of PostHog, run `DEBUG=1 ./manage.py generate_demo_data`. ## Testing