|
| 1 | +--- |
| 2 | +description: Pre-seeding database with schema and data at startup for development environment |
| 3 | +keywords: Pre-seeding, database, postgres, container-supported development |
| 4 | +title: Pre-seeding database with schema and data at startup for development environment |
| 5 | +linktitle: Pre-seeding database |
| 6 | +--- |
| 7 | + |
| 8 | +Pre-seeding databases with essential data and schema during local development is a common practice to enhance the development and testing workflow. By simulating real-world scenarios, this practice helps catch frontend issues early, ensures alignment between Database Administrators and Software Engineers, and facilitates smoother collaboration. Pre-seeding offers benefits like confident deployments, consistency across environments, and early issue detection, ultimately improving the overall development process. |
| 9 | + |
| 10 | +In this guide, you will learn how to: |
| 11 | + |
| 12 | +- Use Docker to launch up a Postgres container |
| 13 | +- Pre-seed Postgres using a SQL script |
| 14 | +- Pre-seed Postgres by copying SQL files into Docker image |
| 15 | +- Pre-seed Postgres using JavaScript code |
| 16 | + |
| 17 | +## Using Postgres with Docker |
| 18 | + |
| 19 | +The [official Docker image for Postgres](https://hub.docker.com/_/postgres) provides a convenient way to run Postgres database on your development machine. A Postgres Docker image is a pre-configured environment that encapsulates the PostgreSQL database system. It's a self-contained unit, ready to run in a Docker container. By using this image, you can quickly and easily set up a Postgres instance without the need for manual configuration. |
| 20 | + |
| 21 | +## Prerequisites |
| 22 | + |
| 23 | +The following prerequisites are required to follow along with this how-to guide: |
| 24 | + |
| 25 | +- [Docker Desktop](https://www.docker.com/products/docker-desktop/) |
| 26 | + |
| 27 | +## Launching Postgres |
| 28 | + |
| 29 | +Launch a quick demo of Postgres by using the following steps: |
| 30 | + |
| 31 | +1. Open the terminal and run the following command to start a Postgres container. |
| 32 | + |
| 33 | + This example will launch a Postgres container, expose port `5432` onto the host to let a native-running application to connect to it with the password `mysecretpassword`. |
| 34 | + |
| 35 | + ```console |
| 36 | + $ docker run -d --name postgres -p 5432:5432 -e POSTGRES_PASSWORD=mysecretpassword postgres |
| 37 | + ``` |
| 38 | + |
| 39 | +2. Verify that Postgres is up and running by selecting the container and checking the logs on Docker Dashboard. |
| 40 | + |
| 41 | + ```plaintext |
| 42 | + PostgreSQL Database directory appears to contain a database; Skipping initialization |
| 43 | + |
| 44 | + 2024-09-08 09:09:47.136 UTC [1] LOG: starting PostgreSQL 16.4 (Debian 16.4-1.pgdg120+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit |
| 45 | + 2024-09-08 09:09:47.137 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 |
| 46 | + 2024-09-08 09:09:47.137 UTC [1] LOG: listening on IPv6 address "::", port 5432 |
| 47 | + 2024-09-08 09:09:47.139 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" |
| 48 | + 2024-09-08 09:09:47.142 UTC [29] LOG: database system was shut down at 2024-09-08 09:07:09 UTC |
| 49 | + 2024-09-08 09:09:47.148 UTC [1] LOG: database system is ready to accept connections |
| 50 | + ``` |
| 51 | + |
| 52 | +3. Connect to Postgres from the local system. |
| 53 | + |
| 54 | + The `psql` is the PostgreSQL interactive shell that is used to connect to a Postgres database and let you start executing SQL commands. Assuming that you already have `psql` utility installed on your local system, it's time to connect to the Postgres database. Run the following command on your local terminal: |
| 55 | + |
| 56 | + ```console |
| 57 | + $ docker exec -it postgres psql -h localhost -U postgres |
| 58 | + ``` |
| 59 | + |
| 60 | + You can now execute any SQL queries or commands you need within the `psql` prompt. |
| 61 | + |
| 62 | + Use `\q` or `\quit` to exit from the Postgres interactive shell. |
| 63 | + |
| 64 | +## Pre-seed the Postgres database using a SQL script |
| 65 | + |
| 66 | +Now that you've familiarized yourself with Postgres, it's time to see how to pre-seed it with sample data. In this demonstration, you'll first create a script that holds SQL commands. The script defines the database, and table structure and inserts sample data. Then you will connect the database to verify the data. |
| 67 | + |
| 68 | +Assuming that you have an existing Postgres database instance up and running, follow these steps to seed the database. |
| 69 | + |
| 70 | +1. Create an empty file named `seed.sql` and add the following content. |
| 71 | + |
| 72 | + ```sql |
| 73 | + CREATE DATABASE sampledb; |
| 74 | + |
| 75 | + \c sampledb |
| 76 | + |
| 77 | + CREATE TABLE users ( |
| 78 | + id SERIAL PRIMARY KEY, |
| 79 | + name VARCHAR(50), |
| 80 | + email VARCHAR(100) UNIQUE |
| 81 | + ); |
| 82 | + |
| 83 | + INSERT INTO users (name, email) VALUES |
| 84 | + |
| 85 | + |
| 86 | + |
| 87 | + ``` |
| 88 | + |
| 89 | + The SQL script creates a new database called `sampledb`, connects to it, and creates a `users` table. The table includes an auto-incrementing `id` as the primary key, a `name` field with a maximum length of 50 characters, and a unique `email` field with up to 100 characters. |
| 90 | + |
| 91 | + After creating the table, the `INSERT` command inserts three users into the `users` table with their respective names and emails. This setup forms a basic database structure to store user information with unique email addresses. |
| 92 | + |
| 93 | +2. Seed the database. |
| 94 | + |
| 95 | + It’s time to feed the content of the `seed.sql` directly into the database by using the `<` operator. The command is used to execute a SQL script named `seed.sql` against a Postgres database named `sampledb`. |
| 96 | + |
| 97 | + ```console |
| 98 | + $ cat seed.sql | docker exec -i postgres psql -h localhost -U postgres -f- |
| 99 | + ``` |
| 100 | + |
| 101 | + Once the query is executed, you will see the following results: |
| 102 | + |
| 103 | + ```plaintext |
| 104 | + CREATE DATABASE |
| 105 | + You are now connected to database "sampledb" as user "postgres". |
| 106 | + CREATE TABLE |
| 107 | + INSERT 0 3 |
| 108 | + ``` |
| 109 | + |
| 110 | +3. Run the following `psql` command to verify if the table named users is populated in the database `sampledb` or not. |
| 111 | + |
| 112 | + ```console |
| 113 | + $ docker exec -it postgres psql -h localhost -U postgres sampledb |
| 114 | + ``` |
| 115 | + |
| 116 | + You can now run `\l` in the `psql` shell to list all the databases on the Postgres server. |
| 117 | + |
| 118 | + ```console |
| 119 | + sampledb=# \l |
| 120 | + List of databases |
| 121 | + Name | Owner | Encoding | Collate | Ctype | ICU Locale | Locale Provider | Access privileges |
| 122 | + -----------+----------+----------+------------+------------+------------+-----------------+----------------------- |
| 123 | + postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 | | libc | |
| 124 | + sampledb | postgres | UTF8 | en_US.utf8 | en_US.utf8 | | libc | |
| 125 | + template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | | libc | =c/postgres + |
| 126 | + | | | | | | | postgres=CTc/postgres |
| 127 | + template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | | libc | =c/postgres + |
| 128 | + | | | | | | | postgres=CTc/postgres |
| 129 | + (4 rows) |
| 130 | + ``` |
| 131 | + |
| 132 | + To retrieve all the data from the users table, enter the following query: |
| 133 | + |
| 134 | + ```console |
| 135 | + sampledb=# SELECT * FROM users; |
| 136 | + id | name | email |
| 137 | + ----+-------+------------------- |
| 138 | + |
| 139 | + |
| 140 | + |
| 141 | + (3 rows) |
| 142 | + ``` |
| 143 | + |
| 144 | + Use `\q` or `\quit` to exit from the Postgres interactive shell. |
| 145 | + |
| 146 | +## Pre-seed the database by bind-mounting a SQL script |
| 147 | + |
| 148 | +In Docker, mounting refers to making files or directories from the host system accessible within a container. This let you to share data or configuration files between the host and the container, enabling greater flexibility and persistence. |
| 149 | + |
| 150 | +Now that you have learned how to launch Postgres and pre-seed the database using an SQL script, it’s time to learn how to mount an SQL file directly into the Postgres containers’ initialisation directory (`/docker-entrypoint-initdb.d`). The `/docker-entrypoint-initdb.d` is a special directory in PostgreSQL Docker containers that is used for initializing the database when the container is first started |
| 151 | + |
| 152 | +Make sure you stop any running Postgres containers (along with volumes) to prevent port conflicts before you follow the steps: |
| 153 | + |
| 154 | +```console |
| 155 | +$ docker container stop postgres |
| 156 | +``` |
| 157 | + |
| 158 | +1. Modify the `seed.sql` with the following entries: |
| 159 | + |
| 160 | + ```sql |
| 161 | + CREATE TABLE IF NOT EXISTS users ( |
| 162 | + id SERIAL PRIMARY KEY, |
| 163 | + name VARCHAR(50), |
| 164 | + email VARCHAR(100) UNIQUE |
| 165 | + ); |
| 166 | + |
| 167 | + INSERT INTO users (name, email) VALUES |
| 168 | + |
| 169 | + |
| 170 | + |
| 171 | + ON CONFLICT (email) DO NOTHING; |
| 172 | + ``` |
| 173 | + |
| 174 | +2. Create a text file named `Dockerfile` and copy the following content. |
| 175 | + |
| 176 | + ```plaintext |
| 177 | + # syntax=docker/dockerfile:1 |
| 178 | + FROM postgres:latest |
| 179 | + COPY seed.sql /docker-entrypoint-initdb.d/ |
| 180 | + ``` |
| 181 | + |
| 182 | + This Dockerfile copies the `seed.sql` script directly into the PostgreSQL container's initialization directory. |
| 183 | + |
| 184 | + |
| 185 | +3. Use Docker Compose. |
| 186 | + |
| 187 | + Using Docker Compose makes it even easier to manage and deploy the PostgreSQL container with the seeded database. This compose.yml file defines a Postgres service named `db` using the latest Postgres image, which sets up a database with the name `sampledb`, along with a user `postgres` and a password `mysecretpassword`. |
| 188 | + |
| 189 | + ```yaml |
| 190 | + services: |
| 191 | + db: |
| 192 | + build: |
| 193 | + context: . |
| 194 | + dockerfile: Dockerfile |
| 195 | + container_name: my_postgres_db |
| 196 | + environment: |
| 197 | + POSTGRES_USER: postgres |
| 198 | + POSTGRES_PASSWORD: mysecretpassword |
| 199 | + POSTGRES_DB: sampledb |
| 200 | + ports: |
| 201 | + - "5432:5432" |
| 202 | + volumes: |
| 203 | + - data_sql:/var/lib/postgresql/data # Persistent data storage |
| 204 | + |
| 205 | + volumes: |
| 206 | + data_sql: |
| 207 | + ``` |
| 208 | + |
| 209 | + It maps port `5432` on the host to the container's `5432`, let you access to the Postgres database from outside the container. It also define `data_sql` for persisting the database data, ensuring that data is not lost when the container is stopped. |
| 210 | + |
| 211 | + It is important to note that the port mapping to the host is only necessary if you want to connect to the database from non-containerized programs. If you containerize the service that connects to the DB, you should connect to the database over a custom bridge network. |
| 212 | + |
| 213 | +4. Bring up the Compose service. |
| 214 | + |
| 215 | + Assuming that you've placed the `seed.sql` file in the same directory as the Dockerfile, execute the following command: |
| 216 | + |
| 217 | + ```console |
| 218 | + $ docker compose up -d --build |
| 219 | + ``` |
| 220 | + |
| 221 | +5. It’s time to verify if the table `users` get populated with the data. |
| 222 | + |
| 223 | + ```console |
| 224 | + $ docker exec -it my_postgres_db psql -h localhost -U postgres sampledb |
| 225 | + ``` |
| 226 | + |
| 227 | + ```sql |
| 228 | + sampledb=# SELECT * FROM users; |
| 229 | + id | name | email |
| 230 | + ----+-------+------------------- |
| 231 | + |
| 232 | + |
| 233 | + |
| 234 | + (3 rows) |
| 235 | +
|
| 236 | + sampledb=# |
| 237 | + ``` |
| 238 | + |
| 239 | + |
| 240 | +## Pre-seed the database using JavaScript code |
| 241 | + |
| 242 | + |
| 243 | +Now that you have learned how to seed the database using various methods like SQL script, mounting volumes etc., it's time to try to achieve it using JavaScript code. |
| 244 | + |
| 245 | +1. Create a .env file with the following: |
| 246 | + |
| 247 | + ```plaintext |
| 248 | + POSTGRES_USER=postgres |
| 249 | + POSTGRES_DB_HOST=localhost |
| 250 | + POSTGRES_DB=sampledb |
| 251 | + POSTGRES_PASSWORD=mysecretpassword |
| 252 | + POSTGRES_PORT=5432 |
| 253 | + ``` |
| 254 | + |
| 255 | +2. Create a new JavaScript file called seed.js with the following content: |
| 256 | + |
| 257 | + The following JavaScript code imports the `dotenv` package which is used to load environment variables from an `.env` file. The `.config()` method reads the `.env` file and sets the environment variables as properties of the `process.env` object. This let you to securely store sensitive information like database credentials outside of your code. |
| 258 | + |
| 259 | + Then, it creates a new Pool instance from the pg library, which provides a connection pool for efficient database interactions. The `seedData` function is defined to perform the database seeding operations. |
| 260 | +It is called at the end of the script to initiate the seeding process. The try...catch...finally block is used for error handling. |
| 261 | + |
| 262 | + ```plaintext |
| 263 | + require('dotenv').config(); // Load environment variables from .env file |
| 264 | + const { Pool } = require('pg'); |
| 265 | +
|
| 266 | + // Create a new pool using environment variables |
| 267 | + const pool = new Pool({ |
| 268 | + user: process.env.POSTGRES_USER, |
| 269 | + host: process.env.POSTGRES_DB_HOST, |
| 270 | + database: process.env.POSTGRES_DB, |
| 271 | + port: process.env.POSTGRES_PORT, |
| 272 | + password: process.env.POSTGRES_PASSWORD, |
| 273 | + }); |
| 274 | +
|
| 275 | + const seedData = async () => { |
| 276 | + try { |
| 277 | + // Drop the table if it already exists (optional) |
| 278 | + await pool.query(`DROP TABLE IF EXISTS todos;`); |
| 279 | + |
| 280 | + // Create the table with the correct structure |
| 281 | + await pool.query(` |
| 282 | + CREATE TABLE todos ( |
| 283 | + id SERIAL PRIMARY KEY, |
| 284 | + task VARCHAR(255) NOT NULL, |
| 285 | + completed BOOLEAN DEFAULT false |
| 286 | + ); |
| 287 | + ` ); |
| 288 | +
|
| 289 | + // Insert seed data |
| 290 | + await pool.query(` |
| 291 | + INSERT INTO todos (task, completed) VALUES |
| 292 | + ('Watch netflix', false), |
| 293 | + ('Finish podcast', false), |
| 294 | + ('Pick up kid', false); |
| 295 | + `); |
| 296 | + console.log('Database seeded successfully!'); |
| 297 | + } catch (err) { |
| 298 | + console.error('Error seeding the database', err); |
| 299 | + } finally { |
| 300 | + pool.end(); |
| 301 | + } |
| 302 | + }; |
| 303 | +
|
| 304 | + // Call the seedData function to run the script |
| 305 | + seedData(); |
| 306 | + ``` |
| 307 | + |
| 308 | +3. Kick off the seeding process |
| 309 | + |
| 310 | + ```console |
| 311 | + $ node seed.js |
| 312 | + ``` |
| 313 | + |
| 314 | + You should see the following command: |
| 315 | + |
| 316 | + ```plaintext |
| 317 | + Database seeded successfully! |
| 318 | + ``` |
| 319 | + |
| 320 | +4. Verify if the database is seeded correctly: |
| 321 | + |
| 322 | + ```console |
| 323 | + $ docker exec -it postgres psql -h localhost -U postgres sampledb |
| 324 | + ``` |
| 325 | + |
| 326 | + ```console |
| 327 | + sampledb=# SELECT * FROM todos; |
| 328 | + id | task | completed |
| 329 | + ----+----------------+----------- |
| 330 | + 1 | Watch netflix | f |
| 331 | + 2 | Finish podcast | f |
| 332 | + 3 | Pick up kid | f |
| 333 | + (3 rows) |
| 334 | + ``` |
| 335 | + |
| 336 | +## Recap |
| 337 | + |
| 338 | +Pre-seeding a database with schema and data at startup is essential for creating a consistent and realistic testing environment, which helps in identifying issues early in development and aligning frontend and backend work. This guide has equipped you with the knowledge and practical steps to achieve pre-seeding using various methods, including SQL script, Docker integration, and JavaScript code. |
| 339 | + |
0 commit comments