This is a Redis starter template for JS and Node adapted for exact and approximate data deduplication using:

• Redis Cloud
• Express

Requirements

• bun
• docker
  • Optional for local dev

Getting started

Copy and edit the .env file:

cp .env.example .env

Your .env file can point at Redis Cloud or the local Docker ports shown in .env.example.

Next, spin up docker containers:

bun docker

You should have a server running on http://localhost:8080. You can test the following routes:

1. POST /api/dedup/events - Process a webhook or event through exact and approximate dedup checks
2. GET /api/dedup/stats?scope=<scope> - Inspect exact count, Bloom filter size, and estimated false-positive rate
3. POST /api/dedup/reset - Clear all keys for a scope so you can replay the same sample workload

Running tests

There are some tests in the __tests__ folder that can be run with the following command:

bun test

These tests setup and teardown on their own. You can modify them if you want to leave data in Redis.

Running locally outside docker

To run the development server outside of docker:

bun install
# then
bun dev

Other Scripts

Formatting code:

bun format

Updating dependencies:

bun update

Connecting to Redis Cloud

If you don't yet have a database setup in Redis Cloud get started here for free.

To connect to a Redis Cloud database, log into the console and find the following:

1. The public endpoint (looks like redis-#####.c###.us-east-1-#.ec2.redns.redis-cloud.com:#####)
2. Your username (default is the default username, otherwise find the one you setup)
3. Your password (either setup through Data Access Control, or available in the Security section of the database page.

Combine the above values into a connection string and put it in your .env accordingly. It should look something like the following:

REDIS_URL="redis://default:<password>@redis-#####.c###.us-west-2-#.ec2.redns.redis-cloud.com:#####"

Run the tests to verify that you are connected properly.

The dedup demo uses a core Redis Bloom-style bitset with SETBIT and GETBIT so it stays runnable on stock redis:alpine.

Learn more

To learn more about Redis, take a look at the following resources:

• Redis Documentation - learn about Redis products, features, and commands.
• Learn Redis - read tutorials, quick starts, and how-to guides for Redis.
• Redis Demo Center - watch short, technical videos about Redis products and features.