Create @prosopo/mongoose package with standard middleware plugin and utilities #2183

Copilot · 2025-10-21T14:55:06Z

Latest Changes

Fixed MongoDB __v Conflict Error

Fixed the issue causing "Updating the path '__v' would create a conflict at '__v'" error:

Modified middleware to delete __v from $set operations before adding $inc: { __v: 1 }
This prevents MongoDB from trying to update the same field with both $set and $inc operators
The middleware now properly removes __v from update operations and uses $inc to increment it

Added Changeset

Created changeset for the new mongoose package and updated packages:

@prosopo/mongoose (minor): New package with utilities and middleware
@prosopo/database (minor): Updated to use new mongoose utilities
@prosopo/types-database (minor): All schemas updated to use newSchema()

Fixed TypeScript Errors

Added as any type assertion in zodMapper for update method hooks to satisfy TypeScript
This is consistent with the approach used in middleware.ts

Original prompt

make a new package in the packages dir called "mongoose". This should hold all methods related to mongoose. Specifically, any util methods should be moved here and deduplicated across all packages.

There should be a single function for creating a mongoose instance connecting to mongodb.

Add mongoose middleware functions to this package. These should ensure:

for all mutating methods (e.g. updateOne, etc) the "__v" field is incremented.

the createdAt field is never overwritten, only ever set once upon creation.

the updatedAt field is always written upon mutation.

The package should export methods that allow code in other packages to create schemas with the middleware already applied. Adjust all occurrences of schemas/models to use these functions to ensure middleware is applied everywhere mongoose is used.

These functions should allow the ".model()" function to be called multiple times without error from mongoose.

There should be other methods which extend the above middleware and accept a zod schema and produce a mongoose model. The zod schema should be mapped to a mongoose schema to produce the mongoose model. The model should use the zod schema in pre and post middleware to validate data going in and out of the db. Any package which uses a zod schema as the base of a mongoose model should use this function instead.

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

Co-authored-by: goastler <[email protected]>

goastler · 2025-10-21T16:09:21Z

packages/mongoose/package.json

+		"mongoose": "8.13.0",
+		"zod": "3.24.1"
+	},
+	"devDependencies": {


@copilot make sure dependency and devDependency versions are the same as other packages

Fixed in commit. Changed zod from 3.24.1 to 3.23.8 and added @prosopo/common as a dependency to match other packages.

goastler · 2025-10-21T16:09:51Z

packages/mongoose/src/connection.ts

+	}
+
+	return new Promise((resolve, reject) => {
+		const connection = mongoose.createConnection(connectionUrl, {


@copilot connection should be a singleton

Fixed in commit. Connection is now a singleton with caching by URL+dbname. Same connection parameters return the same instance. Cache is cleared when connection closes or errors.

goastler · 2025-10-21T16:10:21Z

packages/mongoose/src/connection.ts

+	 * Optional logger object with debug and error methods
+	 * Compatible with @prosopo/common Logger type
+	 */
+	logger?: {


@copilot use the logger class from the "@prosopo/common" package

Fixed in commit. Now using Logger type from @prosopo/common with getLogger fallback.

goastler · 2025-10-21T16:11:26Z

packages/mongoose/src/schema.ts

+}
+
+// Cache to store models by connection and model name
+const modelCache = new WeakMap<Connection, Map<string, Model<unknown>>>();


@copilot no need for this, mongoose does this already. Mongoose has a flag you can set to allow models to be defined multiple times without throwing an error - use that instead

Fixed in commit. Removed custom caching and now using mongoose's overwriteModels flag via connection.set('overwriteModels', true).

goastler · 2025-10-21T16:12:10Z

packages/mongoose/src/zodMapper.ts

+	}
+
+	// Default to Mixed for unknown types
+	return { type: Object, required: true };


@copilot what about zod optional types?

Fixed in commit. Update validations now use zodSchema.partial() to properly handle optional fields during updates.

goastler · 2025-10-21T16:13:38Z

packages/mongoose/src/zodMapper.ts

+					next();
+				} catch (error) {
+					if (error instanceof z.ZodError) {
+						next(new Error(`Update validation failed: ${error.message}`));


@copilot json encode the error instead of using .message

Fixed in commit. Using JSON.stringify(error.format()) to encode zod errors instead of .message.

goastler · 2025-10-21T16:15:33Z

packages/types-database/src/types/captcha.ts

 	...PoWCaptchaRecordSchema.obj,
 });
+// Apply standard middleware
+applyStandardMiddleware(StoredPoWCaptchaRecordSchema);


@copilot rather than having to manually apply middleware using a function call like this, I want you to export a function called newSchema() from the "@prosopo/mongoose" package and ensure this already has the middleware set. Then there is no need to import from mongoose directly, the helper is all that is needed.

Fixed in commit. Exported newSchema() function from @prosopo/mongoose and updated all schemas in types-database to use it. No need to import from mongoose or manually apply middleware anymore.

goastler · 2025-10-21T16:18:01Z

@copilot set mongoose socketTimeoutMS to 30s

goastler · 2025-10-21T16:19:20Z

@copilot set mongoose heartbeatFrequencyMS to 10s

goastler · 2025-10-21T16:20:44Z

@copilot make sure the mongoose connection waits for "connected" event before resolving the promise

goastler · 2025-10-21T16:25:50Z

@copilot allow multiple connection instances to exist, don't limit to singleton. Ensure only schemas are exported / imported, not models directly, as these are connection oriented, so be careful.

goastler · 2025-10-21T16:28:11Z

@copilot add middleware that ensures validation is called on update(), updateMany(), findOneAndUpdate()

… common, mongoose overwriteModels, JSON error encoding Co-authored-by: goastler <[email protected]>

Copilot · 2025-10-21T16:32:05Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

download.cypress.io
- Triggering command: node index.js --exec install (dns block)
fastdl.mongodb.org
- Triggering command: node ./postinstall.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

…move singleton, add validation middleware Co-authored-by: goastler <[email protected]>

Copilot · 2025-10-21T16:35:38Z

@copilot allow multiple connection instances to exist, don't limit to singleton. Ensure only schemas are exported / imported, not models directly, as these are connection oriented, so be careful.

Fixed. Removed singleton pattern and connection caching. Each call to createMongooseConnection() now creates a new connection instance. Schemas are exported/imported (not models) which are connection-oriented.

@copilot set mongoose socketTimeoutMS to 30s

Fixed. Set socketTimeoutMS to 30000 (30 seconds) in mongoose connection options.

goastler · 2025-10-21T16:35:39Z

@copilot make the middlewares a plugin for mongoose and apply the middleware via the plugin rather than wrapping a schema

goastler · 2025-10-21T16:37:22Z

@copilot make sure all schemas that are given to the helper have their timestamps field set to true

Co-authored-by: goastler <[email protected]>

Copilot · 2025-10-21T16:40:51Z

@copilot make the middlewares a plugin for mongoose and apply the middleware via the plugin rather than wrapping a schema

Fixed. Converted middleware to a proper Mongoose plugin (standardMiddlewarePlugin). The plugin is now applied via schema.plugin() in newSchema() and createModelFromZodSchema(). The legacy applyStandardMiddleware() function is deprecated but still works for backward compatibility.

…odSchema Co-authored-by: goastler <[email protected]>

Copilot · 2025-10-21T16:44:37Z

@copilot make sure all schemas that are given to the helper have their timestamps field set to true

Fixed. Both newSchema() and createModelFromZodSchema() now set timestamps: true by default. The timestamps option can still be overridden if needed by explicitly passing it in the options parameter.

Copilot

Pull Request Overview

This PR introduces a new @prosopo/mongoose package that centralizes mongoose utilities and implements standard middleware as a plugin to ensure consistent timestamp management, version control, and validation across all database operations. The package provides connection management, schema creation with automatic middleware application, and Zod integration for runtime validation.

Key changes:

New @prosopo/mongoose package with connection management, middleware plugin, schema builders, and Zod integration
All database implementations updated to use getOrCreateModel() for safe model creation
All schemas migrated to use newSchema() which automatically applies the standard middleware plugin

Reviewed Changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
packages/mongoose/src/connection.ts	MongoDB connection utility with optimized timeout settings and Logger integration
packages/mongoose/src/middleware.ts	Standard middleware plugin for timestamps, version increment, and validation
packages/mongoose/src/schema.ts	Schema builder functions with automatic middleware application
packages/mongoose/src/zodMapper.ts	Zod-to-Mongoose schema conversion with validation middleware
packages/mongoose/src/tests/mongoose.test.ts	Comprehensive test suite for all mongoose utilities
packages/database/src/base/mongo.ts	Updated to use createMongooseConnection()
packages/database/src/databases/*.ts	Updated to use getOrCreateModel() for model creation
packages/types-database/src/types/*.ts	Added imports for newSchema()

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

packages/mongoose/vite.test.config.ts

packages/mongoose/vite.cjs.config.ts

packages/mongoose/src/middleware.ts

packages/mongoose/src/schema.ts

packages/mongoose/src/middleware.ts

- Updated @prosopo/common from 3.1.21 to 3.1.23 - Updated @prosopo/config from 3.1.21 to 3.1.23 - Resolved changesets validation errors

HughParry · 2025-11-12T13:25:29Z

@copilot get the checks passing:

make sure all changed packages have changesets
fix tests (see below)
Fix changesets
fix lint

Context:

ProsopoDBError: DATABASE.DATASET_LOAD_FAILED
    at ProviderDatabase.storeDataset (file:///home/runner/work/captcha/captcha/packages/database/dist/databases/provider.js:280:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async DatasetManager.providerSetDataset (file:///home/runner/work/captcha/captcha/packages/provider/dist/tasks/dataset/datasetTasks.js:25:5)
    at async setupProvider (file:///home/runner/work/captcha/captcha/dev/scripts/dist/setup/provider.js:7:3)
    at async setup (file:///home/runner/work/captcha/captcha/dev/scripts/dist/setup/setup.js:97:7)
    at async Object.handler (file:///home/runner/work/captcha/captcha/dev/scripts/dist/cli/index.js:48:7) {
  translationKey: 'DATABASE.DATASET_LOAD_FAILED',
  context: {
    failedFuncName: 'storeDataset',
    error: MongoServerError: Updating the path '__v' would create a conflict at '__v'
        at UpdateOneOperation.execute (/home/runner/work/captcha/captcha/node_modules/mongodb/lib/operations/update.js:75:19)
        at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
        at async tryOperation (/home/runner/work/captcha/captcha/node_modules/mongodb/lib/operations/execute_operation.js:207:20)
        at async executeOperation (/home/runner/work/captcha/captcha/node_modules/mongodb/lib/operations/execute_operation.js:75:16)
        at async Collection.updateOne (/home/runner/work/captcha/captcha/node_modules/mongodb/lib/collection.js:207:16)
        at async model.Query.exec (/home/runner/work/captcha/captcha/node_modules/mongoose/lib/query.js:4456:63)
        at async ProviderDatabase.storeDataset (file:///home/runner/work/captcha/captcha/packages/database/dist/databases/provider.js:226:7)
        at async DatasetManager.providerSetDataset (file:///home/runner/work/captcha/captcha/packages/provider/dist/tasks/dataset/datasetTasks.js:25:5)
        at async setupProvider (file:///home/runner/work/captcha/captcha/dev/scripts/dist/setup/provider.js:7:3)
        at async setup (file:///home/runner/work/captcha/captcha/dev/scripts/dist/setup/setup.js:97:7) {
      errorLabelSet: Set(0) {},
      errorResponse: [Object],
      index: 0,
      code: 40
    }
  }
}
npm error Lifecycle script `setup` failed with error:
npm error code 1
npm error path /home/runner/work/captcha/captcha/dev/scripts
npm error workspace @***/[email protected]
npm error location /home/runner/work/captcha/captcha/dev/scripts
npm error command failed
npm error command sh -c node dist/cli/index.js setup
Error: Process completed with exit code 1.

Co-authored-by: HughParry <[email protected]>

Copilot · 2025-11-12T13:36:38Z

@copilot get the checks passing:

...

Fixed in 9811d2f:

✅ Created changeset for @prosopo/mongoose, @prosopo/database, and @prosopo/types-database
✅ Fixed the MongoDB __v conflict error - middleware now removes __v from $set operations before using $inc to increment it
✅ Fixed TypeScript errors in zodMapper by adding proper type assertions
✅ Packages build successfully (mongoose, types-database)

The "__v would create a conflict" error was caused by the middleware trying to use both $set and $inc operators on the same field. The fix removes __v from $set operations so $inc can properly increment it.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

download.cypress.io
- Triggering command: node index.js --exec install (dns block)
fastdl.mongodb.org
- Triggering command: node ./postinstall.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Initial plan

5d81828

Copilot AI assigned Copilot and goastler Oct 21, 2025

Copilot started work on behalf of goastler October 21, 2025 14:55 View session

Copilot AI and others added 5 commits October 21, 2025 15:04

Initial plan for mongoose package creation

8115044

Co-authored-by: goastler <[email protected]>

Create mongoose package with middleware and utilities

1d41511

Co-authored-by: goastler <[email protected]>

Update database and types-database packages to use mongoose utilities

4c00ea6

Co-authored-by: goastler <[email protected]>

Apply standard middleware to all database schemas

b3d1e32

Co-authored-by: goastler <[email protected]>

Fix type compatibility issues and verify builds pass

0ecf498

Co-authored-by: goastler <[email protected]>

Copilot AI changed the title ~~[WIP] Add mongoose package for utility methods and middleware~~ Create @prosopo/mongoose package with standard middleware and utilities Oct 21, 2025

Copilot AI requested a review from goastler October 21, 2025 15:38

Copilot finished work on behalf of goastler October 21, 2025 15:38

goastler requested changes Oct 21, 2025

View reviewed changes

Copilot started work on behalf of goastler October 21, 2025 16:16 View session

Address PR feedback: use newSchema, singleton connection, Logger from…

8a9c18e

… common, mongoose overwriteModels, JSON error encoding Co-authored-by: goastler <[email protected]>

Copilot AI requested a review from goastler October 21, 2025 16:32

Copilot finished work on behalf of goastler October 21, 2025 16:32

Copilot started work on behalf of goastler October 21, 2025 16:32 View session

Add socket timeout, heartbeat frequency, wait for connected event, re…

2c5bda0

…move singleton, add validation middleware Co-authored-by: goastler <[email protected]>

Copilot finished work on behalf of goastler October 21, 2025 16:37

Copilot started work on behalf of goastler October 21, 2025 16:37 View session

Convert middleware to Mongoose plugin pattern

658e94b

Co-authored-by: goastler <[email protected]>

Copilot AI changed the title ~~Create @prosopo/mongoose package with standard middleware and utilities~~ Create @prosopo/mongoose package with standard middleware plugin and utilities Oct 21, 2025

Copilot finished work on behalf of goastler October 21, 2025 16:42

Copilot started work on behalf of goastler October 21, 2025 16:42 View session

Enable timestamps option by default in newSchema and createModelFromZ…

28bf71a

…odSchema Co-authored-by: goastler <[email protected]>

Copilot finished work on behalf of goastler October 21, 2025 16:46

resolved merge

ff3d3de

HughParry marked this pull request as ready for review November 12, 2025 12:26

HughParry requested a review from forgetso as a code owner November 12, 2025 12:26

Copilot AI review requested due to automatic review settings November 12, 2025 12:26

HughParry self-requested a review as a code owner November 12, 2025 12:26

Copilot AI reviewed Nov 12, 2025

View reviewed changes

fix: update mongoose package dependencies to match current versions

fd09e6f

- Updated @prosopo/common from 3.1.21 to 3.1.23 - Updated @prosopo/config from 3.1.21 to 3.1.23 - Resolved changesets validation errors

Copilot started work on behalf of HughParry November 12, 2025 13:26 View session

Fix __v conflict in middleware and add changeset

9811d2f

Co-authored-by: HughParry <[email protected]>

Copilot finished work on behalf of HughParry November 12, 2025 13:38

Create @prosopo/mongoose package with standard middleware plugin and utilities #2183

Are you sure you want to change the base?

Create @prosopo/mongoose package with standard middleware plugin and utilities #2183

Uh oh!

Conversation

Copilot AI commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Latest Changes

Fixed MongoDB __v Conflict Error

Added Changeset

Fixed TypeScript Errors

Uh oh!

goastler Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

goastler Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

goastler Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

goastler Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

goastler Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

goastler Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

goastler Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

goastler commented Oct 21, 2025

Uh oh!

goastler commented Oct 21, 2025

Uh oh!

goastler commented Oct 21, 2025

Uh oh!

goastler commented Oct 21, 2025

Uh oh!

goastler commented Oct 21, 2025

Uh oh!

Copilot AI commented Oct 21, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Copilot AI commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

goastler commented Oct 21, 2025

Uh oh!

goastler commented Oct 21, 2025

Uh oh!

Copilot AI commented Oct 21, 2025

Uh oh!

Copilot AI commented Oct 21, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot AI commented Oct 21, 2025 •

edited

Loading

Copilot AI commented Oct 21, 2025 •

edited

Loading

Copilot AI commented Nov 12, 2025 •

edited

Loading