"Reply received from MongoDB, but no request was waiting for the result." after 12~ hours

[elihwyma]

Describe the bug
When using via Fluent in Vapor, after approximately 12 hours the database will no longer be responsive and the console is littered with messages from MongoKitten.

To Reproduce
Continuously use the same opened database for over (roughly) 12 hours. The timings are an average from Sentry.

Environment:

• OS: swift:6.0-jammy-slim
• MongoKitten Version: 6.7.15
• MongoDB: mongo:latest container

Additional context
Paired with issue in Fluent: vapor/fluent-mongo-driver#62

[Joannis]

Hey! I don't actively develop the 6.x series of MongoKitten, but given the nature of the bug I've submitted a patch #355
Let me know if this solves your issue. Unfortunately we didn't have race condition checking back then, but we did migrate to a multithreaded design.

[elihwyma]

Hey, thanks for a quick patch. I will build a new container with that now and report back.

[elihwyma]

Update does not seem to have worked unfortunately.

I will attempt to investigate it myself over the coming days and submit a PR if I figure it out :)

[Joannis]

@elihwyma is it possible that you're running into MongoKitten's timeouts?

[elihwyma]

@elihwyma is it possible that you're running into MongoKitten's timeouts?

I really wouldn't think so, I can try adjusting the timeouts. Both Docker containers (app and mongo) are hosted on the same machine 🤷‍♀️

[Joannis]

@elihwyma can you check the query times on the local MongoDB instance? I don't think it's a TCP timeout, but a slow query

[elihwyma]

@Joannis this may be a monumental me problem. I reset a bunch of stuff and this seemed to work. Some things may be "pointless" but I was trying everything:

• mongo:latest
• Add --maxConns 1000 to post arguments
• Add env GLIBC_TUNABLES=glibc.pthread.rseq=0
• Still using your race condition fix branch
  I'll keep monitoring this setup 😓

[Joannis]

Are you using change streams?

[elihwyma]

Never mind, stopped working at some point.

Are you using change streams?

Can you elaborate please, pretty much everything is default and I don't know much more than that 😅

[Joannis]

You'd know if you were setting up a change stream. Are you able to join the MongoKitten, Hummingbird or Vapor discord server? I'd love to talk more directly about this