Failures when moving delayed messages to a queue are not sufficiently retried, and so are not delivered and instead sent to the error queue.

[LukeButters]

Describe the bug

Description

Failures while moving delayed messages to the queue are not sufficiently retried, the message ends up on the error queue 7s after the deliver at time where it is reasonable to expect it to be retried inline with the Recoverability configuration.

In this case we are moving from one sql backed endpoint to another (both in the same DB).

28/11/2025 00:05:18.694 Z : We log (user code) that we have publish a delayed message CheckScriptStatusSideEffect with id 24836dbe-915a-4a23-8139-b3a30001756c with a delay of 5s.

In the error queue we see that message and it has been moved to the error queue at 00:05:30 so only 7 seconds after it was set for delivery.

"NServiceBus.MessageId": "24836dbe-915a-4a23-8139-b3a30001756c",
"NServiceBus.TimeSent": "2025-11-28 00:05:18:653833 Z",
"NServiceBus.DeliverAt": "2025-11-28 00:05:23:653833 Z",
"NServiceBus.TimeOfFailure": "2025-11-28 00:05:30:344618 Z",
"NServiceBus.ExceptionInfo.Message": "The service has encountered an error processing your request. Please try again. Error code 9001.\nA severe error occurred on the current command.  The results, if any, should be discarded.",
"NServiceBus.ExceptionInfo.Source": "Core Microsoft SqlClient Data Provider",
"NServiceBus.ExceptionInfo.StackTrace": "Microsoft.Data.SqlClient.SqlException (0x80131904): The service has encountered an error processing your request. Please try again. Error code 9001.
A severe error occurred on the current command.  The results, if any, should be discarded.
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, SqlCommand command, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at Microsoft.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction2005(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
   at Microsoft.Data.SqlClient.SqlInternalTransaction.Commit()
   at Microsoft.Data.SqlClient.SqlTransaction.Commit()
   at NServiceBus.Transport.Sql.Shared.ProcessWithNativeTransaction.ProcessMessage(CancellationTokenSource stopBatchCancellationTokenSource, CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.Sql.Shared/Receiving/ProcessWithNativeTransaction.cs:line 53

ClientConnectionId:3404bce4-53f8-42cd-a55c-443314d9c343
Error Number:40197,State:4,Class:20\nClientConnectionId before routing:0c3dd88d-6725-427e-b90e-b08d64748a4e
Routing Destination:hstaging2003.database.windows.net\\e0c5fac33b42,4082",

Expected behavior

Retries are attempted inline with the retry policy of the endpoint, in this case the default behaviour.

Actual behavior

It gives up after 7s.

Versions

Please list the version of the relevant packages or applications in which the bug exists.

NServiceBus.Persistence.Sql 8.2.0
NServiceBus.Transport.SqlServer 8.1.10

Steps to reproduce

Publish delayed messages and have a connection to SQL die just at the time of moving from the delayed queue to the non delayed queue.

Relevant log output

Additional Information

Workarounds

Possible solutions

Additional information

[LukeButters]

With the help of AI I think the issue is:

⏺ TryGetFailureInfoForMessage returns true when a message has previously failed during processing in this endpoint instance and the failure information is still in the in-memory cache.

  Here's the scenario:

  For Delayed Messages Being Forwarded (Your selected code at line 91):

  First Attempt:
  1. Delayed message moves from delayed queue → input queue
  2. TryHandleDelayedMessage is called (ProcessWithNativeTransaction.cs:51)
  3. Tries to forward the message to destination queue (ProcessStrategy.cs:103)
  4. Forwarding fails (e.g., database connection issue, deadlock)
  5. Exception is caught at ProcessWithNativeTransaction.cs:72-78
  6. RecordFailureInfoForMessage is called (line 78) - Failure is stored in memory
  7. Transaction rolls back → message stays in input queue

  Second Attempt:
  1. Same message is received again from input queue
  2. TryHandleDelayedMessage is called again
  3. Checks TryGetFailureInfoForMessage at line 91 → Returns TRUE ✅
  4. Instead of trying to forward again, sends directly to error queue (line 96)
  5. This prevents the message from being stuck forever trying to forward

  For Normal Message Processing (ProcessWithNativeTransaction.cs:84):

  Same pattern but for regular message processing failures:
  - First attempt fails (line 100) → failure recorded
  - Second attempt checks (line 84) → returns true → invokes error handling/recoverability

  Key Points:

  - In-memory storage (FailureInfoStorage.cs) - uses LRU cache with max 50 elements (configurable)
  - Survives transaction rollback - stored in memory, not database
  - Lost on endpoint restart - not persisted
  - Per message ID - tracks failures by message.TransportId

  This mechanism ensures that if forwarding a delayed message fails, it won't retry forwarding indefinitely—it goes straight to the error queue on the next attempt.

So the fix is perhaps to make multiple attempts at forwarding the message.

[ramonsmits]

@LukeButters Thanks for reporting this issue. We've analyzed this and the current code assumption is that when "moving" from the source to the destination/target fails that this must be treated as a "poison" message. Likely the original idea was that this is sufficient issues like queue/table does not exist but not for transient/temporary issues.

You're right that we need to improve the error handling here.

Could you clarify how this impacts your organisation? How many messages fail because of this, the frequency, and any other context you can provide.

[LukeButters]

We are finding about once every few days a deployment process for our customers fails because of this issue. Our customers' deployments are business critical for them and we try to avoid recurring failures. The problem is made worse because we need to detect the failure, which for our domain can only happen by first waiting long enough (many hours) to be sure we are never getting a response message. This in turn can delay when a customer's business critical processes can proceed.

We have had to disable the use of NSB because it critically impacts our customers and our ability to meet our SLAs.

While this doesn't happen often, when the wrong message fails, it's a large impact, resulting in failures for our customers, and for our on-call staff who are woken up to resolve it. For non-delayed messages, we see plenty of transient errors, but these cases never fail entirely - the recoverability pipeline is always enough to bring it back.

[ramonsmits]

@LukeButters thanks for providing that detailed report on the impact. We will triage this and report its outcome here shortly.

[ramonsmits]

@LukeButters

For context it seems that this happens when (some) databases might not be available for writes. The Error code 9001 states that

Error 9001 occurs when the database log file becomes unavailable. When the database log goes offline that means a serious failure has occurred that prevents transactions to occur in the database.

Source: https://learn.microsoft.com/en-us/sql/relational-databases/errors-events/mssqlserver-9001-database-engine-error?view=sql-server-ver17

This seems like a very catastrophic failure.

---

We've analyzed some options and would appreciate your input by answering these questions:

1. Do you know what the state of the database is during these errors? Is this during a fail over or other serious reason for the databases not to be available for writes?
2. Are ALL commits failing or only this specific commit?

If indeed the database is in such a fail over state to us it does not feel right that we would be trying to move messages. We're considering to catch SqlServer specific errors that could hint at the database to be unavailable and gracefully handling these but infinitely retry these with a back off strategy.

3. What do you think about this strategy?