Deadlocks

[uler3161]

I'm randomly seeing deadlocks in the database. Has anyone else seen this:

Services.SignalR.VideoHubService:0: "Error in SQL notification loop Microsoft.Data.SqlClient.SqlException (0x80131904): Transaction (Process ID 100) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, SqlCommand command, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows)
   at Microsoft.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more)
   at IntelliTect.AspNetCore.SignalR.SqlServer.Internal.SqlReceiver.ReadRows(Action`1 beforeExecute)
   at IntelliTect.AspNetCore.SignalR.SqlServer.Internal.SqlReceiver.NotificationLoop(CancellationToken cancellationToken)

[uler3161]

I found this same problem here: SignalR/SignalR#4402. I suspect it's doing pretty much the same kind of logic? I tried increasing table count. Doesn't help.

I ran the SQL profiler and I really have to wonder what is going on. In approximately 38 seconds I only had 4 (very small) messages get added. But there was a total of 1382 queries. The vast majority are either the payload select statement or "end conversation". There are only about 50 servers using the backplane, so I really have to wonder what all this traffic is about. And I suspect it's related to the deadlocks.

Edit: Forgot that I had more processes that counted as servers than I thought. Still, even with 50, I was thinking there'd be at most 200 queries to retrieve messages.

[uler3161]

The thought had occurred to me that I only want one server to listen for messages. That might fix my problems by cutting down on so many queries, but I don't see an option to disable receiving of messages.

[ascott18]

If you're seeing polling queries occur, you've likely not enabled Service Broker per the documentation.

50 servers using the backplane

I'm going to say that you've already exceeded the capabilities of this project, again per the documentation.

only want one server to listen for messages

That's not how SignalR works. Every server that clients can connect to has to be receiving messages from the backplane.

[uler3161]

Thanks for the response.

I do have Service Broker enabled.

It's not clear in the docs what "very high degrees of scale-out" means, but I suppose with 50 servers you might be right. I was thinking with the relatively low throughput I need that it wouldn't be a problem. But perhaps my definition of a server might affect whether that's the case as 49 out of the 50 of these servers have no clients. Clients connect to a web server, but these other servers need to get messages to the clients.

I envision a non-backplane solution where the 49 servers connect to the web server hub as if they were clients, send their messages to the web server, then the web server forwards the messages to the clients. I was hoping to use the backplane instead of this because of at least these reasons:

1. Ideally I'd like the 49 servers to not know about the web server.

2. I'm thinking maybe if the web server is down, when it comes back up it will read in all the queued messages from the backplane and send them out. I don't know if it works that way, but might be nice if it does.

3. It seems cleaner in my code. The web server sends the same kind of messages as the other servers. I thought it'd be better to have a single method in my IHubContext implementation that all servers could call. But I suppose it's not that big of a deal. It's just an extra method that calls the existing method.

None of these are reasons are deal-breakers, especially with the limited types of messages I'm trying to send. So, I guess I'll go in that direction.