Critical Task Section

[TonyValenti]

I have a piece of async code that does the following:

1. Request a time-sensitive download token from a web service (the token expires in under 10 seconds).
2. Download a file using the token.

Under low load, everything works fine, but when using dataflow with a high degree of concurrency, sometimes I see over a minute between step 1 completing and step 2 beginning. I have verified with fiddler that the HTTP request is complete so I suspect that the delay has something to do with the task scheduler running other threads.

Other than reducing concurrency, is there a way for me to say "This task is now time-sensitive. Run it ASAP."?

[Clockwork-Muse]

The actual delay here is probably between step 1 being submitted and the continuation (the code afterwards) being resumed. Step 2 doesn't actually need to be remote for this to be a problem. The actual HTTP request being complete isn't sufficient, it has to get back into C#'s process space, which is going to happen "when possible", essentially.

Your best bet is probably multiple task schedulers with different thread pools for different parts of the data flow - one for the token retrieval and file download, and one for everything else. If your files are very large, you might need to have a custom task scheduler that migrates to a different thread pool after the download is started or something funny like that.
Or possibly just one that runs all continuations synchronously (on the same thread) - you'd still write the consuming code as normal, with await, but under the covers it doesn't yield the thread.

Or you could go back to doing manual task submit and continuation creation, with Task.Run and ContinueWith(someTask, TaskContinuationOptions.ExecuteSynchronously). This is the old pattern, and there's some strange behavior around async calls, but would accomplish the same thing.

[TonyValenti]

Hi Stephen -
Is there a class or method I can call that allows me to get a dedicated task/thread pool or do I essentially have to roll my own?

[Clockwork-Muse]

You're probably going to have to roll your own if you have sufficient tasks - the real issue is getting the continuation for downloads to run immediately, and not be queued.

[TonyValenti]

Hi @Clockwork-Muse -
I'm playing around with a custom scheduler that I have written that guarantees that every task gets started immediately - either by using a new thread or by recycling a thread from a custom thread pool. I've also created a custom synchronization context that makes it possible for .ConfigureAwait(true) to marshal tasks back onto one of my dedicated threads.

However, something that I notice is that the system pool still seems to be involved in some way.

This is the code that I'm running:

        private static async Task DoWorkAsync(HttpClient HttpClient) {
            while (true) {
                var C = await HttpClient.GetAsync(Uri)
                   .ConfigureAwait(true)
                    ;

                await OuterAsync()
                    .ConfigureAwait(true)
                    ;

                await C.Content.ReadAsStringAsync()
                    .ConfigureAwait(true)
                    ;

            }

        }

       //Pretend that Outer and Inner are in a third party assembly.
        private static async Task OuterAsync() {
            await InnerAsync()
                .ConfigureAwait(false)
                ;
           //POINT 1
        }
        private static async Task InnerAsync() {
            await Task.Delay(1000)
                .ConfigureAwait(false)
                ;
        }

And I'm noticing that my custom SynchronizationContext's Post method always seems to be called from a .NET ThreadPool Worker. I expected that after a few iterations, the I would never see the thread pool again and that the Posts would come from a thread in my custom pool.

Is it possible to make that happen or is the system thread pool baked in at such a low level that it is unavoidable?

Related, I notice that when code gets to POINT1, it is back on a thread pool thread. I expect this because the previous line ConfigureAwait(false)ed, but I want to completely eliminate the thread pool from a portion of my code. Is it possible for me to also redirect ConfigureAwait(false) away from the thread pool?

[Clockwork-Muse]

Unfortunately I'm not more versed in this part than you are.

[davidfowl]

Is it possible for me to also redirect ConfigureAwait(false) away from the thread pool?

No, if you want to use a custom scheduler, then you can't use ConfigureAwait(false), it's job is to turn off all of the custom schedulers and use the thread pool. This is the relevant code for Tasks. It skips the TaskScheduler and the SynchronizationContext, if continueOnCapturedContext is false.