Parallel Canvas File Downloads

[eriq-augustine]

Right now when fetching assignment submission files, we compile a list of the file URLs and then download the files in serial.
However fetching the file does not require any authentication (aside from the link itself), so we may be able to download these files in parallel.
Investigate this a bit to see if we can download in parallel and if it makes sense use a modest sized parallel pool (4? 8?) to download it.

Related to #16.

[Ayush-Vish]

@eriq-augustine

In this issue, we can download the submission files in parallel.
It makes sense as an assignment usually has 150-200 students.
It will cut down the downloading time significantly.
We can use concurrent.futures.ThreadPoolExecutor and pass the callback to download a file using the URL, i.e., _download_file().

I am going with the concurrent.futures .

Please confirm if you want to use some other library like threading.

[eriq-augustine]

@Ayush-Vish

I think you are missing the main point here.
We clearly have no technical issues here (of course we have the ability to run things in parallel).
The issue here is whether or not Canvas (or whatever other LMS) allows this.

Also, you say you want to use concurrent.futures.ThreadPoolExecutor, but is that not subject to the GIL?
Will there be enough IO blocking on a download to negate blocking by the GIL?

[Ayush-Vish]

Hi, @eriq-augustine. Sorry for the delay.
So far, I have done some research.

• We are not sending an auth token or user ID for auth, but the canvas rate limiter has IP as an identifier as well, which means the download endpoint is rate-limited.
  There was not much detailed documentation, so I went to code and found some insights:
  The canvas uses a leaky bucket algo for rate limiting.
  Specs:
  Bucket size (HWM) = 600
  refill rate = 10 units/sec
  each req costs 50 units upfront, later changed to the actual cost.

We can work with two workers if we want reliability for 200-250 files.
Also, we need to stagger the request to 5 req/sec to avoid filling up the bucket.
The cost in this case will never exceed 200 units in the ideal case, i.e., not with huge delays due to network conditions.

So yes, we can download the files in parallel.

Coming onto the GIL:
The library is subjected to GIL, but it works well as Python releases the GIL when I/O is happening.
https://stackoverflow.com/questions/32256775/how-does-the-gil-in-python-affect-the-downloading-of-webpages-in-parallel

[eriq-augustine]

Only 2 workers does not seem to match the math you have here.
I think it would be incorrect to think of this in terms of number of workers and not in terms closer to the bucketing system Canvas uses.

[Ayush-Vish]

Hi @eriq-augustine
You are right, let me explain my solution
Canvas metrics
Bucket Size = 600 units (HWM)
Refill rate = 10 units/sec
Each request costs 50 units upfront
So our main aim is to keep the Bucket size < 600 units to avoid 403.

If we take the delay of 200ms i.e, 5 req/sec
Cost = 5*50 = 250 - 10(refill rate ) = 240 units.
we will hit the hwm in ~ 2.5 sec if the req is not complete.
But the Bucket will leak as the request is completed.
That's why we are also limiting the pool size.
If the pool size is 2-4, we will get

• Natural throttling from I/O delays.
• Combined with 200ms delay, i.e., 5 req/sec to stay below the hwm.

so the plan is

• To Enforce 5 req/sec.
• Use 4 workers capped at 5 req/sec.
• Retrying mechanism for reliability.

So this should work.
I need your suggestion on it.
Thanks

[eriq-augustine]

I'm not a huge fan of your plan here.
A lot of these details don't really seem necessary.
For example, limiting to 5 req/sec or 4 workers.
If we know how many files we want to download when we are starting the process, then we don't really need such specific limits.
We can just see how many cores we have available and how many files we want to download, and then just make sure we don't go over any limits.

For Edulinq projects, we always want to KISS first.

Maybe you should just code up something simple and we can start from there.

[Ayush-Vish]

Surely, I'll make a pass in the coming days.
My exams are going on.
I hope you understand.
Thanks.