Inspect the causes for exceeding memory limit while sorting the courses in the uploader #99
Description
In the uploader, we have to sort all of the courses by their subject prefixes, course numbers, and catalog years before inserting them to the database. While uploading the data with new semester 26S, we ran into this problem in sorting:
12:39:28 Connected to MongoDB
12:39:28 Uploading courses.json ...
12:39:45 Sorting courses...
12:39:46 (QueryExceededMemoryLimitNoDiskUseAllowed) Executor error during find command: combinedDB.courses :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting.
panic: (QueryExceededMemoryLimitNoDiskUseAllowed) Executor error during find command: combinedDB.courses :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting.
Investigate the cause of this and fix it so that we can import data of new semester without having to cut off some of the old semesters in the future. Consider the use of disk user or external sorting as suggested in the terminal, or rewrite the way we can sort our data (maybe in chunks?).
Whoever working on this has to ask me for some additional resources because you likely have to test the insertion of data to our databases.