Memory load is way too high

[dbirman]

Describe the bug
Metadata portal is running at around 8GB ram currently, mostly driven by bad memory management in the data tables.

Expected behavior
Reduce memory to a single dataframe if possible, avoid copies. See if we can get back under the 6GB limit that is currently set for EC2. Not really any reason to be going over that even with 70k records.

[dbirman]

Tested out better memory management of DF but this actually doesn't help. Simplest solution is almost certainly to stream to a parquet file while being careful to allow GC and then stream back out of the file at runtime.