Improve Python Performance

[danguddemi]

We are at a stage where we've entered a beta stage that has much of our desired functionality implemented. Now it is important for us to try and optimize our Python code where possible and reduce our resource usage.

Questions to Ask:

1. Do we really need this module?
2. Why are we using this framework?
3. Is it worth the overhead?
4. Can we do this in a simpler way?

Possible Options:

• Reduce memory usage: Strings, Loops, Generators, Dicts/Sets (hashtables)
• Use built-in functions and libraries if possible
• Consider using Numpy for vector arrays
• Convert looped function calls into a local variable reference
• Lazy imports
• Change to C Compiled MySQL driver

Timing:

from timeit import default_timer as timer

start = timer()
# ...
end = timer()
print(end - start) # Time in seconds, e.g. 5.38091952400282

Profiling:

import cProfile
import re
cProfile.run('re.compile("foo|bar")')

https://docs.python.org/3/library/profile.html