Reduce memory usage by scanner

[yaroslav-dudar]

This is related to #603

My assumption that the main reason of this problem is located here

sslyze/sslyze/scanner/_jobs_worker_thread.py

Line 74 in 761892b

exception=e,

JobsWorkerThread pushes exception to queue which has circular reference. Exception (e variable) traceback contain frame which has exception by itself in frame locals.

So I added gc.collect in order to force garbage collection and malloc_trim as python don't release free memory back to kernel immediately

Test code

import os
import gc
import psutil
from sslyze import Scanner, ServerScanRequest, ServerNetworkLocation


def print_memory_used(msg):
    object_count = len(gc.get_objects())
    process = psutil.Process(os.getpid())
    memory_used = process.memory_info().rss
    print(f"{msg}: object count: {object_count}, memory used: {memory_used / 1024**2} MB")


def sslyze_scan(hostname, port):
    results = list()
    request = ServerScanRequest(ServerNetworkLocation(hostname=hostname, port=port))
    scanner = Scanner()
    scanner.queue_scans([request])
    results = list(scanner.get_results())

for i in range(1,5):
    print_memory_used(f"before run {i}")
    sslyze_scan("mozilla.com", 443)
    print_memory_used(f"after run {i}")

Results

before run 1: object count: 50005, memory used: 47.8828125 MB
after run 1: object count: 50420, memory used: 56.57421875 MB
before run 2: object count: 50420, memory used: 56.57421875 MB
after run 2: object count: 50420, memory used: 57.8046875 MB
before run 3: object count: 50420, memory used: 57.8046875 MB
after run 3: object count: 50420, memory used: 58.05859375 MB
before run 4: object count: 50420, memory used: 58.05859375 MB
after run 4: object count: 50420, memory used: 58.73828125 MB

Still minor memory grow, but significantly less than before