spider args can appear in spider logs - how can these be masked out?

[simonsdave]

Important to note this not just for spider logs, it's also for chromedriver logs.

This might be useful in finding a solution for spider logs.

The following code could be modified to redact all sensitive data before calling crawl_response.add_debug().

if self.spider_log_file:
    crawl_response.add_debug('spiderLog', self.spider_log_file)

if self.chromedriver_log_file:
    crawl_response.add_debug('chromeDriverLog', self.chromedriver_log_file)

This now makes me wonder if we should be allowing chromedriver logs in production environments. Think we can use filters to eliminate sensitive data being written to spider logs but don't see how to do that for chromedriver logs.