88: Add per-host config for bypassing robots file#91

grimsa wants to merge 1 commit intobejean:masterbejean/crawl-anywhere:masterfrom

grimsa:bugfix/88Copy head branch name to clipboard