Excavate for folders

[domwhewell-sage]

I think we've discussed this briefly on discord but so we note it down here aswell.

We've currently got excavate extracting stuff from HTTP_RESPONSE and RAW_TEXT but for folders that have been downloaded like git repos, apks, archives (soon) no juicy data is extracted from those.

We tried re-raising these files from within the folder then extracting RAW_TEXT but that really made it run slow so is probably infeasible.

We did discuss running grep over the folder which I think is a great idea something like
`grep -rnw ''
But I am unsure how todo something like that with the Yara rules currently in excavate.

[TheTechromancer]

There are a few tools that are very extremely fast at this; for example ripgrep, which is what vscode uses for ctrl+shift+F. Yara itself can also do this:

// /tmp/yara.rule
rule passwords {
    strings: $pass = /[a-z0-9]+passw[a-z0-9]+/
    condition: any of them
}

$ yara -rs /tmp/yara.rule /etc 2>/dev/null 
passwords /etc/security/pwhistory.conf
0x1d8:$pass: opasswd
0x1f8:$pass: opasswd
passwords /etc/rpc
0x120:$pass: yppasswdd
0x121:$pass: ppasswdd
0x131:$pass: yppasswd
0x132:$pass: ppasswd
0x5af:$pass: nispasswd
...

But since many of our excavate rules use a combination of yara and python, the best solution might be to break excavate out into its own tool. This would help us organize the code and give us a central repository for rules. It would also boost BBOT performance since right now regex is by far the biggest CPU hog.

If we did this, excavate would end up as a lightweight CLI tool and python library, with a collection of signatures, that could act on a string, file, or recursively on a folder. This functionality is super useful on its own (as a python import or grep -R on steroids), and we could keep any BBOT-specific logic in the excavate module, which would end up being a lot more concise and readable.

@liquidsec will likely have thoughts also

[TheTechromancer]

Relevant: #2832