Explore scripts / automation ideas / etc from other 'data mining' style projects

[0xdevalias]

Looks like there are some neat ideas/scripts in @Le0Developer 's captcha-mining repo that could be useful to take inspiration from:

• https://github.com/Le0Developer/captcha-mining
  • https://github.com/Le0Developer/captcha-mining/blob/master/.github/workflows/ci.yml#L26-L30
    • Runs scripts/src/index.js on a schedule (cron: "22 * * * *")
  • https://github.com/Le0Developer/captcha-mining/blob/master/scripts/package.json#L14

    • simple-git

  • https://github.com/Le0Developer/captcha-mining/blob/master/scripts/src/index.ts#L4-L5
    • Runs updateTurnstile then updateHCaptcha
      • https://github.com/Le0Developer/captcha-mining/blob/master/scripts/src/turnstile.ts
        • Fetches the remote asset, checks version, calls cleanJavascript, writes files, calls tryAndPush, calls notify
      • https://github.com/Le0Developer/captcha-mining/blob/master/scripts/src/hcaptcha.ts
        • Fetches the remote asset, checks version, calls cleanJavascript, writes files, fetches additional assets, writes files, if extension is .js then calls cleanJavascript and writes more files, calls tryAndPush, calls notify
  • https://github.com/Le0Developer/captcha-mining/blob/master/scripts/src/lib/clean.ts
    • cleanJavascript: Basically just runs webcrack
  • https://github.com/Le0Developer/captcha-mining/blob/master/scripts/src/lib/git.ts
    • tryAndPush: uses simpleGit to pull, add, commit (but seemingly push is left to CI to do)
  • https://github.com/Le0Developer/captcha-mining/blob/master/scripts/src/lib/notifications.ts
    • notify: calls sendDiscordNotification, which makes a POST to process.env.DISCORD_WEBHOOK

---

From a quick skim, Cloudflare-Mining/Cloudflare-Datamining also looks like it would have some useful ideas/scripts/etc:

• https://cfdata.lol/
• https://github.com/Cloudflare-Mining/Cloudflare-Datamining
  • https://github.com/Cloudflare-Mining/Cloudflare-Datamining/tree/main/.github/workflows
  • https://github.com/Cloudflare-Mining/Cloudflare-Datamining/tree/main/scripts

See Also

• Automate checking for new builds with GitHub action/similar #8