Email scanner

[kaifcodec]

Discussion related to a new feature addition: Email scanning

[kaifcodec]

@VamatoHD
I wanted to open this discussion to propose a possible new feature for user-scanner: an email scanner.
The idea is to check whether an email address is used to create accounts on different platforms. It mostly falls under the OSINT category, but we can keep it scoped strictly as a scanner, similar to how username scanning works.
I just wanted to put this idea out here first and see what you think. If you’re interested in joining the discussion or collaborating on it, let me know and I can open a draft PR later.

[VamatoHD]

@kaifcodec This is a good addition to the project! Even though tools like holehe exist, they haven't been updated in years. Also, we have the infrastructure to easily add them: in each module we can simply add a new function.
It will probably take some effort but should be doable.

[kaifcodec]

Exactly, those tools haven’t been updated for years. We could also improve on them by making the scans more interactive and adding better error handling.

One thing I’m a bit cautious about is simply adding new functions to the existing modules. Email scanning feels more like a dedicated OSINT feature, and it would likely extend beyond just the current popular platforms we already support. Over time, we’d probably end up adding many more sites that rely on email-based sign-ups, which could make the existing module structure confusing, especially if we have to introduce modules that don’t really apply to username scanning.

An alternative could be to separate concerns more clearly, for example by moving the current modules into something like a username_scan/ directory and introducing a parallel email_scan/ directory. This way, both scanners can grow independently without mixing responsibilities.

That said, I’m not fully set on this approach yet. If you have a better idea or a different structure in mind, feel free to share, I’m open to discussing other options.

[VamatoHD]

Well, since we already have a pretty good structure with the Result and Printer, we could adapt them to support either username or email. The code is easily adaptable to adding email checks.

You propose something along these lines:

└── 📁user-scanner
    └── 📁tests
    └── 📁user_scanner
        └── 📁username_scan
            ├── 📁community
            ├── 📁creator
            ├── 📁dev
            ├── 📁donation
            ├── 📁gaming
            └── 📁...
        └── 📁email_scan
            ├── 📁community
            ├── 📁creator
            └── 📁...
        └── 📁cli
        └── 📁core
        └── 📁utils
        ├── __init__.py
        ├── __main__.py
        ├── config.json
        ├── version.json
    ├── .gitignore
    ├── CONTRIBUTING.md
    ├── ...

[kaifcodec]

Yeah, you got it. The project tree you suggested is also what I had in mind.

You’re also right about the Result and Printer classes being easily adaptable. I don’t think we’ll need to tweak them much, since email scanning is conceptually very similar to user_scan. It would still return Result objects, the main difference being that available would translate to something like email used etc.

[kaifcodec]

By the way, if you have any alternative project structure in mind, I’d definitely like to hear about it. My main goal is to keep the codebase very maintenance-friendly.

These kinds of tools are quite fragile and require constant maintenance, so the more readable and well-structured the code is, the easier it will be for newcomers to step in and fix issues, such as outdated API endpoints or broken checks.

[kaifcodec]

@VamatoHD I think it would be better if you work on the Result, Printer, and orchestrator modules to adapt them to the new changes, since you’re more familiar with those parts. By looking at the two new modules, you should be able to easily understand how these checks are going to work. #125

A few points from my side:

• Most likely we’ll need a new function in the orchestrator, similar to generic_validate (for example, something like generic_check).
• I noticed that most sites require CSRF tokens and session cookies to authorize the requests used for email availability checks, so we’ll probably need a helper function to capture cookies and reuse them in subsequent requests.
• I’ve already written the code using asyncio, since concurrent requests will be important for this email scanning feature. It may eventually include a wide range of sites, from less popular ones to very common platforms.
• Meanwhile, I’m documenting and saving how each site module works and how they perform email availability checks.

Let me know what you think, and we can iterate from there.

[kaifcodec]

Yeah it would be a good idea but we have to convert all the functions into async functions and the make_request() to use await.

[VamatoHD]

It takes a bit of work, but I think I can start it today.

[kaifcodec]

Ok, go with it.
Anyways did you think something about the draft pr? #125 .

[VamatoHD]

Can you work on the async? I'm a bit occupied with my school's project…
I saw #125, and I'll work on it when I'm available.
Also, I had an idea: since almost every check is a simple status check or a text in response.text, we could convert all of them into a single file like:

[
  {
    "name":"dockerhub",
    "category":"dev",
    "validate_method":"status",
    "status": 404
  },
  {
    "name":"tiktok",
    "category":"social",
    "validate_method":"search",
    "status": "statuscode\":10221"
  }
]

But this should be a topic for another discussion if you consider it relevant.

[kaifcodec]

Yeah, no issues. Also, an async implementation for username validation isn’t really required right now with around 48 sites to check, ThreadPoolExecutor handles it just fine. We can always revisit async later if needed.

For now, the main goal would be to update the core classes and add a new generic_check function for the email OSINT mode.

I’m a bit busy as well for similar reasons, so there’s absolutely no rush. Even small, incremental commits are totally fine we can work on this gradually over the next few days.

And about your suggestion i am replying to that on #120