running the crawler

[tezoatlipoca]

I like the lemmy-explorer, https://lemmyverse.net/communities,
but Im having a difficult time running the crawler. Or rather, I can run it, but it just seems to sit there and not do anything.

I start redis, then
npm run start:init

that's it? wait? how long before keys and stuff start showing up in redis
I don't see anything happening in the /logs or the .pm2/logs....

Fundamentally all Im looking for is the json spat out by the crawler's out command to generate a flat list of threadiverse communities, but I don't see the crawler actually doing anything.

I could also be an idiot and I don't know what I should be looking at =D

[tgxn]

I put the data up in GitHub now :D
https://github.com/tgxn/lemmy-explorer/tree/main/frontend/public

I'll write up a brief guide on how to start and interact with the crawler shortly.

[tgxn]

Usage

Basic Usage

1. Start redis server (docker-compose up -d)

Redis server config is in lib/const.js

2. Start crawler (npm start)

This will use pm2 to start the crawler in the background.
You can use npm run logs to monitor the crawler.

3. Put some jobs into the queue (node index.js --init)

This will put some jobs into the queue based off the lists in lib/const.js

More Details Here:
https://github.com/tgxn/lemmy-explorer/blob/main/crawler/README.md