Fetch more jobs messages, add diagnostics for job board startup (#435)

* Lots more details around ops and debugging production

* Extend message fetch to 90 days

* Don't limit how many messages get fetched

* Debug logging, why aren't we getting as many HIRING posts as expected

Also it appears the FORHIRE posts aren't getting cleaned up as intended

Author: vcarl

README.md

@@ -12,8 +12,8 @@ See [the contributing guide](./CONTRIBUTING.md) for specific instructions.
 
 You must have a working `.env` file that enables you to run the bot locally. With the values in `.env`, create a `reactibot-env` secret in your local minikube cluster. Everything must be provided a value, but it's okay if some of them are just random strings.
 
-```sh
-kubectl create secret generic reactibot-env \
+```bash
+$ kubectl create secret generic reactibot-env \
   --from-literal=DISCORD_HASH= \
   --from-literal=DISCORD_PUBLIC_KEY= \
   --from-literal=DISCORD_APP_ID= \
@@ -31,29 +31,29 @@ Set up kubectl and minikube locally. It's kinda hard.
 
 Start up a local Docker image registry.
 
-```sh
-docker run -d -p 5000:5000 --name registry registry:2.7
+```bash
+$ docker run -d -p 5000:5000 --name registry registry:2.7
 ```
 
 `-d` means this will run in "detached mode", so it will exit without logs after pulling required images and starting. You can view logs for it with `docker logs -f registry`.
 
 Create a file, `k8s-context`, in the project root, alongside the Dockerfile, with an IMAGE variable for kubectl to use.
 
-```sh
-echo IMAGE=reactibot:latest > k8s-context
+```bash
+$ echo IMAGE=reactibot:latest > k8s-context
 ```
 
 Run a docker build and tag it. We need to retrieve the image ID of the build we run, which complicates the command.
 
-```sh
-docker build . -t reactibot
-docker tag $(docker images reactibot:latest | tr -s ' ' | cut -f3 -d' ' | tail -n 1) localhost:5000/reactibot
+```bash
+$ docker build . -t reactibot
+$ docker tag $(docker images reactibot:latest | tr -s ' ' | cut -f3 -d' ' | tail -n 1) localhost:5000/reactibot
 ```
 
 Run a local deploy.
 
-```sh
-kubectl apply -k .
+```bash
+$ kubectl apply -k .
 ```
 
 If it doesn't deploy correctly (e.g. `kubectl get pods` shows a status other than success), you can debug it with `kubectl describe pod reactibot-deployment`
@@ -64,28 +64,79 @@ I actually couldn't get a local registry working so I fell back on using ghcr.io
 
 [Create a Personal Access Token (Classic)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic) and log in to ghcr.io. Use the PAT(C) as your password.
 
-```sh
-docker login ghcr.io
+```bash
+$ docker login ghcr.io
 ```
 
 Create a file, `k8s-context`, in the project root, alongside the Dockerfile, with an IMAGE variable for kubectl to use.
 
-```sh
-echo IMAGE=ghcr.io/<your gh>/reactibot:test > k8s-context
+```bash
+$ echo IMAGE=ghcr.io/<your gh>/reactibot:test > k8s-context
 ```
 
 Run a docker build, tag it, and push to the registry. We need to retrieve the image ID of the build we run, which complicates the command.
 
-```sh
-docker build . -t <your gh>/reactibot:test
-docker tag $(docker images <your gh>/reactibot:test | tr -s ' ' | cut -f3 -d' ' | tail -n 1) ghcr.io/<your gh>/reactibot:test
-docker push ghcr.io/<your gh>/reactibot:test
+```bash
+$ docker build . -t <your gh>/reactibot:test
+$ docker tag $(docker images <your gh>/reactibot:test | tr -s ' ' | cut -f3 -d' ' | tail -n 1) ghcr.io/<your gh>/reactibot:test
+$ docker push ghcr.io/<your gh>/reactibot:test
 ```
 
 Run a local deploy.
 
-```sh
-kubectl apply -k .
+```bash
+$ kubectl apply -k .
 ```
 
 If it doesn't deploy correctly (e.g. `kubectl get pods` shows a status other than success), you can debug it with `kubectl describe pod reactibot-deployment`
+
+# Inspecting the production operations
+
+```bash
+$ kubectl get ingress
+NAME                CLASS   HOSTS                 ADDRESS   PORTS     AGE
+mod-bot-ingress     nginx   euno.reactiflux.com   ….….….…   80, 443   …
+reactibot-ingress   nginx   api.reactiflux.com    ….….….…   80, 443   …
+
+$ kubectl get svc
+NAME                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
+kubernetes          ClusterIP   10.….….…     <none>        443/TCP   …
+mod-bot-service     ClusterIP   10.….….…     <none>        80/TCP    …
+reactibot-service   ClusterIP   10.….….…     <none>        80/TCP    …
+
+$ kubectl get pods --all-namespaces
+NAMESPACE        NAME                          READY   STATUS    RESTARTS   AGE
+cert-manager     cert-manager-…-…-…            1/1     Running   0          …
+cert-manager     cert-manager-…-…              1/1     Running   0          …
+cert-manager     cert-manager-webhook-…-…      1/1     Running   0          …
+default          mod-bot-set-0                 1/1     Running   0          …
+default          reactibot-deployment-…-…      1/1     Running   0          …
+ingress-nginx    ingress-nginx-controller-…-…  1/1     Running   0          …
+```
+
+## Useful diagnostic references
+
+```bash
+# Create a debug pod
+$ kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot -- /bin/bash
+
+# Then from inside the pod:
+curl http://mod-bot-service
+curl http://reactibot-service
+
+# Check service endpoints
+# This should show the IP addresses of your pods. If empty, the service isn't finding the pods.
+$ kubectl get endpoints mod-bot-service reactibot-service
+NAME                ENDPOINTS           AGE
+mod-bot-service     10.244.0.231:3000   58d
+reactibot-service   10.244.0.244:3000   6d4h
+
+# View the logs for the ingress controller
+$ kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=100
+
+# Check if the pods are actually listening on port 3000
+# mod-bot, a Stateful Set
+$ kubectl exec -it mod-bot-set-0 -- netstat -tulpn | grep 3000
+# reactibot, a Deployment
+$ kubectl exec -it $(kubectl get pod -l app=reactibot -o jsonpath='{.items[0].metadata.name}') -- netstat -tulpn | grep 3000
+```

src/features/jobs-moderation/job-mod-helpers.ts

@@ -85,25 +85,35 @@ export const getJobPosts = () => {
   };
 };
 
-const DAYS_OF_POSTS = 30;
+const DAYS_OF_POSTS = 90;
 
 export const loadJobs = async (bot: Client, channel: TextChannel) => {
   const now = new Date();
 
   let oldestMessage: StoredMessage | undefined;
 
   // Iteratively add all messages that are less than DAYS_OF_POSTS days old.
-  // Fetch by 10 messages at a time, paging through the channel history.
   while (
     !oldestMessage ||
     differenceInDays(now, oldestMessage.createdAt) < DAYS_OF_POSTS
   ) {
-    const newMessages: StoredMessage[] = (
-      await channel.messages.fetch({
-        limit: 10,
-        ...(oldestMessage ? { after: oldestMessage.message.id } : {}),
-      })
-    )
+    const messages = await channel.messages.fetch({
+      ...(oldestMessage ? { after: oldestMessage.message.id } : {}),
+    });
+    console.log(
+      "[DEBUG] loadJobs()",
+      `Oldest message: ${oldestMessage ? oldestMessage.createdAt : "none"}.`,
+      "Just fetched",
+      messages.size,
+      "messages",
+      messages
+        .map(
+          (m) =>
+            `${m.author.username}, ${differenceInDays(now, m.createdAt)} days old`,
+        )
+        .join("; "),
+    );
+    const newMessages: StoredMessage[] = messages
       // Convert fetched messages to be stored in the cache
       .map((message) => {
         const { tags, description } = parseContent(message.content)[0];
@@ -151,7 +161,7 @@ export const deleteAgedPosts = async () => {
   console.log(
     `[INFO] deleteAgedPosts() ${
       jobBoardMessageCache.forHire.length
-    } forhire posts. max age is ${FORHIRE_AGE_LIMIT} JSON: \`${JSON.stringify(
+    } forhire posts. max age is ${FORHIRE_AGE_LIMIT} hours JSON: \`${JSON.stringify(
       jobBoardMessageCache.forHire.map(({ message, ...p }) => ({
         ...p,
         hoursOld: differenceInHours(new Date(), p.createdAt),