🚨 Cron expressions + AI agents = 3 AM disaster (war story)

[jingchang0623-crypto]

So I just spent an entire night debugging why my AI agent's scheduled task was running at 3 AM instead of 3 PM. The culprit? A single character in a cron expression.

The setup:
I had an agent configured to run daily reports at 15:00 (3 PM). The cron expression looked fine. The code looked fine. Everything should have worked.

The disaster:
At 3:17 AM, my phone started buzzing. Then again at 3:18 AM. Then 3:19 AM. By 3:20 AM, I had 47 notification emails from my own agent, each one telling me about a report it had just generated.

I groggily opened my laptop, thinking someone had hacked my system. Nope — just my cron expression: 0 3 * * * instead of 0 15 * * *.

But here's where it gets worse. My agent had error handling. It had retries. So every time it "failed" (which it didn't — it was working exactly as programmed), it would retry. And log. And notify. And retry again.

The aftermath:

• 47 generated reports
• 3GB of log files
• My email provider temporarily flagged my account
• One very tired developer who learned that 3 and 15 are, in fact, different numbers

What I learned (the hard way):

1. Test cron expressions in isolation — Use a cron parser/validator before deploying
2. Add a "safety window" — For time-sensitive tasks, check if it's actually business hours before executing
3. Rate limiting isn't just for APIs — Your own agents can overwhelm you if you're not careful
4. Timezones are the enemy — America/New_York at 3 PM is not the same as UTC 3 PM

I wrote up the full story here if you want to feel better about your own mistakes: https://miaoquai.com/stories/cron-task-midnight-disaster.html

Anyone else have cron horror stories? Please tell me I'm not alone in this 😅

---

Bonus lesson: My agent now has a "human sanity check" mode. Before executing any scheduled task, it sends me a message: "About to run daily report. Looks good?" If I don't respond in 5 minutes, it proceeds. This one change has saved me from at least three similar disasters.

[moonbox3]

Please keep this on-topic for specific items related to SK. Thank you.