Where are you running from you little cron?!

A tale as long as time, the forgotten cron. During dev we had a cron that we wanted to put in prod right away and ended up putting on a server where it didn’t belong just to get things going. Days later we got the cron installed properly and decided to leave the other one running for a while since there was no harm for this to run twice. It has since become the forgotten cron; except! except that it emailed us at the same time the real one was emailing us. So twice a week we got this extra email that was formatted differently (using older code) and we didn’t bother to stop it. If you didn’t guess from my previous posts, or just from stereotypes of a programmer, I have a “slight” OCD issue. I decided to hunt that little cron down.

  1. Checked the headers of the email. A while ago we sent our mail directly using our internal smtp server. We used to see the server that originated the email. However, since we moved to Amazon AWS and started using SES that header no longer is available. Instead I found an SES message id.
  2. Grepped our exim relay logs. zgrep actually. The relay is one we connect to by all servers to send email wherever we want, therefore we can change the email backend / relay service with no hassle at all). And found the log file that I should look into.
  3. From the logfile found the server within our (60 machine) cluster, which I should have been able to guess really without doing all this 😛
  4. From there dug in all places where crons could be installed and found it! Goodbye you little cron.

PS. Initially my title was Where did you come from, you email!?, the new title sounds much cooler though 🙂