Yesterday...

At work, we discovered that one of my co-workers made a mistake that cost most of us the day of meaningful work. Wednesday night, one of my co-workers was setting up an environment for us to test a new version of the code on for the next day.

At least, he thought he was in one of our test environments. In reality, he was in our production environment (the "live" environment where our customers work) and he deleted about 8000 of our customers. Not good.

So the day was spent by him, me, and our boss tracking down what happened, what our options were to fix it, and how quickly those options could be tried. In end, we got it back up and working but we lost about 60 user ids from the last time a backup was ran.

On the plus side, they are now backing up that data on a nightky basis to prevent that probleming from occurring again.