What I learned on my Worst Day as a Network Admin

The guys at AppAssure asked me to write this blog post about how I came to buy their product. Basically I had a really bad day at work (I basically lost a critical disc and could not recover from my backups – not even my tape backups) and I was committed to never having a day like that again – that’s when I found Replay AppImage. If you have recently had a disaster like mine or if you want to make sure you don’t. you should give the AppAssure guys a shout or download a free trial of their software to check it out.

So here’s the story of my worst day at work:

When my company’s Microsoft Exchange Server failed at the end of the quarter, it could not have happened at a worse time. It began when the VP of Sales yelling “Email is down, and customers can’t send us their orders!” Then my Blackberry started going off, calls, emails, IMs —it was relentless. When I logged on to the Exchange Server, I found that some of my most critical mail stores were no longer mounted. When I tried to remount them, I received the ambiguous yet ominous JET-1601 JET_errRecordNotFound error message. I immediately connected to the replication server that runs at one of the company’s remote sites, only to find that I couldn’t mount those mail stores either.

When I called Microsoft, technicians prescribed the standard procedure of running Eseutil. They warned me, however, that the error message probably indicated a corruption problem deep within the database and that running Eseutil might result in cleaning the stores of all user data. “I took the leap, on the chance that it would be quicker than getting the restore process underway. Running Eseutil took hours, then failed with the even more ambiguous JET -1003 JET_errInvalidParameter.” At that point, I knew I HAD to go to the backup.

My company runs full backups every Saturday night and incremental backups the rest of the week. I started by recovering the most recent full backup, then applying the incrementals until I had the backup from the night before the failure. As you can imagine, the calls, emails, etc -- kept coming all the while I was copying the mail stores from my disk to disk backup—although they did taper off a bit after 11:00 P.M., when our west-coast office closed.

Once our data was back on the primary server, it was time to roll the logs and mount the database. However, when the logs were about 80 percent applied, they failed with the JET -501 JET_errLogFileCorrupt. At that point, Microsoft support could only suggest running Eseutil through my entire log chain, noting the corrupted log, deleting anything except log files from the log directory, and deleting the corrupted log and all the logs created thereafter. Then I could finally restart the log roll operation from scratch. This procedure took more than six hours. In the end, my company lost two days of email messages, and recovery took more than thirty hours. The cause turned out to be a problem with the RAID controller driver that had taken months to manifest itself after a previous server upgrade.

As you can imagine, executive management figured it cost the company about $50K so they definitely wanted to know what had happened and how it could have been prevented—and how it would be prevented from happening again. Let’s just say “wanted to know” means, if I didn’t have a good answer my name was going on the top of the next lay off list. I was seriously committed to finding a better recovery solution.

The Right Exchange Recovery Solution: Not Just Backups but Usable Data
After evaluating several potential solutions, of varying price ranges (from $199 to $100K), I liked AppAssure’s Replay AppImage right away. It’s a block-based imaging recovery solution that captures the entire Exchange server environment and supports recovery—anything from bare metal to an individual email message—in just a few clicks, simplifying the entire recovery process. What was totally cool was the Exchange “health checks” which made sure the data I was backing up was absolutely mountable. Before I made the recommendation to my boss, I wanted to be sure I was making the right choice and checked out a few of their customers: Jim Poehlman, Director of IT – Ubicom, needed to protect Exchange from user error and its databases from corruption. Jay Wessel, VP of Technology for the Boston Celtics, needed those capabilities in addition to being able to respond to legal and business discovery requests. Both said they found the solution they required in Replay AppImage and were happy with their choice.

In addition to capturing and validating your Exchange data, Replay AppImage employs a unique instant-replay capability that dramatically reduces volume recovery times from hours to minutes regardless of the data set size being recovered. After a rollback is initiated, the volume and storage groups are automatically and immediately mounted from the Replay AppImage server, providing users with access to email during the recovery process. Apparently they call it Live Replay; all I know is it saved me from being Dead Admin.
I’ve got
Replay AppImage up and running now and it lets me:
• Instantly roll back a server to a point in time before an outage occurred.
• Allow my users to access applications (including e-mail) during a live recovery.
• Recover entire applications from bare-metal in just a few clicks.
So here’s what I learned on my worst day as a Network Admin: You can have multiple copies of your data—on replicated servers, on disk, and on tape—but if you can’t mount the copies, they aren’t any good?