SBS Exchange 2003 Corrupt ESE Database Recovery

A client ran into a problem with a RAID5 array losing a disk. There was a hotspare, but apparently some data corruption occurred. This caused problems with the ESE database used by Exchange. Recovery was relatively easy, but very time consuming.

I was able to recover the primary database without a problem, but the public folders were problematic. I kept getting data corruption errors like these when I tried to use the eseutil program to repair the damage:

"Operation terminated with error -1206 (JET_errDatabaseCorrupted, Non database file or corrupted db) after 8.47 seconds"

"Operation terminated with error -1209 (JET_errInvalidDatabaseVersion, Database engine is incompatible with database) after 9.32 seconds."

"Isinteg cannot initiate verification process.
Please review the log file for more information."

So, I had to do a restore from a tape backup and that created new issues...

Initially, the problem was identified by an error telling users of Outlook that the exchange server was down for repair and maintenance. Since it wasn't, this was disconcerting! A review of the Exchange System Manager showed the two message stores were dismounted and would not mount.

The basic idea of recovery was to:

1. First, do not harm: backup what we have
2. Recover using utilities
3. For what didn't work, recover from tape backups
4. Verify everything came back up
5. Backup again on a different tape we set aside

The details shown below are what I did to get this working. Don't try to follow them step-by-step in your situation as your situation will almost always be different. However, by seeing what I did, you may have a better idea about how to solve your problem! Your mileage may vary. Backup a lot. Don't make things worse. Don't guess.

Here are the details:

1. First, backup the files by hand:

> cd "\Program Files\Exchsrvr\MDBDATA"
> mkdir backup
> copy priv1.edb backup
> copy priv1.stm backup
> copy pub1.edb backup
> copy pub1.stm backup

This took a few hours as the edb files were 6gb and 20mb in size.

2. Try recovering using the eseutil program:

(this is still in the same directory as above)
> ..\bin\eseutil /p priv1.edb /i
> ..\bin\eseutil /p pub1.edb /i
(the pub1.edb attempt failed, but I continued recovering priv1.edb)

3. Move the log and checkpoint files to our backup directory:

> move *.chk backup
> move *.log backup

4. Optionally (recommended, but allow a few hours), you should make a backup of your recovered files before continuing.
5. Defrag the recovered edb (this takes awhile):

> ..\bin\eseutil /d priv1.edb
> ..\bin\eseutil /d pub1.edb

6. Mount the stores and dismount them using the System Manager. If you don't do this, you'll get an error about "cannot initiate verification process" when you run isinteg next. If they mount, it also tells you that the recovery was likely successful.
7. Run the integrity utility:

> ..\bin\isinteg -s servername -fix -test alltests
Databases for server server:
Only databases marked as Offline can be checked

Index  Status       Database-Name
Storage Group Name: First Storage Group
  1    Offline      Mailbox Store (SERVER)
  2    Offline      Public Folder Store (SERVER)
Enter a number to select a database or press Return to exit.
1
You have selected First Storage Group / Mailbox Store (SERVER).
Continue?(Y/N) y

When it has completed, repeat this for the public folder.

8. You should now be able to mount the stores and use Outlook to make sure users can access the system. Let everyone know that the recovery has been completed and have them verify that there hasn't been any significant data loss.
9. This only worked for me on the Mailbox Store, not the Public store. The Public store was corrupt beyond recovery. I decided to recover from the last successful tape. This client uses Veritas 9.0 for Small Business Server, so I fired that up and started to do the restore of just the Public folder. It was not successful.
10. I determined that the rights were incorrectly set for Administrator after I received an error that "The job failed with the following error: Unable to attach to one of the drives." The rights for administrator had to be set by going to the System Manager, open the folders until you get to the server you are working with, open "First Storage Group" (in my case), right click on "Public Folder Store", select Properties, select Security, click on the "Domain Admins" group, then make sure "Full control" is checked. Administrator had full rights, but this group didn't and it seemed to override the ability of the tape software to restore properly. Click OK.
11. Use the restore function of Veritas to restore the Public Store (only, in my case). I used the following options, since I just wanted to restore the public store:

[Destination - Exchange Redirection]
Uncheck "Redirect Exchange sets"
[Settings - Exchange]
Uncheck "automatically recreate user accounts and mailboxes"
Uncheck "exchange v5.5 Restore public folder"
Uncheck "exchange v5.5 Restore private mailboxes"
Check "No loss restore"
C:\TEMP (for temporary location--also make sure this location exists)
Check "Commit after restore completes"
Check "Mount database after restore"

This gave me another error that it couldn't restore. This was because the database was corrupt and dismounted and thus the agent couldn't start the restore. This becomes a "which came first, the chicken or the egg?" problem because I couldn't mount the database because it was corrupt and I couldn't restore it because it wasn't mounted...

12. I renamed the pub1.edb and pub1.stm files to pub1.edb.damaged and pub1.stm.damaged.
13. Using System Manager, try to mount the Public Store. This will give you a message about how "there isn't a database and would you like to create one? (but don't if you are going to do a restore)". Go ahead and create an empty public store.
14. Now try the restore and it should work.

This site was helpful in working thru some of the issues with isinteg and eseutil as was the Microsoft kb.