I have not done this on SBS.... but I would assume it is the same as any other domain controller / Exchange restore.
Install SBS / give it the same IP / Name... Restore the files. On 2k3 you reboot in to AD recovery mode and push the system state, maybe exchange at the same time.
*My raw server Exchange recovery experience is limited
I normally do files and security first because the system state will try to restore the file shares in the registry which gets wacky if the files are missing.
Files > AD > Exchange I would think for SBS at least.
-edit- noted the reboot thing
Before you restore a system state, you need to make sure to install all the apps and services. Since it is a solo DC you may need to set up AD with the old domain name, make sure daemons like DNS etc are there and ready. You may need to run the Exchange install process (pretty sure SBS doesn't preinstall it, again been a real long time...) Make sure the disk structure is the same. If the old machine had "C Q S" as internal HDDs, your replacement will need to have "C Q S" etc.
Pretty sure it is not officially supported. The reboot loop is likely registry issues and a missing driver for the boot disk. System state restores the old systems registry which may not include the driver that the new system needs.
MS posted this:
Restoring Your Server
After a system failure or other disaster, you can restore your server from your latest successful backup. In the event of a software failure, you can restore to the same computer or you can restore to new hardware. You can also restore individual files and e-mail messages if they were permanently deleted.
WARNING: If you are restoring to a different computer than that which was originally running Windows Small Business Server 2003, make sure that the following items are the same on the original and the new computer:
*
SCSI controller. You can restore to a computer that has an IDE controller if the original computer had SCSI.
*
Motherboard chip set.
*
Number of processors.
*
Hard disk size. The new computer's hard disk should contain matching volumes the same size or larger than those on the original computer.
*
Drive letter of the boot partition.
We recommend restoring to the same brand and model of the original computer.