Impossible to restore backup to Intel RAID volume
My main PC is running an MSI X99 Raider motherboard with Intel RAID controller.
The boot drive is on a software RAID volume made up of two Sandisk Ultra II 480GB SSD - ie. one volume of 980GB.
The data drive is a Windows dynamic disk of 8 TB, made of two 4TB Seagate SATA HDDs.
I have been running backups with Acronis for years. I have 4 sets of backups.
1) Onto a NAS. Both volumes are backed up as part of this task. There is only one full backup, and then daily incrementals.
2) Onto a local 6TB SATA HDD #1, which I plug in to a removable SATA drive bay. Both volumes are backed as part of this task. There is only one full backup, and then very occasional incrementals.
3) Onto a local 6TB SATA HDD #2, which I plug in to a removable SATA drive bay. Both volumes are backed as part of this task. There is only one full backup, and then very occasional incrementals.
4) Onto the local data drive (ie. dynamic disk). Only the boot drive is backed up. There are regular full backups and daily incrementals based on maximum size.
Well, wouldn't you know it, a few days ago, my Windows boot drive crashed and burned. Could not figure out why.
I tried restoring backups onto the Intel RAID SSD volume. All attempts have failed so far. I am a software engineer who has been building his own PCs for 25 years. I certainly feel stumped that after 48 hours, I still haven't been able to restore my system to what it was.
Here are the things I tried
1) with the Linux rescue media, tried to restore backup from set #4 (dynamic disk) onto intel RAID volume. Recovery runs for a very variable time (more this), and then always finishes with a pop-up that says "Recover operation failed" at the end. I took a screenshot of it with my cell phone.
I have done this at least 5 times, from 3 different full boot drive backups from the dynamic disk (ie backup set #4).
The backup is about 170GB in size (single TIB file). The read rate on the dynamic disk is about 200MB/s, and the write rate on the RAID SSD array is well over 400MB/s. Thus, if the recovery is I/O bound, the restore should take about 170,000 / 200 = 850 seconds, or about 14 minutes. It certainly isn't CPU bound since I have an i7-5820k CPU.
In reality, the restore time during those various attempts varied greatly, from 1 hour to 2 hours to 6 hours... That is the first extremely strange thing.
If the drive was actually restored correctly, I could actually live with the time. The problem is that it isn't.
After rebooting the machine after each of those restores, Windows 10 either rebooted to a blue screen, or started doing disk repair that never completed (left it for hours). In none of those cases was I ever able to boot Windows 10 to the desktop. I also tried to boot Windows 10 from DVD media and initiate disk repair from there, but to no avail.
As an engineer, I suspected hardware failure with the Sandisk SSD that were part of my RAID volume.
I moved them to another machine with removable SATA drive bay. I upgraded the firmware to the latest. I ran a Sandisk SMART tool that told me there was 99% life remaining on each SSD. I even did a full test of the surface with h2testw that fills the disk with random data, and verifies it. Both SSDs tested fine.
I also ran memtest86 7.5 to verify all 32GB of RAM in the main PC (the one with the drive failure). Again, it all verified fine without any error, after 4 passes and over 6 hours.
So, I had to rule out failure of those components.
Here are some of the other restore attempts I made :
2) Using secondary PC, restored from backup set #2 above onto a single Sandisk SSD
Then put that SSD back in the original PC . That booted succesfully. But at that point, it wasn't a RAID volume anymore.
Running that working OS, installed the Windows 10 AIK . I created WinPE media. I added the Intel RAID driver to it.
I tested the boot media. It worked.
3) Using that WinPE boot media, I tried to restore backup set #4 (from the local dynamic disk).
a) One problem I ran into was that the local dynamic disk wasn't mounted, so I couldn't read the backup image to restore. No matter : I ran DISKPART and imported the foreign dynamic disk, and then ran true image again, all still from the WinPE media. I am not sure if this is intended in Trueimage or not, but it's certainly an inconvenience to have to do that, especially since DISKPART chooses a new seemingly random letter to mount the drive every time I boot the WinPE media. One time it was L:, another time M: .
b) Then, I was able to select one of the full backups. I selected the empty SSD RAID volume as the target.
To my surprise, the recovery actually claimed it succeeded. Except it completed in one about one minute, and didn't actually restore anything at all onto the RAID volume ! It didn't restore the MBR, ID, recovery partition - nothing !
This seems to be a major restore bug in Trueimage.
c) next, instead of trying to restore the whole disk from the backup, I decided to try to restore the OS partition with the NTFS file system. That seemed to work. The restore is in progress as I type. It has been going on for over a half hour, and it's saying there are 3 hours to go. I will let it complete overnight.
Even if it does restore, I will still have other partitions to restore and the MBR. The partitions may not be in the right order. Things may still not boot. I do have bootable Win10 media to attempt startup repair, but not sure it will really help.
I have been using Acronis True image since 2009, and upgraded nearly every version along the way. I currently have 5 licenses to True Image 2018 (many computers in the house). 2 of them are running Intel RAID boot volume, with different motherboards / CPU / chipsets. The backups always go just fine.
Is there anyway they can actually be restored successfully to a RAID volume, and not a single individual disk ?
I am really at my wits end with this one. Please assist.
Steve,
1) re: dynamic disk, just to be clear, I'm not trying to restore a dynamic disk here.
I'm only trying to restore a backup of the boot drive, which was originally a RAID volume, and not a dynamic disk. By definition the boot drive can never be a dynamic disk.
The only reason dynamic disk was mentioned in my post is because the backup I was trying to restore is located on a dynamic disk.
I was surprised that the WinPE media would not automatically mount the dynamic disk. I had to mount it manually, otherwise, I could not access my backup. I would say that's a bug in Acronis that should be addressed. Maybe there is an option in the WinPE media that can be set to automatically import dynamic disks when they are found ?
2) re: WinPE & RAID driver
My WinPE media already has the Intel driver, as my post mentioned.
Trueimage does see the RAID volume as a single drive.
The problem is that when I tell Trueimage to restore to this volume in disk mode, the restore completes nearly instantly (within one minute) and doesn't actually restore anything at all !
Julien,
Since you are using a software RAID which has failed you should consider that the RAID structure on the disks themselves has become corrupted. I have seen this in the past and the cure is to reset the RAID volume.
Pressing CTRL + I upon boot will get you into the Intel Option ROM where you can work with the RAID array. Once there select the Reset option to return the disks selected to non-RAID. Once you have done that, exit the utility and allow the board to reboot to the bios. Now power down the PC. From a cold start press CTRL + I again to enter the Intel Option ROM and reconfigure the RAID 0 option for the SSD disks. Once you have done that and you know the array creation has succeeded you can exit the utility, boot to the recovery media (WinPE for driver support), and run the recovery of the image to the newly created array.
This time the recovery should run in the expected time you mentioned in your OP. When that recovery is completed, shutdown the computer by closing the TI app window, and in the command prompt type wpeutil shutdown. Remove the recovery media from the PC then start the machine and once again enter the bios. Check the boot order to make sure the boot device is first in the order (Windows Boot Manger if UEFI boot, RAID volume if Legacy boot). Once that is confirmed, Save and Exit the bios and attempt booting the PC.
Julien, thank you for clarifying this issue. I do not have any systems that use either dynamic disks or use RAID so cannot advise you based on any personal experience in this area.
As far as I understand, it should be possible to do a disk mode recovery to your target RAID array given it is being seen as a single drive, so strange that the recovery completes very quickly and without doing what it should!
Have you checked the ATI Log messages after doing that type of recovery to see if there are any messages to suggest why that behaviour is happening? The log is volatile so only exists while you are booted from the media (unless you copy it to a permanent location)..
See forum topics:
Restore RAID 0 Backup To Same Machine Fails
How to restore Window 7 OS to raid 0 NVMe SSD
which may be of interest / help.
In reply to Julien,… by truwrikodrorow…
Enchantech,
Thanks. I forgot to mention it, but I had already done the array destroy / rebuild with CTRL I .
This was actually required since I had experimented with restoring to a single SSD instead of the RAID array volume.
Bad news - the restore of the single partition overnight also completed with "Recover operation failed" using the WinPE media. Afterwards,I can see that the SSD volume still shows as "unallocated".
I have checked the BIOS boot order as well and changed boot options multiple time. At this time it is set with UEFI with CD/DVD device first and HDD second. This is from a saved BIOS profile in the MSI motherboard options, actually.
Unfortunately, it looks like both the WinPE media and Linux media are simply unable to restore to the RAID volume on my system.
Is there anyone from Acronis reading this ? There seems to be a major incompatibility with my machine's software RAID. I don't want to drop back down to non-RAID.
In reply to Julien, thank you for… by truwrikodrorow…
Steve Smith,
Thanks for reminding me about the log messages. I just took screenshots of them.
I have two of them :
1) for the disk mode restore that completes instantly
2) for the partition mode restore that took 3 hours and ultimately failed
Both were with the WinPE media that included the proper Intel RAID driver .
I have never checked the log messages with the Linux media. However, the disk mode restore took hours (seemingly random number of hours) and ultimately still failed. I never tried partition mode restore with Linux media.
Julien, thanks for screen shots of the log entries. I am not sure what it means by 'A format/resize error' but suspect that this is directly related to the restore being actioned using sector-by-sector mode, so a question here is to ask if your original drive sizes are the same or larger than your new drive sizes?
The 'Failed to detect GRUB loader' is more of a puzzle as there shouldn't be any need to detect this unless you are either using GRUB instead of the Windows Boot Manager, or else have a dual-boot OS with a Linux OS alongside Windows?
I would recommend opening a Support Case direct with Acronis Support for this recovery issue and ask for their comments on these errors and the overall issue of recovering to your RAID array.
Your volume showing as Unallocated means that no formatting has been done to the volume. I would suggest using the Tools and Utilities Add New Disk option and look for your disk volume there. It will allow you format as GPT disk (required for UEFI boot) or MBR if not UEFI boot.
Do you know how the machine used to boot? By your statement that your CD\DVD is listed first then your HDD if you are using UEFI/GPT is incorrect. Your first in order if the machine does or did boot using UEFI would be Windows Boot Manager.
It sound to me like you may still have a RAID driver issue in your WinPE (maybe incomplete drivers). Can you boot the WinPE and use diskpart.exe from the command window to format and assign a drive letter to the RAID volume? If the answer is yes, use a43.exe to copy some files to the RAID volume as a test. You can bring up a43.exe from the command window:
cd \Program Files\Acronis\TrueImageHome\a43
a43.exe
Drag and drop is not supported in WinPE so you need to right click files in a43 and use copy and paste to copy files to the RAID volume. Sometimes a43 doesn't refresh right in WinPE so you may need to close it and reopen it to see your changes.
I would recommend you use the MVP Tool Steve suggested to build your WinPE. We have included the Intel IRST drivers to support Intel RAID. No need to add the drivers. When you build, all you need to do is say yes to adding "Custom Drivers". That will install the Intel RAID driver. If that doesn't work any better, you may have a strange problem. I can't offer any more help until I know you have a properly functioning WinPE with RAID support that allows you to manually add and delete files in the RAID volume.
Steve Smith, thanks, I'll be opening a case.
Enchantech,
The BIOS profile currently only shows UEFI hard disk because no Windows boot manager partition exists on any drive. It would show the Windows boot manager partition if it existed (and did, before everything went to hell).
I will try the Add new disk option as well.
Mustang,
I will try your suggestion of trying to partition format the RAID volume using DISKPART and copy some files to it. I'm more than comfortable with the command-line, no need for drag & drop.
As for the MVP tool, I would try it, but I think it requires a working system to generate a new WinPE boot media, right ? Kind of a chicken and egg issue. Unless I can generate it from another system with different hardware.
BTW, I tried another restore earlier today before heading to Napa. Was just restoring to a single non-RAID SSD in disk mode. It still took hours, but recovery completed successfully without errors.
Unfortunately, the result was still not a bootable Windows. I switched the BIOS back & forth between RAID and AHCI but neither worked. Startup repair did not work either.
This was the very same backup I had restored earlier in a different machine - when I pulled the SSD and put it in a removable drive bay, and restored a backup from one of the external SATA drives. After I moved the restored SSD back to the original system, it booted just fine on the first try without any issue.
As far as I can tell, Acronis is unable to succesfully do any kind of local restore of the boot drive, either to a RAID or non-RAID volume. Seems like some major incompatibility with my machine.
And of course, it's not possible to restore to a RAID volume from a different machine, because the software RAID will be different/non-existent on the other system.
The MVP Tool can be generated on a different computer with different hardware. You will need to have True Image installed and activated on the other computer in order for the MVP Tool to pick up the TI WinPE files and license code.
Ok, that confirms to me that RAID structure and formatting are the issue here.
When you broke the array did you choose delete or reset? Reset removes RAID structure from the disks where delete does not. Once the structure is correct then format will be successful.
A reset will return drives to single disk conditions allowing for a clean build of the array.
In reply to Mustang,… by truwrikodrorow…
FYI, RAID volume format last night worked OK. However, even a quick format took several minutes where it should normally be just seconds.
I was able to copy files just fine with A43 . Also copied a 170GB TIB from HD backup to the file system from the command line. It took way longer than it should have, over 2 hours. I went to sleep by then.
Mustang,
Thanks. I have many PCs in the house, all with TrueImage 2018 licenses, so I will try generating new media with the MVP tool.
Enchantech,
I had done delete rather than reset. Will try that too.
Just tried the MVP on the Win7 64-bit machine on which I have been typing these messages, which is on a KVM next to the other one that's hosed. This is the advanced version 18.2. Unfortunately, the tool didn't work at all.
If I select "Build from installed ADK", I then get two options "Build from WinRE (no ADK needed)" or "Exit). I guess the MVP tool does not work with the AIK, which is the only one compatible with Windows 7. ADK requires newer versions of the OS.
After I select this, text scrolls, and then some error occurs and the whole window closes. I just ran it again in a terminal window with admin privilege started manually to see what actually occurs. See picture attached.
I will try it on a Win10 machine next.
Julien, two comments here, first put the MVP tool folder in the root of a drive rather than as shown in your screen image. Second, you can use the Windows 10 ADK on your Windows 7 system - I have done the same on an older Samsung Netbook running Windows 7 32-bit on an Atom processor and have been able to create the MVP ATIPE USB media (32-bit version) and use it fine.
In reply to Julien, two comments here,… by truwrikodrorow…
Steve,
I just moved it to the root and chose to build without ADK this time. Unfortunately, the result was the same.
Just installed the Win10 ADK on Windows 7. MVP tool is working fine now. Just copied it to a flash drive now. Hope this will finally work.
BTW, there was a secondary disk controller in my main PC - based on Marvell, PCIe x4 . It was only there to use an external eSATA port multiplier enclosure I rarely use anymore due to it being SATA 2 only. Took out the controller of the machine last night before my latest attempt. Didn't seem to make a difference. I also disconnected the two 4TB HDDs (dynamic disk) after I copied the main TIB file to an external HDD. The main PC also has a built-in USB 3.0 card reader with 4 slots, and 3 optical drives, all of which were showing in WinPE. Not sure if any of this makes any difference. There are other add-on cards in my PC too. One is a Firewire 800 card. The other is a Hauppauge video capture. Have left those in place at this time too. Only other thing is the GPU which is required to boot on the MSI X99 motherboard since that doesn't have a built-in GPU.
Thank you for the screenshot of the MVP Tool. I can see more work is needed to support the build from WinRE option on Windows 7 systems. It does look doable.
I did a "reset" on the RAID array in the RAID BIOS before recreating it.
Then, booted from MVP media just fine. I tried restoring from a removable HDD. The restore proceeded very quickly, only about 25 minutes, which is the expected time for recovering from a single HDD. At the end, I was told to reboot. I was really hopeful that it would work. Unfortunately, the BIOS wouldn't see any partitions on the RAID volume. I went into the BIOS setup and there was no windows boot manager to select.
I booted from the MVP media again. TrueImage saw the RAID volume as "unallocated" in the "add new disk" tool as well as "drive cleanser".
Just checked with DISKPART, and sees one single reserved partition of 128MB at offset 17KB .
Strange that the 2 would disagree.
I'm stumped at this point and have no idea what to try next.
I think I would try the procedure outlined here https://forum.acronis.com/forum/acronis-true-image-2016-forum/guide-restoring-uefigpt-windows-system-new-disk-true-image-2016 to restore. You pre-partition the disk with the layout of the original system and restore one partition at a time instead of doing a full disk restore.
Question, did you run the Add New disk tool and complete the suggested steps to format the volume as GPT prior to performing the restore, sounds you did not.
If the above is true I recommend you run the Add New disk tool and allow it to format the volume as GPT them perform the restore.
If you still have the issue after that then my only thought would be that you could be seeing a very rare case of both disks having the same logical block defective which is a death sentence for a RAID volume structure.
The fix for that would be to replace one of the disks. The removed disk should be ok as a single disk or used in another array with another disk.
Enchantech,
No, I had not formatted the volume as GPT prior to the restore.
However, shouldn't Acronis already do this on its own when using disk mode recovery ? The first step it lists is "clean disk".
Anyway; I just used "add new disk" to do this manually, and started another restore. I will know in about another half hour if this helped.
As far as defective logical blocks, I don't think that theory is correct. As I recall, when I looked in the Sandisk SSD smart tool, there were no visible (non-remapped) defective blocks.
Thanks.
In reply to Enchantech,… by truwrikodrorow…
Well, recovery completed and even after setting the volume to GPT before restore, the result is the same, only one partition is actually restored.
I am not certain why you are getting the restart is required message. Have you tried the restart option to see if that will complete the process?
When you configure the restore process and you are in the what to recover part, do you see all partitions you would expect to see, EFI, Recovery, and OS System? Do you see above the partitions list the word Disk # with a check box next to it and is the box checked? If additional disks were attached to the mobo when the backup file was created those disks partitions may inadvertenly have been selected for backup and will also be listed here. Did you check to make certain that no other partions on other disks are selected for restore.
In reply to I am not certain why you are… by truwrikodrorow…
Enchantech,
Yes, I retried the restart option, and it doesn't work - results in an unbootable disk, and BIOS does not see any partition when trying to select something to boot from.
When I configure restore, I see only the one drive that was part of the backup, the boot drive which was on the RAID volume. And I do see the EFI partition, recovery, OS file system, and track 0. I can post a screenshot later. I have yet another recovery in attempt at the moment - booting the USB key WinPE media in non-EFI mode and doing the restore from there onto the RAID volume.
The previous attempt - using WinPE - booting the USB key WinPE media in non-EFI mode and doing the restore onto a single SSD with the BIOS set to AHCI mode instead of RAID - actually resulting in a bootable OS. I had change the BIOS back to RAID mode after the restore completed since the image on the backup had the RAID drivers configured (not sure if this was really required, but did anyway). So I'm hoping restoring in non-UEFI boot onto RAID volume will now work too. Will know very soon now that each restore attempts only takes 15-30 minutes with the WinPE media from MVP, as opposed to many hours with the other rescue media I was working with before.
Well, restoring from non-EFI mode did not help . Same result - only restored one partition, not the main file system. I took lots of screenshots and even a video.
It looks to me like the original disk may not have been GPT, but rather MBR, even though it contained an EFI partition.
I say this because I tried restoring items from the backup one at a time, and when I restore "track 0", TrueImage gives a message that it is "restoring MBR".
This would make sense since the volume is only 960GiB, a size which doesn't require a GPT drive.
BTW, some comments about the WinPE media :
- the task manager appears to be Win7 style, not Win10 . Is this because I built the media from a Win7 host with Win10 ADK ?
- also, clicking "resource monitor" in that task manager has no effect. I was hoping to monitor disk activity from there, but can't
- lastly, the keyboard layout defaulted to US QWERTY, but I am actually using a French AZERTY layout. I don't see any way to change it from within the WinPE media after booting. In the old DOS days, I would just type "KEYB FR" and be done with it. . No such luck here. Seems like the WinPE generator should prompt for keyboard layout choice. Or detect that the local keyboard layout is non-US, and install the same onto the WinPE boot media
Even setting the volume to MBR before restore didn't work. I still end up with an unbootable disk with just 1 partition.
Sorry to hear it's not working out. The Track 0 & MBR is the master boot record and partition information. Has nothing to do with GPT or MBR format.
If I come up with anything else I will post back.
I finally got it restored on RAID. The only thing that worked was to reinstall Windows from the install DVD.
I aborted the install mid-way after it had created all the partitions and file systems.
Then, I booted the WinPE media in UEFI mode, and restored the main file system from the backup. Went to watched a TV show, and finally when I came back, system booted correctly.
It seems that TrueImage 2018 just cannot handle restore to RAID volume in disk mode on my system. I have no idea why, but I think I have tried just about everything humanly possible. Huge amount of time wasted with main PC being down for about 96 hours.
The machine has only been back up for about 5 minutes, but I have already noticed that Spyder Pro 3 couldn't load the calibrated color profiles for my 3 monitors. No video hardware was changed since the restore, so I have to assume it's related to the disk ID showing different. Recalibration of 3 monitors will take at least an hour, so I wish I didn't have to go through that.
Thanks for reporting back. Strange that you had to do what you did to get it to work. I have recovered a raid 0 volume twice with no issue but one was on an ASUS board the other an ASRock. Both Intel controllers. I have read some reports on issues with raid on your MSI Raider board but nothing like your problem. Others say that bios version was problem.
Yeah, very strange. I have another Intel RAID box in the house with Asus board, Z170-AR. This is my Win7 box that I have been mostly messaging from in the last few days. It's also running dual SSDs in RAID. 128GB Crucial m4 x2 .This is running Skylake. If I was masochistic - or paid by Acronis to do QA - I would try a Trueimage restore of the RAID volume on it. But not now. I have wasted way too much time on this already to do it at this point. Need to get actual work done.
What you ended up doing is basically the same as I suggested earlier. You needed to create the partition structure first and restore the partitions one at a time.
Acronis has always had a hard time recreating partition structures. IE putting UEFI MSR partition in the wrong place. Last time I checked this still wasn't fixed.