Thanks to a combination of a playful cat and a loose power cable one of my storage systems had an inadvertant removal of power and an outage. This was the storage that VCentre was running on and as seen in my previous blog recovering a corrupt vcentre, vcentre really does not like having it's power removed on the plus side this gave me the opportunity to try out a restore from the vcentre backups that I'd configured vcentre to send to a NAS and I'm glad I did as I learned quite a few lessons about how to successfully recover vcentre from such an incident.

Before I start with the details, I should point out that this is VCentre 6.7 - I've not tried out 7.0 yet but it's very much on the to do list.

Once I'd recovered power to the storage array I needed to have a look at the state of the VM's and I was able to log directly in to the ESXi host to see that most recovered just fine. VCentre, as we've seen before, really hates having the rug pulled from under it and so it was no surprise that it was not exactly in a heathly state. It would not even boot into VCentre due to corruption in the file system itself. I used https://kb.vmware.com/s/article/2149838 to get VCentre to at least partially boot but the postgresdb was in such a state that the only option was a restore. Thanks Cat!!

Now, me being me, I did not read any instructions on how to do this as I thought it would be a point and click excercise but it is a little bit more involved than that so if you want to see how to do a restore skip down to the part titled "Getting it all to work".

Not getting it to work

I thought I'd be able to kick off a restore by firing up the GUI and selecting restore from the menu but that doesn't work at all.

For some reason, performing the restore this way leads to a screen where no matter what you try you get an "unsupported protocol" message. I did track down a KB article which stated that restores from NFS and SMB were not supported which just seems strange to me.
The NAS that I have the backups hosted on does support FTP but getting FTP setup is quite involved and I thought that it would be easier to restore using another method so I thought I could perform the restore using the restore.job API.

And nope. It seems that VCentre doesn't support a restore from a system where the firstboot has run - i.e. any system that has previosuly actually worked! I did have a quick search to see if there was a flag I could reset or delete to remove this firstboot restore lockout but couldn't find anything so I gave up that approach.

Getting it all to work

After some searching, the process I followed to get the restore working was to delete the corrupted VCentre appliance and then deploy a fresh one but rather than select restore on the install screen I selected install in the same way as I would for deploying a new appliance.

The appliance install is split into two parts. For the restore you need to complete stage 1 only.
After that you select the restore option. Note that the restore path must be the full path down to the JSON file because if you select the root folder where the backups are sent then you will get a non-helpful error.
Do make sure you're using a file path down to a folder that looks like the screenshot below. The folders themselves are in date order so make sure you select the folder with the date you want to restore back to.

Once done you should see a screen looking like this. My backups are held on an NFS volume but I found it interesting that the restore shows it is using port 21.

This process took me about an hour as it was a mix of install and then restore. I am not sure why selecting restore from the installer doesn't go through this particular process but this method worked well and it was fully GUI driven. Once done the GUI started all the services without needing to do a reboot and VCentre was back.

In summary, the restore process is pretty easy and quite slick if you restore the components in the right order and the best right order appears to be to ignore the restore option on the installer and go for a stage 1 VCentre install and then select a restore.