Recently did a datastore migration of a Virtual Server to a faster 15K LUN.

Our client only has the VMWare Essentials Plus so we cannot do a live migration of the store.

Virtual Server was stuck at 99% and I could not do anything with the Server. I checked Snapshot manager and 1 snapshot did exist.

I checked the file size in the Datastore directly and the VM was about 210GB but the snapshot was 1GB so something had gone wrong previously.

First I checked we had a backup of the server (I had data but not Image as the image backup had been failing due to this “corrupt” snapshot)

I tried restarting the VMWARE Server service to confirm the data being received was accurate.

Logged back in and same problem.

I had made the problem slightly worse as I could no longer see the running task at 99% so I was completely blind as to what the Host/VM was doing.

Next I SSH to the host that was running the stuck VM (Host1 – in my case)

I found a list of tasks on the Host by running the following command:

vim-cmd vimsvc/task_list

I then run run the following command to find all the tasks running under all the VM’s:

vim-cmd vmsvc/getallvms

Output:

# vim-cmd vmsvc/getallvms
Vmid Name File Guest OS Version Annotation
1 CARLADM .252 [vMStore-LUN10] CARLADM .252/CARLADM .252.vmx winLonghorn64Guest vmx-07
2 CARBIAPP [vMStore-LUN14] Sharepoint2010/Sharepoint2010.vmx windows7Server64Guest vmx-08
3 WSS [vMStore-LUN13] WSS/WSS.vmx windows7_64Guest vmx-07
5 CARLSP10 [LUN-15K-2] CARLSP10/CARLSP01.vmx windows7Server64Guest vmx-08

On the output take note of the “vmid” column as we need this in the next step to identify the VM we wish to run the command against. (In my case it was 5 – CARLSP10)

Once you have the vmid you can run the command below to get all tasks associated with that VM:

vim-cmd vmsvc/get.tasklist <enter vmid number>

vim-cmd vmsvc/get.tasklist 5

Output:

(ManagedObjectReference) [
‘vim.Task:haTask-5-vim.vm.Snapshot.remove-314434234‘,
‘vim.Task:haTask-5-vim.VirtualMachine.powerOn-314436989’,
‘vim.Task:haTask-5-vim.VirtualMachine.powerOn-314437036’

Next you need to take note of the task identifier that you are interested in to be able to run the next command:

(In my case i was interested in the Stuck Snap Shot Remove ID Number: 314434234)

vim-cmd vmsvc/task_info <task identifier>

vim-cmd vimsvc/task_info 314434234

Output:

(vmodl.fault.ManagedObjectNotFound) {
dynamicType = <unset>,
faultCause = (vmodl.MethodFault) null,
obj = ‘vim.Task:314434234’,
msg = “The object has already been deleted or has not been completely created”,

The Msg output lead me to this:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1039326

To Resolve this I could not do as suggested which was remove from Inventory so instead I migrated all VM’s from Host 1 to Host 2 and so the only VM left was CARLSP10.

Still in the SSH Window I restarted the services but running the following command:

/sbin/services.sh restart

Please do so at own risk as this can cause damage/corruption to VM. I did this as I had multiple backups and that Snapshot was only a fraction of the size it should of been so damaged should be minimal if any.

Once services all restarted you can then go back to VMWARE GUI and then reconnect the host and try and power on the VM.

All the above worked for me and saved me alot of time waiting to see if that 99% would actually complete.

 

One thought on “VMWare ESXi Snapshot Stuck at 99%

  1. Thank you. That’s informative.
    Just notice that the task_info must be like this
    task_info haTask-5-vim.vm.Snapshot.remove-314434234 instead of simply the number.
    Cheers.

Leave a Reply

Your email address will not be published. Required fields are marked *