[one-users] VM removed from one, but still running

[Fabian Zimmermann]

Hi,

just wondered, why one of my VMs had a completely broken filesystem.
After some research I found an old instance of my template (I'm using
persistent storage) running on one of my nodes. It looks like OpenNebula
didn't handle the broken auth correctly (caused by NFS-issues) and
marked the storage as free even if the VM is still running, isn't it?

--
Sat Jun 14 20:20:13 2014 [LCM][I]: New VM state is RUNNING
Sat Jun 28 14:11:07 2014 [LCM][I]: New VM state is UNKNOWN
Sat Jun 28 14:11:08 2014 [LCM][I]: New VM state is CLEANUP.
Sat Jun 28 14:11:08 2014 [VMM][I]: Driver command for 74 cancelled
Sat Jun 28 14:11:09 2014 [VMM][I]: Command execution fail:
/var/tmp/one/vmm/kvm/cancel 'one-74' 'node1' 74 node1
Sat Jun 28 14:11:09 2014 [VMM][I]: Warning: Permanently added
'node1,192.168.20.21' (ECDSA) to the list of known hosts.
Sat Jun 28 14:11:09 2014 [VMM][I]: Permission denied, please try again.
Sat Jun 28 14:11:09 2014 [VMM][I]: Permission denied, please try again.
Sat Jun 28 14:11:09 2014 [VMM][I]: Permission denied (publickey,password).
Sat Jun 28 14:11:09 2014 [VMM][I]: ExitSSHCode: 255
Sat Jun 28 14:11:09 2014 [VMM][E]: Error connecting to node1
Sat Jun 28 14:11:09 2014 [VMM][I]: Failed to execute virtualization
driver operation: cancel.
Sat Jun 28 14:11:09 2014 [VMM][I]: Command execution fail:
/var/tmp/one/vnm/dummy/clean
PFZNPjxJRD43NDwvSUQ+PERFUExPWV9JRD5vbmUtNzQ8L0RFUExPWV9JRD48VEVNUExBVEU+PE5JQz48QlJJREdFPjwhW0NEQVRBW2JyMF1dPjwvQlJJREdFPjxJUD48IVtDREFUQVs4OS4yMzguODkuMzZdXT48L0lQPjxJUDZfTElOSz48IVtDREFUQVtmZTgwOjoyMTY6M2VmZjpmZWI5OjU2ZTBdXT48L0lQNl9MSU5LPjxNQUM+PCFbQ0RBVEFbMDA6MTY6M2U6Yjk6NTY6ZTBdXT48L01BQz48TkVUV09SSz48IVtDREFUQVs4OS4yMzguODkuMzItMjldXT48L05FVFdPUks+PE5FVFdPUktfSUQ+PCFbQ0RBVEFbNF1dPjwvTkVUV09SS19JRD48TkVUV09SS19VTkFNRT48IVtDREFUQVtvbmVhZG1pbl1dPjwvTkVUV09SS19VTkFNRT48TklDX0lEPjwhW0NEQVRBWzBdXT48L05JQ19JRD48VkxBTj48IVtDREFUQVtOT11dPjwvVkxBTj48L05JQz48L1RFTVBMQVRFPjxISVNUT1JZX1JFQ09SRFM+PEhJU1RPUlk+PEhPU1ROQU1FPm5vZGUxPC9IT1NUTkFNRT48L0hJU1RPUlk+PC9ISVNUT1JZX1JFQ09SRFM+PC9WTT4=
Sat Jun 28 14:11:09 2014 [VMM][I]: Warning: Permanently added
'node1,192.168.20.21' (ECDSA) to the list of known hosts.
Sat Jun 28 14:11:09 2014 [VMM][I]: Permission denied, please try again.
Sat Jun 28 14:11:09 2014 [VMM][I]: Permission denied, please try again.
Sat Jun 28 14:11:09 2014 [VMM][I]: Permission denied (publickey,password).
Sat Jun 28 14:11:09 2014 [VMM][I]: ExitSSHCode: 255
Sat Jun 28 14:11:09 2014 [VMM][E]: Error connecting to node1
Sat Jun 28 14:11:09 2014 [VMM][I]: Failed to execute network driver
operation: clean.
Sat Jun 28 14:11:09 2014 [VMM][I]: Successfully execute transfer manager
driver operation: tm_delete.
Sat Jun 28 14:11:09 2014 [VMM][I]: Command execution fail:
/var/lib/one/remotes/tm/shared/delete
node1:/var/lib/one//datastores/0/74 74 0
Sat Jun 28 14:11:09 2014 [VMM][I]: delete: Deleting
/var/lib/one/datastores/0/74
Sat Jun 28 14:11:09 2014 [VMM][E]: delete: Command "[ -e
"/var/lib/one/datastores/0/74" ] || exit 0
Sat Jun 28 14:11:09 2014 [VMM][I]:
Sat Jun 28 14:11:09 2014 [VMM][I]: times=10
Sat Jun 28 14:11:09 2014 [VMM][I]: function="rm -rf
/var/lib/one/datastores/0/74"
Sat Jun 28 14:11:09 2014 [VMM][I]:
Sat Jun 28 14:11:09 2014 [VMM][I]: count=1
Sat Jun 28 14:11:09 2014 [VMM][I]:
Sat Jun 28 14:11:09 2014 [VMM][I]: ret=$($function)
Sat Jun 28 14:11:09 2014 [VMM][I]: error=$?
Sat Jun 28 14:11:09 2014 [VMM][I]:
Sat Jun 28 14:11:09 2014 [VMM][I]: while [ $count -lt $times -a "$error"
!= "0" ]; do
Sat Jun 28 14:11:09 2014 [VMM][I]: sleep 1
Sat Jun 28 14:11:09 2014 [VMM][I]: count=$(( $count + 1 ))
Sat Jun 28 14:11:09 2014 [VMM][I]: ret=$($function)
Sat Jun 28 14:11:09 2014 [VMM][I]: error=$?
Sat Jun 28 14:11:09 2014 [VMM][I]: done
Sat Jun 28 14:11:09 2014 [VMM][I]:
Sat Jun 28 14:11:09 2014 [VMM][I]: [ "x$error" = "x0" ]" failed:
Warning: Permanently added 'node1,192.168.20.21' (ECDSA) to the list of
known hosts.
Sat Jun 28 14:11:09 2014 [VMM][I]: Permission denied, please try again.
Sat Jun 28 14:11:09 2014 [VMM][I]: Permission denied, please try again.
Sat Jun 28 14:11:09 2014 [VMM][I]: Permission denied (publickey,password).
Sat Jun 28 14:11:09 2014 [VMM][E]: Error deleting
/var/lib/one/datastores/0/74
Sat Jun 28 14:11:09 2014 [VMM][I]: ExitCode: 255
Sat Jun 28 14:11:09 2014 [VMM][I]: Failed to execute transfer manager
driver operation: tm_delete.
Sat Jun 28 14:11:09 2014 [VMM][I]: Host successfully cleaned.
Sat Jun 28 14:11:09 2014 [DiM][I]: New VM state is PENDING
Sat Jun 28 14:11:33 2014 [DiM][I]: New VM state is ACTIVE.
Sat Jun 28 14:11:33 2014 [LCM][I]: New VM state is PROLOG.
Sat Jun 28 14:32:57 2014 [LCM][I]: New VM state is CLEANUP.
Sat Jun 28 14:32:57 2014 [DiM][I]: New VM state is DONE
Sat Jun 28 14:32:57 2014 [TM][W]: Ignored: LOG I 74 Driver command for
74 cancelled
Sat Jun 28 14:32:59 2014 [TM][W]: Ignored: TRANSFER SUCCESS 74 -

--

Fabian