[one-users] Manually solving bug #1605 - VMs in CLEANUP state
Carlos Martín Sánchez
cmartin at opennebula.org
Fri Jan 25 02:17:48 PST 2013
I forgot to ask if you know what caused the VMs to get stuck in CLEANUP.
Were you doing onevm shutdown/cancel/delete/resubmit operations? Do you
have a consistent set of steps to reproduce the bug?
I'd appreciate if you could attach some of those VM individual logs, and
also a chunk of oned.log with the operations that led to the bug. Feel free
to email me off-list if you prefer to.
--
Carlos Martín, MSc
Project Engineer
OpenNebula - The Open-source Solution for Data Center Virtualization
www.OpenNebula.org | cmartin at opennebula.org |
@OpenNebula<http://twitter.com/opennebula><cmartin at opennebula.org>
On Fri, Jan 25, 2013 at 11:11 AM, Carlos Martín Sánchez <
cmartin at opennebula.org> wrote:
> Hi,
>
> The second fsck command refused to do anything because the backup file
> already exists, and then replaced the original DB because of bug 1564 [1].
> So you ended using the DB you had right after you manually deleted the
> rows, without any fsck fix.
>
> You can upgrade to 3.8.3 (I'd recommend it), or move the backup file
> elsewhere after each fsck command.
>
> Regards
>
> [1] http://opennebula.org/documentation:rel3.8:known_issues
>
> --
> Carlos Martín, MSc
> Project Engineer
> OpenNebula - The Open-source Solution for Data Center Virtualization
> www.OpenNebula.org | cmartin at opennebula.org | @OpenNebula<http://twitter.com/opennebula><cmartin at opennebula.org>
>
>
> On Thu, Jan 24, 2013 at 6:38 PM, Carlos Jiménez <
> cjimenez at eneotecnologia.com> wrote:
>
>> Hi Carlos,
>>
>> Sorry for the delay and thank you for your help.
>>
>> We've deleted the content of the table vm_pool and then executed onedb
>> fsck. This is the output:
>>
>> *MySQL dump stored in /var/lib/one/mysql_localhost_opennebula.sql
>> Use 'onedb restore' or restore the DB using the mysql command:
>> mysql -u user -h server -P port db_name < backup_file
>>
>> Image 4 RUNNING_VMS has 1 is 0
>> VM 29 is in Image 4 VM id list, but it should not
>> Image 4 has STATE USED_PERS is READY
>> Image 12 RUNNING_VMS has 1 is 0
>> VM 26 is in Image 12 VM id list, but it should not
>> Image 12 has STATE USED_PERS is READY
>> Image 7 RUNNING_VMS has 1 is 0
>> VM 25 is in Image 7 VM id list, but it should not
>> Image 7 has STATE USED_PERS is READY
>> Image 18 RUNNING_VMS has 1 is 0
>> VM 24 is in Image 18 VM id list, but it should not
>> Image 18 has STATE USED_PERS is READY
>> Image 21 RUNNING_VMS has 1 is 0
>> VM 44 is in Image 21 VM id list, but it should not
>> Image 21 has STATE USED_PERS is READY
>> Image 16 RUNNING_VMS has 1 is 0
>> VM 27 is in Image 16 VM id list, but it should not
>> Image 16 has STATE USED_PERS is READY
>> Image 15 RUNNING_VMS has 1 is 0
>> VM 30 is in Image 15 VM id list, but it should not
>> Image 15 has STATE USED_PERS is READY
>> Image 19 RUNNING_VMS has 1 is 0
>> VM 35 is in Image 19 VM id list, but it should not
>> Image 19 has STATE USED_PERS is READY
>> Image 23 RUNNING_VMS has 1 is 0
>> VM 47 is in Image 23 VM id list, but it should not
>> Image 23 has STATE USED_PERS is READY
>> VNet 0 has used lease 192.168.15.118 (VM 25) but it is free
>> VNet 0 has used lease 192.168.15.116 (VM 24) but it is free
>> VNet 0 has used lease 192.168.15.124 (VM 44) but it is free
>> VNet 0 has used lease 192.168.15.128 (VM 35) but it is free
>> VNet 0 has used lease 192.168.15.122 (VM 29) but it is free
>> VNet 0 has used lease 192.168.15.123 (VM 30) but it is free
>> VNet 0 has used lease 192.168.15.119 (VM 26) but it is free
>> VNet 0 has used lease 192.168.15.120 (VM 27) but it is free
>> VNet 0 has used lease 192.168.15.129 (VM 35) but it is free
>> VNet 0 has used lease 192.168.15.125 (VM 44) but it is free
>> VNet 0 has used lease 192.168.15.127 (VM 47) but it is free
>> VNet 0 TOTAL_LEASES has 28 is 17**
>>
>> Total errors found: 39*
>>
>>
>> We've executed the fsck again and this is the new output:
>>
>> *File /var/lib/one/mysql_localhost_opennebula.sql exists, backup
>> aborted. Use -f to overwrite.**
>> **Error running fsck version 3.8.1**
>> **The database will be restored**
>> **MySQL DB opennebula at localhost restored.*
>>
>> After that, we've started OpenNebula again and there are no VM instances,
>> but Status of most of the (persistent) Images is USED_PERS. It seems that
>> images table has not been correctly updated, isn't it? What could we do
>> (apart from deleting images table data) to correct it?
>>
>> Is there any way to avoid the "CLEANUP status" bug? I. e. something that
>> triggers the bug and we should avoid executing...
>>
>>
>> Thanks in advance,
>>
>> Carlos.
>>
>>
>>
>> On 12/18/2012 01:41 PM, Carlos Martín Sánchez wrote:
>>
>> Hi,
>>
>> You can stop opennebula, delete the VM rows from the DB, and then
>> execute the onedb fsck [1] command.
>>
>> Regards
>>
>> [1] http://opennebula.org/documentation:rel3.8:onedb
>> --
>> Carlos Martín, MSc
>> Project Engineer
>> OpenNebula - The Open-source Solution for Data Center Virtualization
>> www.OpenNebula.org | cmartin at opennebula.org | @OpenNebula<http://twitter.com/opennebula>
>>
>>
>>
>> On Mon, Dec 17, 2012 at 2:11 PM, Carlos Jiménez <
>> cjimenez at eneotecnologia.com> wrote:
>>
>>> Hi all,
>>>
>>> I have one machine with OpenNebula 3.8 and several hosts running KVM.
>>> Also, I have some VMs created.
>>> The problem is that after rebooting the OpenNebula server, several VMs
>>> are in RUNNING state but most of them are in CLEANUP state. Once in that
>>> state, it is not possible to delete or to move them to a different state.
>>> I've found a bug related to this issue:
>>> http://dev.opennebula.org/issues/1605 but it seems that we will have
>>> to wait up to release 4.0 to solve it.
>>>
>>> I've tried manually deleting a VM with onevm delete ID and later
>>> manually deleting the VM instance folder/files (logs and disk) without
>>> success, because the VM appears again in the list of VMs with the same
>>> CLEANUP state. Would it be possible to manually remove the affected entries
>>> in the db and then reinstantiate the VM/template again? Any other idea?
>>>
>>> I would need to get those VMs up and running.
>>>
>>>
>>> Thanks in advance.
>>>
>>>
>>> Carlos.
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.opennebula.org
>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opennebula.org
>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20130125/83b95893/attachment-0002.htm>
More information about the Users
mailing list