[one-users] One and MySQL deadlocks and primary key reuse

Thu Apr 10 07:53:50 PDT 2014

Hi,
please, see below.

I have tried to restart both oned and mysql server. The duplicate entry appeared right after the restart, then only deadlocks show up occasionally.

Duplicate entry for primary key:

Wed Apr  9 23:22:27 2014 [ONE][E]: SQL command was: INSERT INTO vm_monitoring (vmid, last_poll, body) VALUES (20479,1397078547,'<VM><ID>20479</ID><UID>7</UID><GID>102</GID><UNAME>jenkins-heap</UNAME><GNAME>HEAP</GNAME><NAME>HEAP-algernon-MAIL-1</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>1</GROUP_U><GROUP_M>1</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>1</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><LAST_POLL>1397078547</LAST_POLL><STATE>3</STATE><LCM_STATE>3</LCM_STATE><RESCHED>0</RESCHED><STIME>1397043832</STIME><ETIME>0</ETIME><DEPLOY_ID>one-20479</DEPLOY_ID><MEMORY>2097152</MEMORY><CPU>15</CPU><NET_TX>2236</NET_TX><NET_RX>231269297</NET_RX><TEMPLATE><AUTOMATIC_REQUIREMENTS><![CDATA[!(PUBLIC _CLOUD = YES)]]></AUTOMATIC_REQUIREMENTS><CONTEXT><ALIAS><![CDATA[HEAP-algernon-MAIL-1]]></ALIAS><DISK_ID><![CDATA[1]]></DISK_ID><DOMAIN><![CDATA[cz.intinfra.com]]></DOMAIN><ETH0_DNS><![CDATA[10.33.138.59 10.226.32.44 10.226.32.45]]></ETH0_DNS><ETH0_GATEWAY><![CDATA[10.230.16.1]]></ETH0_GATEWAY><ETH0_IP><![CDATA[10.230.21.96]]></ETH0_IP><ETH0_IP6><![CDATA[fdbe:e0be:e0bb:0:c0e0:aff:fee6:1560]]></ETH0_IP6><ETH0_MAC><![CDATA[be:e0:0a:e6:15:60]]></ETH0_MAC><ETH0_MASK><![CDATA[255.255.240.0]]></ETH0_MASK><ETH0_NETWORK><![CDATA[10.230.16.0]]></ETH0_NETWORK><NETWORK><![CDATA[YES]]></NETWORK><NTP><![CDATA[10.63.255.100]]></NTP><SSH_PUBLIC_KEY>SOME SSH KEYS</SSH_PUBLIC_KEY><TARGET><![CDATA[hdb]]></TARGET></CONTEXT><CPU><![CDATA[0.4]]></CPU><DISK><CACHE><![CDATA[writeback]]></CACHE><CLONE><![CDATA[YES]]></CLONE><CLONE_TARGET><![CDATA[SYSTEM]]></CLONE_TARGET><DATASTORE><![CDATA[Images 1]]></DATASTORE><DATASTORE_ID><![CDATA[101]]></DATASTORE_ID><DEV_PREFIX><![CDATA[hd]]></DEV_PREFIX><DISK_ID><![CDATA[0]]></DISK_ID><DRIVER><![CDATA[qcow2]]></DRIVER><IMAGE><![CDATA[IMAGE NAME]]></IMAGE><IMAGE_ID><![CDATA[37]]></IMAGE_ID><IMAGE_UNAME><![CDATA[oneadmin]]></IMAGE_UNAME><LN_TARGET><![CDATA[NONE]]></LN_TARGET><READONLY><![CDATA[NO]]></READONLY><SAVE><![CDATA[NO]]></SAVE><SIZE><![CDATA[36864]]></SIZE><SOURCE><![CDATA[/mnt/one_images_1/101/1cc62cdc44c465cf819c2642eb45377e]]></SOURCE><TARGET><![CDATA[hda]]></TARGET><TM_MAD><![CDATA[qcow2]]></TM_MAD><TYPE><![CDATA[FILE]]></TYPE></DISK><GRAPHICS><LISTEN><![CDATA[0.0.0.0]]></LISTEN><PORT><![CDATA[26379]]></PORT><TYPE><![CDATA[VNC]]></TYPE></GRAPHICS><MEMORY><![CDATA[2048]]></MEMORY><NIC><BRIDGE><![CDATA[brvirttraffic0]]></BRIDGE><IP><![CDATA[10.230.21.96]]></IP><IP6_LINK><![CDATA[fe80 ::c0e0:aff:fee6:1560]]></IP6_LINK><IP6_SITE><![CDATA[fdbe:e0be:e0bb:0:c0e0:aff:fee6:1560]]></IP6_SITE><MAC><![CDATA[be:e0:0a:e6:15:60]]></MAC><NETWORK><![CDATA[IPv6 10.230.18-19]]></NETWORK><NETWORK_ID><![CDATA[2]]></NETWORK_ID><NETWORK_UNAME><![CDATA[oneadmin]]></NETWORK_UNAME><NIC_ID><![CDATA[0]]></NIC_ID><VLAN><![CDATA[NO]]></VLAN></NIC><OS><ARCH><![CDATA[x86_64]]></ARCH><BOOT><![CDATA[hd]]></BOOT></OS><TEMPLATE_ID><![CDATA[58]]></TEMPLATE_ID><VCPU><![CDATA[2]]></VCPU><VMID><![CDATA[20479]]></VMID></TEMPLATE><USER_TEMPLATE><CLUSTER_100><![CDATA[100]]></CLUSTER_100><CLUSTER_100><![CDATA[100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100]]></CLUSTER_100><CLUSTER_101><![CDATA[101]]></CLUSTER_101><CLUSTER_101><![CDATA[101,101]]></CLUSTER_101><CLUSTER_102><![CDATA[102]]></CLUSTER_102><SCH ED_RANK><![CDATA[-RUNNING_VMS]]></SCHED_RANK><SCHED_REQUIREMENTS><![CDATA[CLUSTER_ID=\"101\"]]></SCHED_REQUIREMENTS></USER_TEMPLATE><HISTORY_RECORDS><HISTORY><OID>20479</OID><SEQ>0</SEQ><HOSTNAME>mess11-node</HOSTNAME><HID>15</HID><CID>101</CID><STIME>1397043920</STIME><ETIME>0</ETIME><VMMMAD>kvm</VMMMAD><VNMMAD>dummy</VNMMAD><TMMAD>ssh</TMMAD><DS_LOCATION>/mnt/datastores/</DS_LOCATION><DS_ID>109</DS_ID><PSTIME>1397043920</PSTIME><PETIME>1397043921</PETIME><RSTIME>1397043921</RSTIME><RETIME>0</RETIME><ESTIME>0</ESTIME><EETIME>0</EETIME><REASON>0</REASON><ACTION>0</ACTION></HISTORY></HISTORY_RECORDS></VM>'), error 1062 : Duplicate entry '20479-1397078547' for key 'PRIMARY'

Deadlocks:

Thu Apr 10 16:07:40 2014 [ONE][E]: SQL command was: REPLACE INTO datastore_pool (oid, name, body, uid, gid, owner_u, group_u, other_u, cid) VALUES (108,'System 3','<DATASTORE><ID>108</ID><UID>0</UID><GID>0</GID><UNAME>oneadmin</UNAME><GNAME>oneadmin</GNAME><NAME>System 3</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>1</GROUP_U><GROUP_M>1</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>1</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><DS_MAD>-</DS_MAD><TM_MAD>qcow2</TM_MAD><BASE_PATH>/mnt/one_system_3/108</BASE_PATH><TYPE>1</TYPE><DISK_TYPE>0</DISK_TYPE><CLUSTER_ID>100</CLUSTER_ID><CLUSTER>KVM cluster</CLUSTER><TOTAL_MB>551263</TOTAL_MB><FREE_MB>429572</FREE_MB><USED_MB>93492</USED_MB><IMAGES></IMAGES><TEMPLATE><SHARED><![CDATA[YES]]></SHARED><TM_MAD><![CDATA[qcow2]]></TM_MAD><TYPE><![CDATA[SYSTEM_DS]]></TYPE></TEMPLATE></DATASTORE>',0,0,1,1,1,100), error 1213 : Deadlock found when trying to get lock; try restarting transaction
Thu Apr 10 16:10:25 2014 [ImM][I]: --Mark--
Thu Apr 10 16:13:18 2014 [VMM][I]: --Mark--
Thu Apr 10 16:13:31 2014 [InM][I]: --Mark--
Thu Apr 10 16:17:10 2014 [ONE][E]: SQL command was: REPLACE INTO host_pool (oid, name, body, state, last_mon_time, uid, gid, owner_u, group_u, other_u, cid) VALUES (12,'mess09-node','<HOST><ID>12</ID><NAME>mess09-node</NAME><STATE>2</STATE><IM_MAD>kvm</IM_MAD><VM_MAD>kvm</VM_MAD><VN_MAD>dummy</VN_MAD><LAST_MON_TIME>1397139430</LAST_MON_TIME><CLUSTER_ID>100</CLUSTER_ID><CLUSTER>KVM cluster</CLUSTER><HOST_SHARE><DISK_USAGE>0</DISK_USAGE><MEM_USAGE>16777216</MEM_USAGE><CPU_USAGE>210</CPU_USAGE><MAX_DISK>100794</MAX_DISK><MAX_MEM>19717932</MAX_MEM><MAX_CPU>1600</MAX_CPU><FREE_DISK>84193</FREE_DISK><FREE_MEM>15324609</FREE_MEM><FREE_CPU>1588</FREE_CPU><USED_DISK>1911123</USED_DISK><USED_MEM>4393324</USED_MEM><USED_CPU>11</USED_CPU><RUNNING_VMS>6</RUNNING_VMS><DATASTORES><DS><FREE_MB><![CDATA[84193]]></FREE_MB><ID><![CDATA[109]]></ID><TOTAL_MB><![CDATA[100794]]></TOTAL_MB><USED_MB><![CDATA[1]]></USED_MB></DS><DS><FREE_MB><![CDATA[420666]]></FREE_MB><ID><![CDATA[110]]></ID><TOTAL_MB><![CDATA[2064246]]></TOTAL_MB><USED_MB><![CDATA[1]]></U SED_MB></DS></DATASTORES></HOST_SHARE><VMS><ID>4805</ID><ID>5152</ID><ID>12575</ID><ID>13260</ID><ID>13358</ID><ID>21240</ID></VMS><TEMPLATE><ARCH><![CDATA[x86_64]]></ARCH><CPUSPEED><![CDATA[2266]]></CPUSPEED><HOSTNAME><![CDATA[mess09]]></HOSTNAME><HYPERVISOR><![CDATA[kvm]]></HYPERVISOR><MODELNAME><![CDATA[Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz]]></MODELNAME><NETRX><![CDATA[136264617591]]></NETRX><NETTX><![CDATA[198812810614]]></NETTX><VERSION><![CDATA[4.4.0]]></VERSION></TEMPLATE></HOST>',2,1397139430,0,0,1,0,0,100), error 1213 : Deadlock found when trying to get lock; try restarting transaction
Thu Apr 10 16:19:46 2014 [ImM][I]: --Mark--
Thu Apr 10 16:21:10 2014 [ONE][E]: SQL command was: REPLACE INTO datastore_pool (oid, name, body, uid, gid, owner_u, group_u, other_u, cid) VALUES (108,'System 3','<DATASTORE><ID>108</ID><UID>0</UID><GID>0</GID><UNAME>oneadmin</UNAME><GNAME>oneadmin</GNAME><NAME>System 3</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>1</GROUP_U><GROUP_M>1</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>1</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><DS_MAD>-</DS_MAD><TM_MAD>qcow2</TM_MAD><BASE_PATH>/mnt/one_system_3/108</BASE_PATH><TYPE>1</TYPE><DISK_TYPE>0</DISK_TYPE><CLUSTER_ID>100</CLUSTER_ID><CLUSTER>KVM cluster</CLUSTER><TOTAL_MB>551263</TOTAL_MB><FREE_MB>429567</FREE_MB><USED_MB>93497</USED_MB><IMAGES></IMAGES><TEMPLATE><SHARED><![CDATA[YES]]></SHARED><TM_MAD><![CDATA[qcow2]]></TM_MAD><TYPE><![CDATA[SYSTEM_DS]]></TYPE></TEMPLATE></DATASTORE>',0,0,1,1,1,100), error 1213 : Deadlock found when trying to get lock; try restarting transaction

Ondra

On 04/10/2014 04:45 PM, Carlos Martín Sánchez wrote:
Hi,

Can you send the relevant section of oned.log? It will be useful to know the operations that oned was trying to do when this happens.

Regards

--
Carlos Martín, MSc
Project Engineer
OpenNebula - Flexible Enterprise Cloud Made Simple
www.OpenNebula.org<http://www.OpenNebula.org> | cmartin at opennebula.org<mailto:cmartin at opennebula.org> | @OpenNebula<http://twitter.com/opennebula>

On Wed, Apr 9, 2014 at 12:13 AM, Hamada, Ondrej <ondrej.hamada at acision.com<mailto:ondrej.hamada at acision.com>> wrote:
Hi everyone,

my opennebula 4.4 reports problems when issuing sql commands. It reports mainly:

"error 1213 : Deadlock found when trying to get lock; try restarting transaction"
and occasionally also:
"error 1062 : Duplicate entry '19608-1396968279' for key 'PRIMARY'"

I'm using mysql 5.5 as db. One seems to continue working, but I'm afraid I've missed something.

Do you have any idea what might be wrong and how to fix it?

Thank you in advance

Ondra

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

_______________________________________________
Users mailing list
Users at lists.opennebula.org<mailto:Users at lists.opennebula.org>
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org

--
Ondrej Hamada

Software Engineer

_________________________________________

Acision. Innovation. Assured.

www.acision.com<http://www.acision.com>

Holandská 5
639 00 Brno
Czech Republic

T: +420 531 017 040
F: +420 543 125 114
M: +420 732 558 013
E: ondrej.hamada at acision.com<mailto:ondrej.hamada at acision.com>

________________________________
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20140410/5070c00d/attachment-0001.htm>