<div dir="ltr">Could you check the reason for the deadlock?<div><br></div><div>mysql> SHOW ENGINE INNODB STATUS\G<br></div><div><br></div><div>It probably refers to a next-lock issue, could you try setting:</div><div><br></div><div>mysql> SET GLOBAL TRANSACTION ISOLATION LEVEL READ COMMITTED;<br></div><div><br></div><div>And check if the deadlocks are gone?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 11, 2014 at 9:39 PM, Galimba <span dir="ltr"><<a href="mailto:galimba@gmail.com" target="_blank">galimba@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>uppon closer examination, we figured out that the deadlocks were not related to the image problem.<br></div>I was able to fix the uploading problem using this [1], but the random deadlock issue persists. I tried to onedb fsck, backup it and restore it... but it's all the same.<br><br><br><br>[1] -> <a href="http://docs.opennebula.org/4.6/release_notes/release_notes/known_issues.html" target="_blank">http://docs.opennebula.org/4.6/release_notes/release_notes/known_issues.html</a><br></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Thu, Sep 11, 2014 at 4:23 AM, Hamada, Ondrej <span dir="ltr"><<a href="mailto:ondrej.hamada@acision.com" target="_blank">ondrej.hamada@acision.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">no workarounds so far. But my mysql db is using default options and also runs in a virtual machine – so I suspect it just needs some tweaking. I’m going to
change few parameters during this week. I’ll post the results to mailing list if it works.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Ondra<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Galimba [mailto:<a href="mailto:galimba@gmail.com" target="_blank">galimba@gmail.com</a>]
<br>
<b>Sent:</b> Wednesday, September 10, 2014 10:04 PM<br>
<b>To:</b> <a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a>; <a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a>; Hamada, Ondrej<br>
<b>Subject:</b> Re: MySQL deadlock when trying to register a new vm image<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div><div><div>
<div>
<p class="MsoNormal">I found this [1] mail and I believe the deadlock issue is the same I'm having.<u></u><u></u></p>
</div>
<p class="MsoNormal">Any pointers on a workaround?<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Best regards<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">galimba<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><br>
[1] <a href="http://lists.opennebula.org/pipermail/users-opennebula.org/2014-April/044546.html" target="_blank">
http://lists.opennebula.org/pipermail/users-opennebula.org/2014-April/044546.html</a><u></u><u></u></p>
</div>
</div></div><div>
<p class="MsoNormal"><u></u> <u></u></p>
<div><div><div>
<p class="MsoNormal">On Tue, Sep 9, 2014 at 5:53 PM, Galimba <<a href="mailto:galimba@gmail.com" target="_blank">galimba@gmail.com</a>> wrote:<u></u><u></u></p>
</div></div><div><div><div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Greetings!<u></u><u></u></p>
</div>
<p class="MsoNormal">I've recently upgraded from v4.4 to 4.6. As part of the upgrade, we decided to create a new db using MySQL. The configurations ran smooth, all nodes and datastores are being monitored without issues but when I tried to upload a ttylinux
image for testing, sunstone didn't show the image uploaded on the list. As if it wasn' t even there. I ran the #oneimage list command, and there's no image there either. So I tried uploading again, while I checked the oned.log. I then tried to upload other
images.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I have OpenNebula front end installed on a host, several other nodes that are speciffic for running the VMs and a NAS with 20TB that holds the datastores. While I was uploading an ubuntu image (20GB), I checked
the transmition rate on the network and I was supprised to see that the traffic between my desktop (where I was uploading from) and the frontend was very heavy, but the datastore was not recieving the image I was uploading.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Interestingly, the log showed this error:<br>
<br>
Tue Sep 9 16:20:04 2014 [ONE][E]: SQL command was: REPLACE INTO datastore_pool (oid, name, body, uid, gid, owner_u, group_u, other_u, cid) VALUES (0,'system','<DATASTORE><ID>0</ID><UID>0</UID><GID>0</GID><UNAME>oneadmin</UNAME><GNAME>oneadmin</GNAME><NAME>system</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>1</GROUP_U><GROUP_M>0</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><DS_MAD><![CDATA[-]]></DS_MAD><TM_MAD><![CDATA[shared]]></TM_MAD><BASE_PATH><![CDATA[/var/lib/one//datastores/0]]></BASE_PATH><TYPE>1</TYPE><DISK_TYPE>0</DISK_TYPE><CLUSTER_ID>-1</CLUSTER_ID><CLUSTER></CLUSTER><TOTAL_MB>9461612</TOTAL_MB><FREE_MB>8956832</FREE_MB><USED_MB>1</USED_MB><IMAGES></IMAGES><TEMPLATE><BASE_PATH><![CDATA[/var/lib/one//datastores/]]></BASE_PATH><SHARED><![CDATA[YES]]></SHARED><TM_MAD><![CDATA[shared]]></TM_MAD><TYPE><![CDATA[SYSTEM_DS]]></TYPE></TEMPLATE></DATASTORE>',0,0,1,1,0,-1),
error 1213 : Deadlock found when trying to get lock; try restarting transaction<br>
<br>
/*******************************************************/<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I leave down here a copy of my oned.conf<br>
<br>
<u></u><u></u></p>
</div>
<p class="MsoNormal">/*******************************************************/<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">LOG = [<br>
system = "file",<br>
debug_level = 3<br>
]<br>
MONITORING_INTERVAL = 60<br>
MONITORING_THREADS = 50<br>
SCRIPTS_REMOTE_DIR=/var/tmp/one<br>
PORT = 2633<br>
DB = [ backend = "mysql",<br>
server = "localhost",<br>
port = 0,<br>
user = "oneadmin",<br>
passwd = "__________________",<br>
db_name = "opennebula" ]<br>
VNC_BASE_PORT = 5900<br>
FEDERATION = [<br>
MODE = "STANDALONE",<br>
ZONE_ID = 0,<br>
MASTER_ONED = ""<br>
]<br>
NETWORK_SIZE = 254<br>
MAC_PREFIX = "02:00"<br>
DATASTORE_CAPACITY_CHECK = "yes"<br>
DEFAULT_IMAGE_TYPE = "OS"<br>
DEFAULT_DEVICE_PREFIX = "hd"<br>
DEFAULT_CDROM_DEVICE_PREFIX = "vd"<br>
IM_MAD = [<br>
name = "collectd",<br>
executable = "collectd",<br>
arguments = "-p 4124 -f 5 -t 50 -i 20" ]<br>
IM_MAD = [<br>
name = "kvm",<br>
executable = "one_im_ssh",<br>
arguments = "-r 3 -t 15 kvm" ]<br>
VM_MAD = [<br>
name = "kvm",<br>
executable = "one_vmm_exec",<br>
arguments = "-t 15 -r 0 kvm",<br>
default = "vmm_exec/vmm_exec_kvm.conf",<br>
type = "kvm" ]<br>
TM_MAD = [<br>
executable = "one_tm",<br>
arguments = "-t 15 -d dummy,lvm,shared,fs_lvm,qcow2,ssh,vmfs,ceph"<br>
]<br>
DATASTORE_MAD = [<br>
executable = "one_datastore",<br>
arguments = "-t 15 -d dummy,fs,vmfs,lvm,ceph"<br>
]<br>
HM_MAD = [<br>
executable = "one_hm" ]<br>
AUTH_MAD = [<br>
executable = "one_auth_mad",<br>
authn = "ssh,x509,ldap,server_cipher,server_x509"<br>
]<br>
SESSION_EXPIRATION_TIME = 900<br>
DEFAULT_UMASK = 177<br>
VM_RESTRICTED_ATTR = "CONTEXT/FILES"<br>
VM_RESTRICTED_ATTR = "NIC/MAC"<br>
VM_RESTRICTED_ATTR = "NIC/VLAN_ID"<br>
VM_RESTRICTED_ATTR = "NIC/BRIDGE"<br>
IMAGE_RESTRICTED_ATTR = "SOURCE"<br>
INHERIT_DATASTORE_ATTR = "CEPH_HOST"<br>
INHERIT_DATASTORE_ATTR = "CEPH_SECRET"<br>
INHERIT_DATASTORE_ATTR = "CEPH_USER"<br>
INHERIT_DATASTORE_ATTR = "RBD_FORMAT"<br>
INHERIT_DATASTORE_ATTR = "GLUSTER_HOST"<br>
INHERIT_DATASTORE_ATTR = "GLUSTER_VOLUME"<br>
INHERIT_VNET_ATTR = "VLAN_TAGGED_ID"<br>
TM_MAD_CONF = [<br>
name = "dummy", ln_target = "NONE", clone_target = "SYSTEM", shared = "yes"<br>
]<br>
TM_MAD_CONF = [<br>
name = "lvm", ln_target = "NONE", clone_target = "SELF", shared = "yes"<br>
]<br>
TM_MAD_CONF = [<br>
name = "shared", ln_target = "NONE", clone_target = "SYSTEM", shared = "yes"<br>
]<br>
TM_MAD_CONF = [<br>
name = "fs_lvm", ln_target = "SYSTEM", clone_target = "SYSTEM", shared="yes"<br>
]<br>
TM_MAD_CONF = [<br>
name = "qcow2", ln_target = "NONE", clone_target = "SYSTEM", shared = "yes"<br>
]<br>
TM_MAD_CONF = [<br>
name = "ssh", ln_target = "SYSTEM", clone_target = "SYSTEM", shared = "no"<br>
]<br>
TM_MAD_CONF = [<br>
name = "vmfs", ln_target = "NONE", clone_target= "SYSTEM", shared = "yes"<br>
]<br>
TM_MAD_CONF = [<br>
name = "ceph", ln_target = "NONE", clone_target = "SELF", shared = "yes"<br>
]<br>
/**********************************************************/<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">best regards<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">galimba<u></u><u></u></span></p>
</div>
</div></div><div>
<div>
<p class="MsoNormal"><span style="color:#888888"><br clear="all">
<u></u><u></u></span></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888"><br>
-- <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="color:#888888;border:solid windowtext 1.0pt;padding:0in"><img alt="Image removed by sender." width="100" border="0" height="100"></span><span style="color:#888888"><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<br>
-- <u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="border:solid windowtext 1.0pt;padding:0in"><img alt="Image removed by sender." width="100" border="0" height="100"></span><u></u><u></u></p>
</div>
</div>
</div>
</div>
<hr>
<p class="MsoNormal"><span style="font-size:7.5pt;line-height:115%;font-family:"Arial","sans-serif";color:gray">This e-mail and any attachment
<span>is</span> for <span>authorised</span> use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained
or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.</span></p>
</div>
</blockquote></div><br><br clear="all"><br></div></div><span class="HOEnZb"><font color="#888888">-- <br><div dir="ltr"><img><br></div>
</font></span></div>
<br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div><div>-- <br></div></div>Ruben S. Montero, PhD<br>Project co-Lead and Chief Architect<div>OpenNebula - Flexible Enterprise Cloud Made Simple<br><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a> | @OpenNebula</div></div>
</div>