Hi again,<br><br>cpu pinning does not result in improved availability.<br>Executing test with just VCPU=1 and CPU=0.5 results on 55% of availability.<br>Before CPU pinning, there was 51% of availability.<br><br>Facts points to disk access, as sad on previous messages.<br>
There is a huge overload when squid is reading disks from VMs.<br><br><span id="result_box" class="short_text" lang="en"><span class="hps">I'll be</span> <span class="hps">grateful for any</span> <span class="hps">guidance.<br>
<br>Thank you in advance,<br><br>Erico.<br></span></span><br><br><div class="gmail_quote">2012/8/27 Erico Augusto Cavalcanti Guedes <span dir="ltr"><<a href="mailto:eacg@cin.ufpe.br" target="_blank">eacg@cin.ufpe.br</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br><br><div class="gmail_quote">2012/8/26 Ruben S. Montero <span dir="ltr"><<a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a>></span><div class="im">
<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>Hi<br>
<br>
If you want to try the cpu pinning suggested by Steven, simply add a<br>
RAW attribute in your VM template. Something similar to:<br>
<br>
RAW=[<br>
TYPE="kvm",<br>
DATA="<cputune><vcpupin vcpu=\"0\" cpuset=\"1\"/></cputune>" ]<br></div></blockquote></div><div><br>Added to vm template. Running an expirement right now to observe availability.<br>
</div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<br>
Cheers<br>
<br>
Ruben<br>
<br>
<br>
On Sun, Aug 26, 2012 at 3:27 AM, Steven C Timm <<a href="mailto:timm@fnal.gov" target="_blank">timm@fnal.gov</a>> wrote:<br>
</div><div><div>> I run high-availability squid servers on virtual machines although not yet<br>
> in OpenNebula.<br>
><br>
> It can be done with very high availability.<br>
><br>
> I am not familiar with Ubuntu Server 12.04 but if it has libvirt 0.9.7 or<br>
> better, and you are<br>
><br>
> Using KVM hypervisor, you should be able to use the cpu-pinning and<br>
> numa-aware features of libvirt to pin<br> <br></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>
><br>
> each virtual machine to a given physical cpu. That will beat the migration<br>
> issue you are seeing now.<br></div></div></blockquote></div><div><br>Done. migration processes was beaten, as you sad. Thank you.<br> </div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><div>
><br>
> With Xen hypervisor you can (and should) also pin.<br>
><br>
> I think if you beat the cpu and memory pinning problem you will be OK.<br>
><br>
><br>
><br>
> However, you did not say what network topology you are using for your<br>
> virtual machine, </div></div></blockquote></div><div><br>That is my vnet template:<br><br><span style="font-family:courier new,monospace">TYPE = RANGED</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">BRIDGE = br0</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">NETWORK_SIZE = C</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">NETWORK_ADDRESS = 192.168.15.0</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">NETMASK = 255.255.255.0</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">GATEWAY = 192.168.15.10</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">IP_START = 192.168.15.110</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">IP_END = 192.168.15.190</span><br>
<br>futhermore, there is one squid instance in execution on each VM, connected on a mesh architecture, with LRU replacement policy. Web-polygraph clients are configured to send requests on round-robin, to each of VMs (only three currently: 192.168.15.110-112). Web-polygraph server, that represents Internet, is running on 192.168.15.10, closing the topology.<br>
<br>Remember that when same test is performed on Physical Machines, availability is <span lang="en"><span>100%</span></span> <span lang="en"><span>invariably.</span></span><br>
<br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>and what kind of virtual network drivers,<br></div></div></blockquote>
</div><div><br>VMs are configured on VirtualBox, based on Ubuntu Server 12.04. <br>
Executing:<br>ps ax | grep kvm<br><br>on 192.168.15.110 cloud node:<br><br>/usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name one-609 -uuid a7115950-3f0e-0fab-ea20-3f829d835889 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-609.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -no-acpi -boot c -drive file=/srv3/cloud/one/var/datastores/0/609/disk.0,if=none,id=drive-ide0-0-0,format=qcow2 -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/srv3/cloud/one/var/datastores/0/609/disk.1,if=none,id=drive-ide0-1-1,format=raw -device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -netdev tap,fd=20,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=02:00:c0:a8:0f:70,bus=pci.0,addr=0x3 -usb -vnc <a href="http://0.0.0.0:609" target="_blank">0.0.0.0:609</a> -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4<br>
<br>network driver used on VMs is rtl8139.<br><br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br> </div></div>
</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><div>
><br>
> That is important too. Also—is your squid cache mostly disk-resident or<br>
> mostly RAM-resident? </div></div></blockquote></div><div><br>disk-resident (from squid.conf):<br>cache_mem 128 MB<br>cache_dir aufs /var/spool/squid3 1024 16 256<br><br>Note that disk space is reduced to 1024MB for testing purposes.<br>
</div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>If the former then the virtual disk drivers matter<br>
> too, a lot.<br></div></div></blockquote></div><div><br><span lang="en"><span>Some</span> <span>reading suggestions</span></span>? I'll appreciate a lot.<br><br>
One additional information that can be relevant. Monitoring VMs processes on D status, through command:<br><font size="1"><span style="font-family:courier new,monospace">ps -eo pid,tid,class,rtprio,psr,pcpu,stat,wchan:25,comm | grep D</span><br>
<span style="font-family:courier new,monospace"> PID TID CLS RTPRIO PSR %CPU STAT WCHAN COMMAND</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> 170 170 TS - 0 0.1 D sleep_on_page jbd2/sda1-8</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> 1645 1645 TS - 0 11.4 Dsl get_write_access squid3</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> 1647 1647 TS - 0 0.2 D get_write_access autExperiment.sh</span><span style="font-family:courier new,monospace"></span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> 1679 1679 TS - 0 0.8 D get_write_access autExperiment.sh</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> 1680 1680 TS - 0 0.5 D get_write_access autExperiment.sh</span><br style="font-family:courier new,monospace">
</font><br>It reveals that journally ext4 process(jdb2/sda1-8), squid and my monitoring scripting are on D status, always waiting on sleep_on_page and get_write_access kernel functions. I am researching the relevance/influence of these WCHANs. I performed tests without my monitoring scripts and availability is as bad as with them.<br>
<br>Thanks for Steve and Ruben, for your time and answers.<span class="HOEnZb"><font color="#888888"><br><br>Erico.<br><br> </font></span></div><div><div class="h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><div>
><br>
><br>
><br>
> Steve Timm<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> From: <a href="mailto:users-bounces@lists.opennebula.org" target="_blank">users-bounces@lists.opennebula.org</a><br>
> [mailto:<a href="mailto:users-bounces@lists.opennebula.org" target="_blank">users-bounces@lists.opennebula.org</a>] On Behalf Of Erico Augusto<br>
> Cavalcanti Guedes<br>
> Sent: Saturday, August 25, 2012 6:33 PM<br>
> To: <a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a><br>
> Subject: [one-users] Very high unavailable service<br>
><br>
><br>
><br>
> Dears,<br>
><br>
> I 'm running Squid Web Cache Proxy server on Ubuntu Server 12.04 VMs (kernel<br>
> 3.2.0-23-generic-pae), OpenNebula 3.4.<br>
> My private cloud is composed by one frontend and three nodes. VMs are<br>
> running on that 3 nodes, initially one by node.<br>
> Outside cloud, there are 2 hosts, one working as web clients and another as<br>
> web server, using Web Polygraph Benchmakring Tool.<br>
><br>
> The goal of tests is stress Squid cache running on VMs.<br>
> When same test is executed outside the cloud, using the three nodes as<br>
> Physical Machines, there are 100% of cache service availability.<br>
> Nevertheless, when cache service is provided by VMs, nothing better than 45%<br>
> of service availability is reached.<br>
> Web clients do not receive responses from squid when it is running on VMs in<br>
> 55% of the time.<br>
><br>
> I have monitored load average of VMs and PMs where VMs are been executed.<br>
> First load average field reaches 15 after some hours of tests on VMs, and 3<br>
> on physical machines.<br>
> Furthermore, there is a set of processes, called migration/X, that are<br>
> champions in CPU TIME when VMs are in execution. A sample:<br>
><br>
> top - 20:01:38 up 1 day, 3:36, 1 user, load average: 5.50, 5.47, 4.20<br>
><br>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ TIME COMMAND<br>
> 13 root RT 0 0 0 0 S 0 0.0 408:27.25 408:27<br>
> migration/2<br>
> 8 root RT 0 0 0 0 S 0 0.0 404:13.63 404:13<br>
> migration/1<br>
> 6 root RT 0 0 0 0 S 0 0.0 401:36.78 401:36<br>
> migration/0<br>
> 17 root RT 0 0 0 0 S 0 0.0 400:59.10 400:59<br>
> migration/3<br>
><br>
><br>
> It isn't possible to offer web cache service via VMs in the way the service<br>
> is behaving, with so small availability.<br>
><br>
> So, my questions:<br>
><br>
> 1. Does anybody has experienced a similar problem of unresponsive service?<br>
> (Whatever service).<br>
> 2. How to state the bootleneck that is overloading the system, so that it<br>
> can be minimized?<br>
><br>
> Thanks a lot,<br>
><br>
> Erico.<br>
><br>
><br>
</div></div><div>> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
> <a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
><br>
<br>
<br>
<br>
--<br>
</div>Ruben S. Montero, PhD<br>
Project co-Lead and Chief Architect<br>
OpenNebula - The Open Source Solution for Data Center Virtualization<br>
<a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a> | @OpenNebula<br>
</blockquote></div></div></div><br>
</blockquote></div><br>