Hi<br><br>Thanks for the reply. I agree that a VM will not be allowed to use more than what has been specified in its template during creation but I was referring to a scenario where memory available on the host is over committed. I guess the problem here is with allowing multiple VMs to be dispatched on to a single host in one schedule cycle.<br>
<br><br><div class="gmail_quote">On 11 November 2010 14:47, Ruben S. Montero <span dir="ltr"><<a href="mailto:rubensm@dacya.ucm.es" target="_blank">rubensm@dacya.ucm.es</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Hi,<br>
<br>
Regarding capacity (CPU, Memory), it is updated every time a VM is<br>
submitted, so no overcommitment is possible (beyond that specified by<br>
the CPU attribute in the VM template). This also works in 1.4<br>
<br>
Cheers<br>
<br>
Ruben<br>
<div><div></div><div><br>
On Thu, Nov 11, 2010 at 10:08 AM, <<a href="mailto:opennebula@nerling.ch" target="_blank">opennebula@nerling.ch</a>> wrote:<br>
> Hallo Shashank.<br>
> I'm having the same problem in 1.4.<br>
> You must workaround it by youself, so instead of using onevm directly, use<br>
> it with a wrapper script who looks and waits for deploy if a VM is pending.<br>
> I hope this behaviour is fixed on 2.0 (Hallo developers??)<br>
><br>
> Best regards<br>
><br>
> Marlon<br>
> Zitat von Shashank Rachamalla <<a href="mailto:shashank.rachamalla@hexagrid.com" target="_blank">shashank.rachamalla@hexagrid.com</a>>:<br>
><br>
>> Hi Javier<br>
>><br>
>> Thanks for the inputs but I came across another problem while testing:<br>
>><br>
>> If opennebula receives multiple vm requests in a short span of time, the<br>
>> scheduler might take decisions for all these vms considering the host<br>
>> monitoring information available from the last monitoring cycle. Ideally,<br>
>> before processing every pending request, fresh host monitoring<br>
>> information<br>
>> has to be taken into account as the previous set of requests might have<br>
>> already changed the host’s state. This can result in over committing when<br>
>> host is being used close to its full capacity.<br>
>><br>
>> *Is there any workaround which helps the scheduler to overcome the above<br>
>> problem ?*<br>
>><br>
>> steps to reproduce the problem scenario:<br>
>><br>
>> Host 1 : Total memory = 3GB<br>
>> Host 2 : Total memory = 2GB<br>
>> Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have a<br>
>> higher RANK value )<br>
>><br>
>> VM1: memory = 2GB<br>
>> VM2: memroy = 2GB<br>
>><br>
>> Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 will<br>
>> come up on Host1. ( Thus over committing )<br>
>><br>
>> Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up on<br>
>> Host1 and VM2 will come up on Host2. This is true because opennebula would<br>
>> have fetched a fresh set of host monitoring information in that time.<br>
>><br>
>><br>
>> On 4 November 2010 02:04, Javier Fontan <<a href="mailto:jfontan@gmail.com" target="_blank">jfontan@gmail.com</a>> wrote:<br>
>><br>
>>> Hello,<br>
>>><br>
>>> It looks fine to me. I think that taking out the memory the hypervisor<br>
>>> may be consuming is key to make it work.<br>
>>><br>
>>> Bye<br>
>>><br>
>>> On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula<br>
>>> <<a href="mailto:rbabu@hexagrid.com" target="_blank">rbabu@hexagrid.com</a>> wrote:<br>
>>> > Javier<br>
>>> ><br>
>>> > Yes we are using KVM and OpenNebula 1.4.<br>
>>> ><br>
>>> > We have been having this problem since a long time and we were doing<br>
>>> > all<br>
>>> > kinds of validations ourselves before submitting the request to<br>
>>> OpenNebula.<br>
>>> > (there should be enough memory in the cloud that matches the requested<br>
>>> > memory & there should be atleast one host that has memory > requested<br>
>>> memory<br>
>>> > ) We had to do those because OpenNebula would schedule to an<br>
>>> > arbitrary<br>
>>> > host based on the existing logic it had.<br>
>>> > So at last we thought that we need to make OpenNebula aware of memory<br>
>>> > allocated of running VM's on the host and started this discussion.<br>
>>> ><br>
>>> > Thanks for taking up this issue as priority. Appreciate it.<br>
>>> ><br>
>>> > Shashank came up with this patch to kvm.rb. Please take a look and let<br>
>>> > us<br>
>>> > know if that will work until we get a permanent solution.<br>
>>> ><br>
>>> ><br>
>>><br>
>>> ====================================================================================<br>
>>> ><br>
>>> > $mem_allocated_for_running_vms=0<br>
>>> > for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do<br>
>>> > $dominfo=`virsh dominfo #{i}`<br>
>>> > $dominfo.split(/\n/).each{|line|<br>
>>> > if line.match('^Max memory')<br>
>>> > $mem_allocated_for_running_vms += line.split("<br>
>>> > ")[2].strip.to_i<br>
>>> > end<br>
>>> > }<br>
>>> > end<br>
>>> ><br>
>>> > $mem_used_by_base_hypervisor = [some xyz kb that we want to set aside<br>
>>> > for<br>
>>> > hypervisor]<br>
>>> ><br>
>>> > $free_memory = $total_memory.to_i - (<br>
>>> > $mem_allocated_for_running_vms.to_i<br>
>>> +<br>
>>> > $mem_used_by_base_hypervisor.to_i )<br>
>>> ><br>
>>> ><br>
>>><br>
>>> ======================================================================================<br>
>>> ><br>
>>> > Ranga<br>
>>> ><br>
>>> > On Wed, Nov 3, 2010 at 2:16 PM, Javier Fontan <<a href="mailto:jfontan@gmail.com" target="_blank">jfontan@gmail.com</a>><br>
>>> > wrote:<br>
>>> >><br>
>>> >> Hello,<br>
>>> >><br>
>>> >> Sorry for the delay in the response.<br>
>>> >><br>
>>> >> It looks that the problem is OpenNebula calculating available memory.<br>
>>> >> For xen >= 3.2 there is a reliable way to get available memory that is<br>
>>> >> calling "xm info" and getting "max_free_memory" attribute.<br>
>>> >> Unfortunately for kvm or xen < 3.2 there is not such attribute. I<br>
>>> >> suppose you are using kvm as you tell about "free" command.<br>
>>> >><br>
>>> >> I began analyzing the kvm IM probe that gets memory information and<br>
>>> >> there is a problem on the way to get total memory. Here is how it now<br>
>>> >> gets memory information:<br>
>>> >><br>
>>> >> TOTALMEMORY: runs virsh info that gets the real physical memory<br>
>>> >> installed in the machine<br>
>>> >> FREEMEMORY: runs free command and gets the free column data without<br>
>>> >> buffers and cache<br>
>>> >> USEDMEMORY: runs top command and gets used memory from it (this counts<br>
>>> >> buffers and cache)<br>
>>> >><br>
>>> >> This is a big problem as those values do not match one with another (I<br>
>>> >> don't really know how I failed to see this before). Here is the<br>
>>> >> monitoring data from a host without VMs.<br>
>>> >><br>
>>> >> --8<------<br>
>>> >> TOTALMEMORY=8193988<br>
>>> >> USEDMEMORY=7819952<br>
>>> >> FREEMEMORY=7911924<br>
>>> >> ------>8--<br>
>>> >><br>
>>> >> As you can see it makes no sense at all. Even the TOTALMEMORY that is<br>
>>> >> got from virsh info is very misleading for oned as the host linux<br>
>>> >> instance does not have access to all that memory (some is consumed by<br>
>>> >> the hypervisor itself) as seen calling a free command:<br>
>>> >><br>
>>> >> --8<------<br>
>>> >> total used free shared buffers<br>
>>> >> cached<br>
>>> >> Mem: 8193988 7819192 374796 0 64176<br>
>>> 7473992<br>
>>> >> ------>8--<br>
>>> >><br>
>>> >> I am also copying this text as an issue to solve this problem<br>
>>> >> <a href="http://dev.opennebula.org/issues/388" target="_blank">http://dev.opennebula.org/issues/388</a>. It is masked to be solved for<br>
>>> >> 2.0.1 but the change will be compatible with 1.4 as it seems the the<br>
>>> >> only changed needed is the IM problem.<br>
>>> >><br>
>>> >> I can not offer you an immediate solution but we'll try to come up<br>
>>> >> with one as soon as possible.<br>
>>> >><br>
>>> >> Bye<br>
>>> >><br>
>>> >> On Wed, Nov 3, 2010 at 7:08 PM, Rangababu Chakravarthula<br>
>>> >> <<a href="mailto:rbabu@hexagrid.com" target="_blank">rbabu@hexagrid.com</a>> wrote:<br>
>>> >> > Hello Javier<br>
>>> >> > Please let us know if you want us to provide more detailed<br>
>>> >> > information<br>
>>> >> > with<br>
>>> >> > examples?<br>
>>> >> ><br>
>>> >> > Ranga<br>
>>> >> ><br>
>>> >> > On Fri, Oct 29, 2010 at 9:46 AM, Rangababu Chakravarthula<br>
>>> >> > <<a href="mailto:rbabu@hexagrid.com" target="_blank">rbabu@hexagrid.com</a>> wrote:<br>
>>> >> >><br>
>>> >> >> Javier<br>
>>> >> >><br>
>>> >> >> We saw that VM's were being deployed to the host where the<br>
>>> >> >> allocated<br>
>>> >> >> memory of all the VM's was higher than the available memory on the<br>
>>> >> >> host.<br>
>>> >> >><br>
>>> >> >> We think OpenNebula is executing free command on the host to<br>
>>> determine<br>
>>> >> >> if<br>
>>> >> >> there is any room and since free would always return the actual<br>
>>> memory<br>
>>> >> >> that<br>
>>> >> >> is being consumed and not the allocated, opennebula would push the<br>
>>> new<br>
>>> >> >> jobs<br>
>>> >> >> to the host.<br>
>>> >> >><br>
>>> >> >> That's the reason we want OpenNebula to be aware of memory<br>
>>> >> >> allocated<br>
>>> to<br>
>>> >> >> the VM's on the host.<br>
>>> >> >><br>
>>> >> >> Ranga<br>
>>> >> >><br>
>>> >> >> On Thu, Oct 28, 2010 at 2:02 PM, Javier Fontan <<a href="mailto:jfontan@gmail.com" target="_blank">jfontan@gmail.com</a>><br>
>>> >> >> wrote:<br>
>>> >> >>><br>
>>> >> >>> Hello,<br>
>>> >> >>><br>
>>> >> >>> Could you describe the problem you had? By default the scheduler<br>
>>> will<br>
>>> >> >>> not overcommit cpu nor memory.<br>
>>> >> >>><br>
>>> >> >>> Bye<br>
>>> >> >>><br>
>>> >> >>> On Thu, Oct 28, 2010 at 4:50 AM, Shashank Rachamalla<br>
>>> >> >>> <<a href="mailto:shashank.rachamalla@hexagrid.com" target="_blank">shashank.rachamalla@hexagrid.com</a>> wrote:<br>
>>> >> >>> > Hi<br>
>>> >> >>> ><br>
>>> >> >>> > We have a requirement where in the scheduler should not allow<br>
>>> memory<br>
>>> >> >>> > over<br>
>>> >> >>> > committing while choosing a host for new vm. In order to achieve<br>
>>> >> >>> > this,<br>
>>> >> >>> > we<br>
>>> >> >>> > have changed the way in which FREEMEMORY is being calculated for<br>
>>> >> >>> > each<br>
>>> >> >>> > host:<br>
>>> >> >>> ><br>
>>> >> >>> > FREE MEMORY = TOTAL MEMORY - [ Sum of memory values allocated<br>
>>> >> >>> > to<br>
>>> >> >>> > VMs<br>
>>> >> >>> > which<br>
>>> >> >>> > are currently running on the host ]<br>
>>> >> >>> ><br>
>>> >> >>> > Please let us know if the above approach is fine or is there any<br>
>>> >> >>> > better<br>
>>> >> >>> > way<br>
>>> >> >>> > to accomplish the task. We are using opennebula 1.4.<br>
>>> >> >>> ><br>
>>> >> >>> > --<br>
>>> >> >>> > Regards,<br>
>>> >> >>> > Shashank Rachamalla<br>
>>> >> >>> ><br>
>>> >> >>> > _______________________________________________<br>
>>> >> >>> > Users mailing list<br>
>>> >> >>> > <a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
>>> >> >>> > <a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
>>> >> >>> ><br>
>>> >> >>> ><br>
>>> >> >>><br>
>>> >> >>><br>
>>> >> >>><br>
>>> >> >>> --<br>
>>> >> >>> Javier Fontan, Grid & Virtualization Technology<br>
>>> >> >>> Engineer/Researcher<br>
>>> >> >>> DSA Research Group: <a href="http://dsa-research.org" target="_blank">http://dsa-research.org</a><br>
>>> >> >>> Globus GridWay Metascheduler: <a href="http://www.GridWay.org" target="_blank">http://www.GridWay.org</a><br>
>>> >> >>> OpenNebula Virtual Infrastructure Engine:<br>
>>> >> >>> <a href="http://www.OpenNebula.org" target="_blank">http://www.OpenNebula.org</a><br>
>>> >> >>> _______________________________________________<br>
>>> >> >>> Users mailing list<br>
>>> >> >>> <a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
>>> >> >>> <a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
>>> >> >><br>
>>> >> ><br>
>>> >> ><br>
>>> >><br>
>>> >><br>
>>> >><br>
>>> >> --<br>
>>> >> Javier Fontan, Grid & Virtualization Technology Engineer/Researcher<br>
>>> >> DSA Research Group: <a href="http://dsa-research.org" target="_blank">http://dsa-research.org</a><br>
>>> >> Globus GridWay Metascheduler: <a href="http://www.GridWay.org" target="_blank">http://www.GridWay.org</a><br>
>>> >> OpenNebula Virtual Infrastructure Engine: <a href="http://www.OpenNebula.org" target="_blank">http://www.OpenNebula.org</a><br>
>>> ><br>
>>> ><br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> Javier Fontan, Grid & Virtualization Technology Engineer/Researcher<br>
>>> DSA Research Group: <a href="http://dsa-research.org" target="_blank">http://dsa-research.org</a><br>
>>> Globus GridWay Metascheduler: <a href="http://www.GridWay.org" target="_blank">http://www.GridWay.org</a><br>
>>> OpenNebula Virtual Infrastructure Engine: <a href="http://www.OpenNebula.org" target="_blank">http://www.OpenNebula.org</a><br>
>>><br>
>><br>
>><br>
>><br>
>> --<br>
>> Regards,<br>
>> Shashank Rachamalla<br>
>><br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
> <a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
><br>
<br>
<br>
<br>
</div></div><font color="#888888">--<br>
Dr. Ruben Santiago Montero<br>
Associate Professor (Profesor Titular), Complutense University of Madrid<br>
<br>
URL: <a href="http://dsa-research.org/doku.php?id=people:ruben" target="_blank">http://dsa-research.org/doku.php?id=people:ruben</a><br>
Weblog: <a href="http://blog.dsa-research.org/?author=7" target="_blank">http://blog.dsa-research.org/?author=7</a><br>
</font><div><div></div><div>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Regards,<br>Shashank Rachamalla<br>