[one-users] Making scheduler allocation aware
Ruben S. Montero
rubensm at dacya.ucm.es
Thu Nov 11 04:08:17 PST 2010
Hi
Yes that is true for 1.4, OpenNebula 2.0 also updates the running vms
each time a VM is scheduled...
Cheers
Ruben
On Thu, Nov 11, 2010 at 12:58 PM, Shashank Rachamalla
<shashank.rachamalla at hexagrid.com> wrote:
> Hi
>
> What Marlon said regarding RANK=RUNNING_VMS is the exact concern. However, I
> will check out the source code once. We are using opennebula 1.4 with the
> following template: ( Note that the value of PRIORITY is calculated using
> FREEMEMORY, FREECPU and RUNNING_VMS )
>
> DISK=[
> clone=no,
> source=/mnt/dev_store_100000/images/glpnnu0,
> target=hda,
> type=disk
> ]
> DISK=[
> clone=no,
>
> source=/mnt/dev_store_100000/iso/FABCAC11-768B-4683-EB99-085ECB800000,
> target=hdb,
> type=cdrom
> ]
> MEMORY=2000
> REQUIREMENTS="FREEMEMORY>2048000"
> RANK=PRIORITY
> NAME=glpnnu
> OS=[
> boot=cdrom
> ]
> NIC=[
> model=e1000,
> bridge=br101,
> MAC=00:89:10:36:10:26
> ]
> INPUT=[
> type=tablet
> ]
> VCPU=1
> GRAPHICS=[
> port=5971,
> type=vnc,
> listen=0.0.0.0
> ]
>
> On 11 November 2010 16:08, <opennebula at nerling.ch> wrote:
>>
>> Hi Rubens.
>> I'm sure the RANK=-RUNNING_VMS will not apply in such a scenario, because
>> the scheduler does not update the RUNNING_VMS after the creation of the VM
>> but just after monitoring the host!
>> So between this events the RUNNING_VMS value stays unchanged and by my
>> experience this Host will become the 'chosen one' for new deployed VMs up to
>> the next host monitoring.
>> And i'm not really sure if the scheduler sums USED MEMORY (ALLOCATED) and
>> the memory used by the VM to prevent overcommiting, we could look in the
>> source code for this.
>> I must say never have experienced a host who became more VMs as possible.
>>
>> Best regards
>>
>> Marlon
>>
>> Zitat von "Ruben S. Montero" <rubensm at dacya.ucm.es>:
>>
>>> Hi,
>>>
>>> We use the total memory to allocate VM's, this is not going to change
>>> between monitoring events. Right now, we are able to schedule multiple
>>> VMs in the same scheduling step, to the same host without
>>> overcommitting memory.
>>>
>>> Cheers
>>>
>>> On Thu, Nov 11, 2010 at 10:42 AM, Shashank Rachamalla
>>> <shashank.rachamalla at hexagrid.com> wrote:
>>>>
>>>> Hi
>>>>
>>>> Thanks for the reply. I agree that a VM will not be allowed to use more
>>>> than
>>>> what has been specified in its template during creation but I was
>>>> referring
>>>> to a scenario where memory available on the host is over committed. I
>>>> guess
>>>> the problem here is with allowing multiple VMs to be dispatched on to a
>>>> single host in one schedule cycle.
>>>>
>>>>
>>>> On 11 November 2010 14:47, Ruben S. Montero <rubensm at dacya.ucm.es>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Regarding capacity (CPU, Memory), it is updated every time a VM is
>>>>> submitted, so no overcommitment is possible (beyond that specified by
>>>>> the CPU attribute in the VM template). This also works in 1.4
>>>>>
>>>>> Cheers
>>>>>
>>>>> Ruben
>>>>>
>>>>> On Thu, Nov 11, 2010 at 10:08 AM, <opennebula at nerling.ch> wrote:
>>>>> > Hallo Shashank.
>>>>> > I'm having the same problem in 1.4.
>>>>> > You must workaround it by youself, so instead of using onevm
>>>>> > directly,
>>>>> > use
>>>>> > it with a wrapper script who looks and waits for deploy if a VM is
>>>>> > pending.
>>>>> > I hope this behaviour is fixed on 2.0 (Hallo developers??)
>>>>> >
>>>>> > Best regards
>>>>> >
>>>>> > Marlon
>>>>> > Zitat von Shashank Rachamalla <shashank.rachamalla at hexagrid.com>:
>>>>> >
>>>>> >> Hi Javier
>>>>> >>
>>>>> >> Thanks for the inputs but I came across another problem while
>>>>> >> testing:
>>>>> >>
>>>>> >> If opennebula receives multiple vm requests in a short span of time,
>>>>> >> the
>>>>> >> scheduler might take decisions for all these vms considering the
>>>>> >> host
>>>>> >> monitoring information available from the last monitoring cycle.
>>>>> >> Ideally,
>>>>> >> before processing every pending request, fresh host monitoring
>>>>> >> information
>>>>> >> has to be taken into account as the previous set of requests might
>>>>> >> have
>>>>> >> already changed the host’s state. This can result in over committing
>>>>> >> when
>>>>> >> host is being used close to its full capacity.
>>>>> >>
>>>>> >> *Is there any workaround which helps the scheduler to overcome the
>>>>> >> above
>>>>> >> problem ?*
>>>>> >>
>>>>> >> steps to reproduce the problem scenario:
>>>>> >>
>>>>> >> Host 1 : Total memory = 3GB
>>>>> >> Host 2 : Total memory = 2GB
>>>>> >> Assume Host1 and Host2 have same number of CPU cores. ( Host1 will
>>>>> >> have
>>>>> >> a
>>>>> >> higher RANK value )
>>>>> >>
>>>>> >> VM1: memory = 2GB
>>>>> >> VM2: memroy = 2GB
>>>>> >>
>>>>> >> Start VM1 and VM2 immediately one after the other. Both VM1 and VM2
>>>>> >> will
>>>>> >> come up on Host1. ( Thus over committing )
>>>>> >>
>>>>> >> Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come
>>>>> >> up
>>>>> >> on
>>>>> >> Host1 and VM2 will come up on Host2. This is true because opennebula
>>>>> >> would
>>>>> >> have fetched a fresh set of host monitoring information in that
>>>>> >> time.
>>>>> >>
>>>>> >>
>>>>> >> On 4 November 2010 02:04, Javier Fontan <jfontan at gmail.com> wrote:
>>>>> >>
>>>>> >>> Hello,
>>>>> >>>
>>>>> >>> It looks fine to me. I think that taking out the memory the
>>>>> >>> hypervisor
>>>>> >>> may be consuming is key to make it work.
>>>>> >>>
>>>>> >>> Bye
>>>>> >>>
>>>>> >>> On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
>>>>> >>> <rbabu at hexagrid.com> wrote:
>>>>> >>> > Javier
>>>>> >>> >
>>>>> >>> > Yes we are using KVM and OpenNebula 1.4.
>>>>> >>> >
>>>>> >>> > We have been having this problem since a long time and we were
>>>>> >>> > doing
>>>>> >>> > all
>>>>> >>> > kinds of validations ourselves before submitting the request to
>>>>> >>> OpenNebula.
>>>>> >>> > (there should be enough memory in the cloud that matches the
>>>>> >>> > requested
>>>>> >>> > memory & there should be atleast one host that has memory >
>>>>> >>> > requested
>>>>> >>> memory
>>>>> >>> > ) We had to do those because OpenNebula would schedule to an
>>>>> >>> > arbitrary
>>>>> >>> > host based on the existing logic it had.
>>>>> >>> > So at last we thought that we need to make OpenNebula aware of
>>>>> >>> > memory
>>>>> >>> > allocated of running VM's on the host and started this
>>>>> >>> > discussion.
>>>>> >>> >
>>>>> >>> > Thanks for taking up this issue as priority. Appreciate it.
>>>>> >>> >
>>>>> >>> > Shashank came up with this patch to kvm.rb. Please take a look
>>>>> >>> > and
>>>>> >>> > let
>>>>> >>> > us
>>>>> >>> > know if that will work until we get a permanent solution.
>>>>> >>> >
>>>>> >>> >
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> ====================================================================================
>>>>> >>> >
>>>>> >>> > $mem_allocated_for_running_vms=0
>>>>> >>> > for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
>>>>> >>> > $dominfo=`virsh dominfo #{i}`
>>>>> >>> > $dominfo.split(/\n/).each{|line|
>>>>> >>> > if line.match('^Max memory')
>>>>> >>> > $mem_allocated_for_running_vms += line.split("
>>>>> >>> > ")[2].strip.to_i
>>>>> >>> > end
>>>>> >>> > }
>>>>> >>> > end
>>>>> >>> >
>>>>> >>> > $mem_used_by_base_hypervisor = [some xyz kb that we want to set
>>>>> >>> > aside
>>>>> >>> > for
>>>>> >>> > hypervisor]
>>>>> >>> >
>>>>> >>> > $free_memory = $total_memory.to_i - (
>>>>> >>> > $mem_allocated_for_running_vms.to_i
>>>>> >>> +
>>>>> >>> > $mem_used_by_base_hypervisor.to_i )
>>>>> >>> >
>>>>> >>> >
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> ======================================================================================
>>>>> >>> >
>>>>> >>> > Ranga
>>>>> >>> >
>>>>> >>> > On Wed, Nov 3, 2010 at 2:16 PM, Javier Fontan <jfontan at gmail.com>
>>>>> >>> > wrote:
>>>>> >>> >>
>>>>> >>> >> Hello,
>>>>> >>> >>
>>>>> >>> >> Sorry for the delay in the response.
>>>>> >>> >>
>>>>> >>> >> It looks that the problem is OpenNebula calculating available
>>>>> >>> >> memory.
>>>>> >>> >> For xen >= 3.2 there is a reliable way to get available memory
>>>>> >>> >> that
>>>>> >>> >> is
>>>>> >>> >> calling "xm info" and getting "max_free_memory" attribute.
>>>>> >>> >> Unfortunately for kvm or xen < 3.2 there is not such attribute.
>>>>> >>> >> I
>>>>> >>> >> suppose you are using kvm as you tell about "free" command.
>>>>> >>> >>
>>>>> >>> >> I began analyzing the kvm IM probe that gets memory information
>>>>> >>> >> and
>>>>> >>> >> there is a problem on the way to get total memory. Here is how
>>>>> >>> >> it
>>>>> >>> >> now
>>>>> >>> >> gets memory information:
>>>>> >>> >>
>>>>> >>> >> TOTALMEMORY: runs virsh info that gets the real physical memory
>>>>> >>> >> installed in the machine
>>>>> >>> >> FREEMEMORY: runs free command and gets the free column data
>>>>> >>> >> without
>>>>> >>> >> buffers and cache
>>>>> >>> >> USEDMEMORY: runs top command and gets used memory from it (this
>>>>> >>> >> counts
>>>>> >>> >> buffers and cache)
>>>>> >>> >>
>>>>> >>> >> This is a big problem as those values do not match one with
>>>>> >>> >> another
>>>>> >>> >> (I
>>>>> >>> >> don't really know how I failed to see this before). Here is the
>>>>> >>> >> monitoring data from a host without VMs.
>>>>> >>> >>
>>>>> >>> >> --8<------
>>>>> >>> >> TOTALMEMORY=8193988
>>>>> >>> >> USEDMEMORY=7819952
>>>>> >>> >> FREEMEMORY=7911924
>>>>> >>> >> ------>8--
>>>>> >>> >>
>>>>> >>> >> As you can see it makes no sense at all. Even the TOTALMEMORY
>>>>> >>> >> that
>>>>> >>> >> is
>>>>> >>> >> got from virsh info is very misleading for oned as the host
>>>>> >>> >> linux
>>>>> >>> >> instance does not have access to all that memory (some is
>>>>> >>> >> consumed
>>>>> >>> >> by
>>>>> >>> >> the hypervisor itself) as seen calling a free command:
>>>>> >>> >>
>>>>> >>> >> --8<------
>>>>> >>> >> total used free shared buffers
>>>>> >>> >> cached
>>>>> >>> >> Mem: 8193988 7819192 374796 0 64176
>>>>> >>> 7473992
>>>>> >>> >> ------>8--
>>>>> >>> >>
>>>>> >>> >> I am also copying this text as an issue to solve this problem
>>>>> >>> >> http://dev.opennebula.org/issues/388. It is masked to be solved
>>>>> >>> >> for
>>>>> >>> >> 2.0.1 but the change will be compatible with 1.4 as it seems the
>>>>> >>> >> the
>>>>> >>> >> only changed needed is the IM problem.
>>>>> >>> >>
>>>>> >>> >> I can not offer you an immediate solution but we'll try to come
>>>>> >>> >> up
>>>>> >>> >> with one as soon as possible.
>>>>> >>> >>
>>>>> >>> >> Bye
>>>>> >>> >>
>>>>> >>> >> On Wed, Nov 3, 2010 at 7:08 PM, Rangababu Chakravarthula
>>>>> >>> >> <rbabu at hexagrid.com> wrote:
>>>>> >>> >> > Hello Javier
>>>>> >>> >> > Please let us know if you want us to provide more detailed
>>>>> >>> >> > information
>>>>> >>> >> > with
>>>>> >>> >> > examples?
>>>>> >>> >> >
>>>>> >>> >> > Ranga
>>>>> >>> >> >
>>>>> >>> >> > On Fri, Oct 29, 2010 at 9:46 AM, Rangababu Chakravarthula
>>>>> >>> >> > <rbabu at hexagrid.com> wrote:
>>>>> >>> >> >>
>>>>> >>> >> >> Javier
>>>>> >>> >> >>
>>>>> >>> >> >> We saw that VM's were being deployed to the host where the
>>>>> >>> >> >> allocated
>>>>> >>> >> >> memory of all the VM's was higher than the available memory
>>>>> >>> >> >> on
>>>>> >>> >> >> the
>>>>> >>> >> >> host.
>>>>> >>> >> >>
>>>>> >>> >> >> We think OpenNebula is executing free command on the host to
>>>>> >>> determine
>>>>> >>> >> >> if
>>>>> >>> >> >> there is any room and since free would always return the
>>>>> >>> >> >> actual
>>>>> >>> memory
>>>>> >>> >> >> that
>>>>> >>> >> >> is being consumed and not the allocated, opennebula would
>>>>> >>> >> >> push
>>>>> >>> >> >> the
>>>>> >>> new
>>>>> >>> >> >> jobs
>>>>> >>> >> >> to the host.
>>>>> >>> >> >>
>>>>> >>> >> >> That's the reason we want OpenNebula to be aware of memory
>>>>> >>> >> >> allocated
>>>>> >>> to
>>>>> >>> >> >> the VM's on the host.
>>>>> >>> >> >>
>>>>> >>> >> >> Ranga
>>>>> >>> >> >>
>>>>> >>> >> >> On Thu, Oct 28, 2010 at 2:02 PM, Javier Fontan
>>>>> >>> >> >> <jfontan at gmail.com>
>>>>> >>> >> >> wrote:
>>>>> >>> >> >>>
>>>>> >>> >> >>> Hello,
>>>>> >>> >> >>>
>>>>> >>> >> >>> Could you describe the problem you had? By default the
>>>>> >>> >> >>> scheduler
>>>>> >>> will
>>>>> >>> >> >>> not overcommit cpu nor memory.
>>>>> >>> >> >>>
>>>>> >>> >> >>> Bye
>>>>> >>> >> >>>
>>>>> >>> >> >>> On Thu, Oct 28, 2010 at 4:50 AM, Shashank Rachamalla
>>>>> >>> >> >>> <shashank.rachamalla at hexagrid.com> wrote:
>>>>> >>> >> >>> > Hi
>>>>> >>> >> >>> >
>>>>> >>> >> >>> > We have a requirement where in the scheduler should not
>>>>> >>> >> >>> > allow
>>>>> >>> memory
>>>>> >>> >> >>> > over
>>>>> >>> >> >>> > committing while choosing a host for new vm. In order to
>>>>> >>> >> >>> > achieve
>>>>> >>> >> >>> > this,
>>>>> >>> >> >>> > we
>>>>> >>> >> >>> > have changed the way in which FREEMEMORY is being
>>>>> >>> >> >>> > calculated
>>>>> >>> >> >>> > for
>>>>> >>> >> >>> > each
>>>>> >>> >> >>> > host:
>>>>> >>> >> >>> >
>>>>> >>> >> >>> > FREE MEMORY = TOTAL MEMORY - [ Sum of memory values
>>>>> >>> >> >>> > allocated
>>>>> >>> >> >>> > to
>>>>> >>> >> >>> > VMs
>>>>> >>> >> >>> > which
>>>>> >>> >> >>> > are currently running on the host ]
>>>>> >>> >> >>> >
>>>>> >>> >> >>> > Please let us know if the above approach is fine or is
>>>>> >>> >> >>> > there
>>>>> >>> >> >>> > any
>>>>> >>> >> >>> > better
>>>>> >>> >> >>> > way
>>>>> >>> >> >>> > to accomplish the task. We are using opennebula 1.4.
>>>>> >>> >> >>> >
>>>>> >>> >> >>> > --
>>>>> >>> >> >>> > Regards,
>>>>> >>> >> >>> > Shashank Rachamalla
>>>>> >>> >> >>> >
>>>>> >>> >> >>> > _______________________________________________
>>>>> >>> >> >>> > Users mailing list
>>>>> >>> >> >>> > Users at lists.opennebula.org
>>>>> >>> >> >>> >
>>>>> >>> >> >>> > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>>> >>> >> >>> >
>>>>> >>> >> >>> >
>>>>> >>> >> >>>
>>>>> >>> >> >>>
>>>>> >>> >> >>>
>>>>> >>> >> >>> --
>>>>> >>> >> >>> Javier Fontan, Grid & Virtualization Technology
>>>>> >>> >> >>> Engineer/Researcher
>>>>> >>> >> >>> DSA Research Group: http://dsa-research.org
>>>>> >>> >> >>> Globus GridWay Metascheduler: http://www.GridWay.org
>>>>> >>> >> >>> OpenNebula Virtual Infrastructure Engine:
>>>>> >>> >> >>> http://www.OpenNebula.org
>>>>> >>> >> >>> _______________________________________________
>>>>> >>> >> >>> Users mailing list
>>>>> >>> >> >>> Users at lists.opennebula.org
>>>>> >>> >> >>>
>>>>> >>> >> >>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>>> >>> >> >>
>>>>> >>> >> >
>>>>> >>> >> >
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >> --
>>>>> >>> >> Javier Fontan, Grid & Virtualization Technology
>>>>> >>> >> Engineer/Researcher
>>>>> >>> >> DSA Research Group: http://dsa-research.org
>>>>> >>> >> Globus GridWay Metascheduler: http://www.GridWay.org
>>>>> >>> >> OpenNebula Virtual Infrastructure Engine:
>>>>> >>> >> http://www.OpenNebula.org
>>>>> >>> >
>>>>> >>> >
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> Javier Fontan, Grid & Virtualization Technology Engineer/Researcher
>>>>> >>> DSA Research Group: http://dsa-research.org
>>>>> >>> Globus GridWay Metascheduler: http://www.GridWay.org
>>>>> >>> OpenNebula Virtual Infrastructure Engine: http://www.OpenNebula.org
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Regards,
>>>>> >> Shashank Rachamalla
>>>>> >>
>>>>> >
>>>>> > _______________________________________________
>>>>> > Users mailing list
>>>>> > Users at lists.opennebula.org
>>>>> > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dr. Ruben Santiago Montero
>>>>> Associate Professor (Profesor Titular), Complutense University of
>>>>> Madrid
>>>>>
>>>>> URL: http://dsa-research.org/doku.php?id=people:ruben
>>>>> Weblog: http://blog.dsa-research.org/?author=7
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at lists.opennebula.org
>>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shashank Rachamalla
>>>>
>>>
>>>
>>>
>>> --
>>> Dr. Ruben Santiago Montero
>>> Associate Professor (Profesor Titular), Complutense University of Madrid
>>>
>>> URL: http://dsa-research.org/doku.php?id=people:ruben
>>> Weblog: http://blog.dsa-research.org/?author=7
>>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opennebula.org
>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>
>
> --
> Regards,
> Shashank Rachamalla
>
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>
--
Dr. Ruben Santiago Montero
Associate Professor (Profesor Titular), Complutense University of Madrid
URL: http://dsa-research.org/doku.php?id=people:ruben
Weblog: http://blog.dsa-research.org/?author=7
More information about the Users
mailing list