[one-users] monitor_ds.sh in a shared storage environement

[Jaime Melis]

Hi Armin,

apologies for the delay in the reponse.

The problem here is that ideally we want to report the maximum size of the
images if they are sparse images. If we have a 10G sparse image that is
taking up only 1G in space, we would like to report the 10G amount, and not
1G, since we would like to guarantee available space in the datastore.
That's why we cannot use `df`.

In any case, the system now in place is not properly reporting the image
size, since we would need to be using `du --aparent-size`. So as for now,
switching from `du` to `df` should be a plausible workaround with no real
impact, except better performance. On the other hand, as you have reported
`du` generates a lot of I/O, so ideally we would need to be doing is to get
the USED disk space directly from OpenNebula.

To sum it up, we have these two related issues to improve this:
http://dev.opennebula.org/issues/2783
http://dev.opennebula.org/issues/2402

As for your particular case, I think you should modify monitor_ds.sh with a
call to `df` instead of `du`.

 > - is it needed to check the datastores from every single host (if it is
a shared environment and the opennebula server itself has access to the
datastores)?
The monitorization driver is very simple (in order to be easily adapted),
and there is no way to orchestrate this. It would be a very nice thing to
have though, feel free to open a feature request to implement this.

 > - why is the base_path checked at all? isn't it enough to check the
datastores? there is du run on all the directories twice and du is very
resource consuming ...
This is a tricky one. If the mountpoint does not exist
(/var/lib/one/datastores/<id>) it can be created automatically by
opennebula, but only when the first VM is executed. OpenNebula needs to
know in this case if there will be enough space in the datastore, and it
assumes that if the mountpoint doesn't exist yet it will have the same
space available as the parent folder, which is the datastore location:
/var/lib/one/datastores. That's why we need to be checking that too.

Regards,
Jaime

On Fri, Jun 6, 2014 at 2:57 PM, Armin Deliomini <
armin.deliomini at runtastic.com> wrote:

> Hi!
>
> Platform: Ubuntu 12.04 + KVM
> Opennebula 4.6.1
>
> I recently upgraded from opennebula 4.2 to 4.6.1 ending up in hosts not
> beeing monitored anymore. After some investigation I found out, that there
> is a new remote monitoring script, monitor_ds.sh.
>
> I have a shared storage environment (some central Netapps sharing NFS) and
> so my question is: the script is executed on all the hosts, so all hosts
> run a du -sLm on all datastores and the base_path (/var/lib/on/datastores)
> which can consume quite some storage performance unnecessarily.
>
> In my case hosts haven't been monitored at all, because there was a
> driectory containing millions of small files in one of the datastores (used
> as a temp location without thinking of the consequences) and du would never
> finish in time.
>
> There are two questions:
>
>  - is it needed to check the datastores from every single host (if it is a
> shared environment and the opennebula server itself has access to the
> datastores)?
>
>  - why is the base_path checked at all? isn't it enough to check the
> datastores? there is du run on all the directories twice and du is very
> resource consuming ...
>
> Is there a reason for this I am missing?
>
> Thx, Armin
>
>
> --
>
> runtastische Grüße · runtastic regards
>
>
>
> *Armin Deliomini*
>
> *Infrastructure engineer*
>
>
> *Virtualization specialist *
>
> *Storage engineer*
>
> *DevOPS*
>
>
> *OPS*
>
> *[image: runtastic - makes sports funtastic] <http://www.runtastic.com>*
>
> runtastic GmbH
>
> Pluskaufstraße 7 · Business Center
>
> 4061 Pasching / Linz (Austria)
>
> Tel. +43(0)67689692450
>
> Fax. +43(0)7229.23327
>
> armin.deliomini at runtastic.com
>
> www.runtastic.com
>
>
>
>  <http://www.facebook.com/runtastic>   <http://www.twitter.com/runtastic>
>    <http://www.youtube.com/user/runtastic>
>
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>


-- 
Jaime Melis
Project Engineer
OpenNebula - Flexible Enterprise Cloud Made Simple
www.OpenNebula.org | jmelis at opennebula.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20140611/6968ece3/attachment-0001.htm>