[one-users] GRID-like functionalities with OpenNebula as a base? [Hadoop, or other possible solutions]

[Olivier Sallou]

Already answered in the list:

Le 3/20/12 5:24 PM, biro lehel a écrit :

> Dear all,
>
> After having successfully set up a small-scale (but up-and-running)
> experimental Private Cloud (using OpenSUSE 12.1 64bit machines -also
> VM's-, KVM as hypervisor, and ONE 3.2.0), my next task is to find a way
> in which GRID-like services (see below) could be implemented inside an
> OpenNebula-based Cloud. Basically, and architecturally-speaking, I think
> I would need a PaaS layer above my existing IaaS layer.
>
> I am interested in the following tasks:
>
> *job distribution among a set of dynamically created (or, as a start,
> fixed) VM's, initiated by a master-VM
> * job execution on the worker-VM's and, more importantly, execution
> monitoring of these jobs by the master-VM
> * results gathering on the master-VM
>
> I have briefly looked though Hadoop, which uses MapReduce, but, having
> no experience with it, I am not sure whether if it is the best tool to
> use for such a use-case, and also, I don't know if it could be at all
> used in my environment. So, my fundamental question is the following:
> which is the best way to start on, a way that would lead to satisfying
> my above-mentioned 3 necessities?
> Is Hadoop perfectly compatible with OpenNebula, and could be used for
> such purposes, or are there other and more fitting solutions or tools to
> use? Basically, all I'm asking is, what is the most appropriate way for
> obtaining these basic GRID-computing functionalities on a Platform
> constructed above the Infrastructure?
Hadoop will fit, but Hadoop is mainly focused on map/reduce jobs, and
use of HDFS storage.

There is no issue at setting up Hadoop on "top of" Open Nebula, I did
it. IT is up to you to configure your images to setup Hadoop
automatically (with ssh password-less communication).

What you refer too would be more likely a really grid system like Torque
ou Sun Grid Engine. Those can also be put above Open Nebula (or other
cloud system). But, same thing, you need to create images that will
automatically set them up (I did too with Torque)

> Any suggestions, thoughts, and hints are extremely welcome, since this
> is a rather important and urgent matter for me.
>
> Thank you,
> Lehel.
>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
-- gpg key id: 4096R/326D8438 (keyring.debian.org) Key fingerprint =
5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
_______________________________________________ Users mailing list
Users at lists.opennebula.org
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20120321/db3f5565/attachment-0003.htm>