[one-users] MPI Cluster on OpenNebula

[Shankhadeep Shome]

Hi Quynh

Well, you can do the following


   - Create a master node template and slave node template, make their
   configuration such that they keep that relationship, one way ssh key etc
   - Deploy the master node and configure the software on it then deploy as
   many slave nodes to connect back to the master node.
   - In our configuration, we deploy M identical nodes, then we pick one of
   the nodes as master and install the master node software (cloudera manager
   in our case).
   - Then we use cloudera manager to deploy the rest of the nodes, in our
   case this includes one hdfs name node, one job tracker, (M-3) hdfs data
   nodes and (M-3) map reduce task trackers.
   - We have deployed around 50 VMs within a 30 min period using this
   configuration.


Shank

On Fri, Jul 13, 2012 at 2:58 AM, Quynh Le <lhnquynh at yahoo.com> wrote:

> Hello Shankhadeep,
>
> Thank you for your information. I am able to setup such a virtual cluster
> using another cloud middleware like OpenNebula so I can understand the
> situation. What I wanna make clear is:
> - This is a kind of Master/Slave cluster: 1 head node and N worker nodes.
> - We can launch a group of VMs to make N+1 VMs for the cluster.
> - Then, do you have to setup hadoop master node and worker nodes manually,
> OR are they (VMs) automatically configured to be "1 master + N workers".
> - In this case, how many VM images you use? 1 VM image for master node, 1
> for worker nodes, or 1 for all?
>
> I'm looking forward to your sharing.
>
> Cheers,
> Quynh
>
>   ------------------------------
> *From:* Shankhadeep Shome <shank15217 at gmail.com>
> *To:* Quynh Le <lhnquynh at yahoo.com>
> *Cc:* "users at lists.opennebula.org" <users at lists.opennebula.org>
> *Sent:* Friday, July 13, 2012 12:22 PM
> *Subject:* Re: [one-users] MPI Cluster on OpenNebula
>
> Yes, clusters are particularly easy on this environment, I work with our
> analytics group on hadoop clusters. The most important thing to do is
> install good parallel computing utilities like pdsh and have ssh
> equivalency setup, then creating a cluster of machine sis realyl easy
> because you get N machines configured the same with full access to each
> other. Then you use ssh based scripts to distribute your software. In our
> environment we use Cloudera manager to distribute and start and manage our
> hadoop cluster, there should be something similar for mpi clusters. The
> nice thing about open nebula is that you can distribute your resource
> allocation over generic cpu, memory and disk resources and hardware
> optimization is easy because you can customize one template and redeploy
> the entire cluster.
>
> On Thu, Jul 12, 2012 at 3:51 AM, Quynh Le <lhnquynh at yahoo.com> wrote:
>
> Hello,
>
> I intend to setup a ready-to-go MPI Cluster on OpenNebula from a VM image.
> Is it possible? Can each VM know its role (master or slave), as well as
> other VM, to fill in the mpdhost file? I don't know much about context yet.
>
> Thanks for any explanation.
>
> Best regards,
> Quynh
>
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20120713/8bcdf2c7/attachment-0002.htm>