<div>Hi Quynh</div><div><br></div>Well, you can do the following<div><br></div><div><ul><li>Create a master node template and slave node template, make their configuration such that they keep that relationship, one way ssh key etc</li>
<li>Deploy the master node and configure the software on it then deploy as many slave nodes to connect back to the master node.</li><li>In our configuration, we deploy M identical nodes, then we pick one of the nodes as master and install the master node software (cloudera manager in our case).</li>
<li>Then we use cloudera manager to deploy the rest of the nodes, in our case this includes one hdfs name node, one job tracker, (M-3) hdfs data nodes and (M-3) map reduce task trackers.</li><li>We have deployed around 50 VMs within a 30 min period using this configuration.</li>
</ul></div><div><br></div><div>Shank</div><div><br></div><div><div class="gmail_quote">On Fri, Jul 13, 2012 at 2:58 AM, Quynh Le <span dir="ltr"><<a href="mailto:lhnquynh@yahoo.com" target="_blank">lhnquynh@yahoo.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="font-family:arial,helvetica,sans-serif;font-size:12pt">Hello Shankhadeep,<br><br>Thank you for your information. I am able to setup such a virtual cluster using another cloud middleware like OpenNebula so I can understand the situation. What I wanna make clear is:<br>
- This is a kind of Master/Slave cluster: 1 head node and N worker nodes.<br>- We can launch a group of VMs to make N+1 VMs for the cluster. <br>- Then, do you have to setup hadoop master node and worker nodes manually, OR are they (VMs) automatically configured to be "1 master + N workers".<br>
- In this case, how many VM images you use? 1 VM image for master node, 1 for worker nodes, or 1 for all?<br><br>I'm looking forward to your sharing.<br><br>Cheers,<br>Quynh</div><br><div class="hm HOEnZb"> </div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt">
<div class="hm HOEnZb"> </div><div style="font-family:times new roman,new york,times,serif;font-size:12pt"><div class="hm HOEnZb"> <div dir="ltr"> <font face="Arial"> <hr size="1"> <b><span style="font-weight:bold">From:</span></b> Shankhadeep Shome <<a href="mailto:shank15217@gmail.com" target="_blank">shank15217@gmail.com</a>><br>
<b><span style="font-weight:bold">To:</span></b> Quynh Le <<a href="mailto:lhnquynh@yahoo.com" target="_blank">lhnquynh@yahoo.com</a>> <br><b><span style="font-weight:bold">Cc:</span></b> "<a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a>" <<a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a>> <br>
<b><span style="font-weight:bold">Sent:</span></b> Friday, July 13, 2012 12:22 PM<br> <b><span style="font-weight:bold">Subject:</span></b> Re: [one-users] MPI Cluster on OpenNebula<br> </font> </div></div><div><div class="h5">
<br>
<div>Yes, clusters are particularly easy on this environment, I work with our analytics group on hadoop clusters. The most important thing to do is install good parallel computing utilities like pdsh and have ssh equivalency setup, then creating a cluster of machine sis realyl easy because you get N machines configured the same with full access to each other. Then you use ssh based scripts to distribute your software. In our environment we use Cloudera manager to distribute and start and manage our hadoop cluster, there should be something similar for mpi clusters. The nice thing about open nebula is that you can distribute your resource allocation over generic cpu, memory and disk resources and hardware optimization is easy because you can customize one template and redeploy the entire cluster.<br>
<br><div>On Thu, Jul 12, 2012 at 3:51 AM, Quynh Le <span dir="ltr"><<a rel="nofollow" href="mailto:lhnquynh@yahoo.com" target="_blank">lhnquynh@yahoo.com</a>></span> wrote:<br><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div style="font-size:12pt;font-family:arial,helvetica,sans-serif"><div>Hello,</div><div><br></div><div>I intend to setup a ready-to-go MPI Cluster on OpenNebula from a VM image. Is it possible? Can each VM know its role (master or slave), as well as other VM, to fill in the mpdhost file? I don't know much about context yet. <br>
</div><div><br></div><div>Thanks for any explanation.</div><div><br></div><div>Best regards,</div><div>Quynh</div></div></div><br>_______________________________________________<br>
Users mailing list<br>
<a rel="nofollow" href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
<br></blockquote></div><br>
</div><br><br> </div></div></div> </div> </blockquote></div><br></div>