[one-users] MPI Cluster on OpenNebula

[Quynh Le]

Hello Shank,

I thank you so much for your description. 

But we are using virtual machines, so changes and configuration on each virtual node will be lost when it terminates. (1)
Every time we launch a cluster, we may change the number of slave nodes, up to the problem size. (2)

So could you be kind enough to help me make a small thing clear once again, as I don't know about Cloudera yet.

For example, we do not know in advance how many slave nodes will be launched (because of (2)), what their IP addresses are, so the configuration on /etc/hostsis left util the time the cluster is launched. If the cluster size is big, it'd be difficult. Do you configure this manually? Or will OpenNebula context script help? Or will Cloudera do?

Thank you again.

Quynh

 
Anyway, 30min for 50-VM cluster is impressive. What propagation method you used? How big a VM image? As the typical LAN speed is 12MB/s (100Mbps) if you use scp. (I don't try NFS yet).




Well, you can do the following

	* Create a master node template and slave node template, make their 
configuration such that they keep that relationship, one way ssh key etc
	* Deploy the master node and configure the software on it then deploy as many slave nodes to connect back to the master node.
	* In our configuration, we deploy M identical nodes, then we pick one of the nodes as master and install the master node software (cloudera manager 
in our case).
	* Then we use cloudera manager to deploy the rest of the nodes, in our case this includes one hdfs name node, one job tracker, (M-3) hdfs data nodes and (M-3) map reduce task trackers.
	* We have deployed around 50 VMs within a 30 min period using this configuration.

Shank

On Fri, Jul 13, 2012 at 2:58 AM, Quynh Le <lhnquynh at yahoo.com> wrote:

Hello Shankhadeep,
>
>Thank you for your information. I am able to setup such a virtual cluster using another cloud middleware like OpenNebula so I can understand the situation. What I wanna make clear is:
>- This is a kind of Master/Slave cluster: 1 head node and N worker nodes.
>- We can launch a group of VMs to make N+1 VMs for the cluster. 
>- Then, do you have to setup hadoop master node and worker nodes manually, OR are they (VMs) automatically configured to be "1 master + N workers".
>- In this case, how many VM images you use? 1 VM image for master node, 1 for worker nodes, or 1 for all?
>
>I'm looking forward to your sharing.
>
>Cheers,
>Quynh
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20120713/822decd4/attachment-0001.htm>