<div dir="ltr"><div class="gmail_default" style="font-size:small"><b>Dear Marco, </b></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Try this guide which i also get some clarification from Javier. I have try and it work better.</div><div class="gmail_default" style="font-size:small">Somehow, how you physical environment and network setup will give impact to the performance.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">I assume you fresh install and want to share "system" datastores 0 and "images" datastore 1. If you need add additional both system for cluster them just creat an empty template .... Datastore -> fill in name -> select datastore type either images or system -> click save</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">----------- Begin --------------</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">1. Please ensure turn-off iptables to all nodes to ensure it works.</div><div class="gmail_default" style="font-size:small">2. You share gluster and cluster node. but will slow down if you dont have dedicated network card to it.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">---------------------</div><div class="gmail_default" style="font-size:small">Gluster Storage - (standalone)</div><div class="gmail_default" style="font-size:small">---------------------</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><div class="gmail_default"># wget -P /etc/yum.repos.d <a href="http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo">http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo</a></div><div class="gmail_default"># yum -y install glusterfs glusterfs-fuse glusterfs-server</div><div class="gmail_default"># chkconfig glusterd on</div><div class="gmail_default"><br></div></div><div class="gmail_default" style><div class="gmail_default" style># vi /etc/glusterfs/glusterd.vol</div><div class="gmail_default" style><br></div><div class="gmail_default" style><span class="" style="white-space:pre"> </span>volume management</div><div class="gmail_default" style><span class="" style="white-space:pre"> </span> option rpc-auth-allow-insecure on</div><div class="gmail_default" style><span class="" style="white-space:pre"> </span> option bind-address 172.20.11.11 <----- change to your gluster storage ip address</div><div class="gmail_default" style><span class="" style="white-space:pre"> </span>end-volume</div><div class="gmail_default" style><br></div><div class="gmail_default" style># service glusterd start</div><div class="gmail_default" style><br></div><div class="gmail_default" style>We create glusterfs mount folder. Pls ensure the userid and groupid exist in the gluster storage</div><div class="gmail_default" style># mkdir -p /mnt/ds0/datastores0</div><div class="gmail_default" style># chown oneadmin:oneadmin /mnt/ds0/datastores0</div><div class="gmail_default" style># chmod 775 /mnt/ds0/datastores0</div><div class="gmail_default" style><br></div><div class="gmail_default" style><div class="gmail_default"># gluster volume create datastores0 172.20.11.11:/mnt/ds0/datastores0</div><div class="gmail_default"># gluster volume start datastores0</div><div class="gmail_default"># gluster volume set datastores0 server.allow-insecure on</div><div class="gmail_default"># gluster volume set datastores0 storage.owner-uid 9869</div><div class="gmail_default"># gluster volume set datastores0 storage.owner-gid 9869</div><div class="gmail_default"># gluster volume set datastores0 auth.allow 172.20.* <---- change to you ip subnet</div><div class="gmail_default"><br></div><div class="gmail_default"># gluster volume info</div></div><div class="gmail_default" style><br></div><div class="gmail_default" style><div class="gmail_default">------------------</div><div class="gmail_default">Front Node</div><div class="gmail_default">------------------</div></div><div class="gmail_default" style><div class="gmail_default"><br></div><div class="gmail_default"># wget -P /etc/yum.repos.d <a href="http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo">http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo</a></div><div class="gmail_default"># yum -y install glusterfs glusterfs-fuse</div></div><div class="gmail_default" style><br></div><div class="gmail_default" style><div class="gmail_default"># su - oneadmin</div><div class="gmail_default"><span class="" style="white-space:pre"> </span>$ cd datastores</div><div class="gmail_default"><span class="" style="white-space:pre"> </span>$ rm -rf /var/lib/one/datastores/1</div><div class="gmail_default"><span class="" style="white-space:pre"> </span>$ rm -rf /var/lib/one/datastores/2</div><div class="gmail_default"><br></div><div class="gmail_default"><div class="gmail_default"><span class="" style="white-space:pre"> </span>$ ln -s /var/lib/one/datastores/0 /var/lib/one/datastores/1</div><div class="gmail_default"><span class="" style="white-space:pre"> </span>$ ln -s /var/lib/one/datastores/0 /var/lib/one/datastores/2</div></div><div class="gmail_default"><br></div></div><div class="gmail_default" style><div class="gmail_default"><div class="gmail_default">Put permanent setting during boot</div></div><div class="gmail_default"><div class="gmail_default"># vi /etc/fstab</div><div class="gmail_default"><span class="" style="white-space:pre"> </span>172.20.11.11:/datastores0 /var/lib/one/datastores/0 glusterfs defaults,_netdev 0 0</div><div class="gmail_default"><br></div><div class="gmail_default"><div class="gmail_default"># mount -a</div><div># df -h</div><div><br></div><div>hope no error.</div></div><div class="gmail_default"><br></div><div class="gmail_default">Go sunstone n refresh datastore page. makesure you already add your first cluster node.</div><div class="gmail_default"><br></div><div class="gmail_default">in front end should mount all gluster datastores</div></div></div><div class="gmail_default" style><br></div><div class="gmail_default" style>------------------</div><div class="gmail_default" style>Cluster Node</div><div class="gmail_default" style>------------------</div><div class="gmail_default" style><div class="gmail_default"># wget -P /etc/yum.repos.d <a href="http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo">http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo</a></div><div class="gmail_default"># yum -y install glusterfs glusterfs-fuse</div><div class="gmail_default"><br></div></div><div class="gmail_default" style># su - oneadmin</div><div class="gmail_default" style><br></div><div class="gmail_default" style><span class="" style="white-space:pre"> </span>We create default folder</div><div class="gmail_default" style><span class="" style="white-space:pre"> </span>$ mkdir -p /var/lib/one/datastores/0 </div><div class="gmail_default" style><span class="" style="white-space:pre"> </span>$ ln -s /var/lib/one/datastores/0 /var/lib/one/datastores/1</div><div class="gmail_default" style><span class="" style="white-space:pre"> </span>$ ln -s /var/lib/one/datastores/0 /var/lib/one/datastores/2</div><div class="gmail_default" style><span class="" style="white-space:pre"><br></span></div><div class="gmail_default" style><span class="" style="white-space:pre"> </span>$ exit<br></div><div class="gmail_default" style><br></div><div class="gmail_default" style><span class="" style="white-space:pre"> NOTE: Next two line if you create additional datastores as i mention earlier. </span>We only create system and images datastores in cluster node</div><div class="gmail_default" style><span class="" style="white-space:pre"> </span>$ mkdir -p /var/lib/one/datastores/100</div><div class="gmail_default" style><span class="" style="white-space:pre"> </span>$ ln -s /var/lib/one/datastores/100 /var/lib/one/datastores/101</div><div class="gmail_default" style><br></div><div class="gmail_default" style><span class="" style="white-space:pre"> </span>$ exit</div><div class="gmail_default" style><br></div><div class="gmail_default" style>Put permanent setting during boot and point to system datastore. image just a symlink.</div></div><div class="gmail_default" style="font-size:small"><div class="gmail_default"># vi /etc/fstab</div><div class="gmail_default"><span class="" style="white-space:pre"> </span>172.20.11.11:/datastores0 /var/lib/one/datastores/0 glusterfs defaults,_netdev 0 0</div><div class="gmail_default"><br></div><div class="gmail_default"># mount -a</div><div># df -h</div><div><br></div><div>--------------- end ------------------</div><div><br></div><div>hope this will help you.</div></div><div class="gmail_extra"><div><p>Regards & Best Wishes,<br><br><br><b>.: Amier Anis :.</b><br><span style="font-size:x-small">Mobile: +6012-260-0819</span></p></div>
<br><div class="gmail_quote">On Wed, Sep 10, 2014 at 6:09 PM, Marco Aroldi <span dir="ltr"><<a href="mailto:marco.aroldi@gmail.com" target="_blank">marco.aroldi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi, thanks for the answer.<br>
<br>
The hypervisor hosts can telnet both clu100 and clu001 on port 24007<br>
<br>
I'm sorry, I'm a OpenNebual newbie, so it's not clear to me where to<br>
put the DISK_TYPE="GLUSTER" option<br>
So I've tired to put the option first in the image attributes, then in<br>
the template custom tags, but nothing has changed<br>
I don't get any message related to Gluster in logs<br>
<br>
I'm scratching my head.<br>
Any help is very appreciated. Thanks!<br>
<br>
Wed Sep 10 11:49:22 2014 [Z0][DiM][I]: New VM state is ACTIVE.<br>
Wed Sep 10 11:49:22 2014 [Z0][LCM][I]: New VM state is PROLOG.<br>
Wed Sep 10 11:49:46 2014 [Z0][LCM][I]: New VM state is BOOT<br>
Wed Sep 10 11:49:46 2014 [Z0][VMM][I]: Generating deployment file:<br>
/var/lib/one/vms/53/deployment.0<br>
Wed Sep 10 11:49:47 2014 [Z0][VMM][I]: ExitCode: 0<br>
Wed Sep 10 11:49:47 2014 [Z0][VMM][I]: Successfully execute network<br>
driver operation: pre.<br>
<br>
The deployment file:<br>
<br>
<domain type='kvm' xmlns:qemu='<a href="http://libvirt.org/schemas/domain/qemu/1.0" target="_blank">http://libvirt.org/schemas/domain/qemu/1.0</a>'><br>
<name>one-53</name><br>
<cputune><br>
<shares>1024</shares><br>
</cputune><br>
<memory>786432</memory><br>
<os><br>
<type arch='x86_64'>hvm</type><br>
<boot dev='hd'/><br>
</os><br>
<devices><br>
<emulator>/usr/libexec/qemu-kvm</emulator><br>
<disk type='network' device='disk'><br>
<source protocol='gluster' name='sys-one/53/disk.0'><br>
<host name='clu100' port='24007'<br>
transport='tcp'/><br>
</source><br>
<target dev='hda'/><br>
<driver name='qemu' type='qcow2' cache='none'/><br>
</disk><br>
<disk type='file' device='cdrom'><br>
<source file='/var/lib/one//datastores/110/53/disk.1'/><br>
<target dev='hdb'/><br>
<readonly/><br>
<driver name='qemu' type='raw'/><br>
</disk><br>
<interface type='bridge'><br>
<source bridge='br0'/><br>
<mac address='02:00:c0:a8:16:7e'/><br>
</interface><br>
<graphics type='vnc' listen='0.0.0.0' port='5953'/><br>
</devices><br>
<features><br>
<acpi/><br>
</features><br>
</domain><br>
<br>
<br>
oneimage show:<br>
<br>
IMAGE 19 INFORMATION<br>
ID : 19<br>
NAME : CentOS-6.5-one-4.8_GLUSTER<br>
USER : oneadmin<br>
GROUP : oneadmin<br>
DATASTORE : GLUSTER<br>
TYPE : OS<br>
REGISTER TIME : 08/30 10:41:12<br>
PERSISTENT : No<br>
SOURCE : /var/lib/one//datastores/108/c6780b5c1667ec829b9ed92f7853f934<br>
PATH :<br>
<a href="http://marketplace.c12g.com/appliance/53e767ba8fb81d6a69000001/download/0" target="_blank">http://marketplace.c12g.com/appliance/53e767ba8fb81d6a69000001/download/0</a><br>
SIZE : 10G<br>
STATE : used<br>
RUNNING_VMS : 1<br>
<br>
PERMISSIONS<br>
OWNER : um-<br>
GROUP : ---<br>
OTHER : ---<br>
<br>
IMAGE TEMPLATE<br>
DEV_PREFIX="hd"<br>
DISK_TYPE="GLUSTER"<br>
FROM_APP="53e767ba8fb81d6a69000001"<br>
FROM_APP_FILE="0"<br>
FROM_APP_NAME="CentOS 6.5 - KVM - OpenNebula 4.8"<br>
MD5="9d937b8fe70c403330c9284538f07cfc"<br>
<br>
VIRTUAL MACHINES<br>
<br>
ID USER GROUP NAME STAT UCPU UMEM HOST TIME<br>
53 oneadmin oneadmin CentOS 6.5 - KV boot 0 768M pre-openne 0d 00h12<br>
<br>
<br>
onetemplate show<br>
<br>
TEMPLATE 8 INFORMATION<br>
ID : 8<br>
NAME : CentOS 6.5 - KVM - OpenNebula 4.8_GLSUTER<br>
USER : oneadmin<br>
GROUP : oneadmin<br>
REGISTER TIME : 08/30 10:41:13<br>
<br>
PERMISSIONS<br>
OWNER : um-<br>
GROUP : ---<br>
OTHER : ---<br>
<br>
TEMPLATE CONTENTS<br>
CONTEXT=[<br>
NETWORK="YES",<br>
SSH_PUBLIC_KEY="$USER[SSH_PUBLIC_KEY]" ]<br>
CPU="1"<br>
DISK=[<br>
DRIVER="qcow2",<br>
IMAGE="CentOS-6.5-one-4.8_GLUSTER",<br>
IMAGE_UNAME="oneadmin" ]<br>
DISK_TYPE="GLUSTER"<br>
FROM_APP="53e767ba8fb81d6a69000001"<br>
FROM_APP_NAME="CentOS 6.5 - KVM - OpenNebula 4.8"<br>
GRAPHICS=[<br>
LISTEN="0.0.0.0",<br>
TYPE="vnc" ]<br>
MEMORY="768"<br>
NIC=[<br>
NETWORK="private",<br>
NETWORK_UNAME="oneadmin" ]<br>
<div class=""><div class="h5"><br>
<br>
2014-09-09 12:38 GMT+02:00 Javier Fontan <<a href="mailto:jfontan@opennebula.org">jfontan@opennebula.org</a>>:<br>
> That's right. Even if it is using GlusterFS the way of accessing the<br>
> files is using the fuse filesystem. This makes the IO performance<br>
> suffer.<br>
><br>
> Do you get any errors in the log files related to gluster when you try<br>
> to boot a machine with DISK_TYPE="GLUSTER" is activated? It could be a<br>
> firewall or a permission problem. Make sure that the hypervisor host<br>
> can access clu100 port 24007.<br>
><br>
> Also make sure that the server has the rpc-auth-allow-insecure option<br>
> configured and was restarted after the change.<br>
><br>
> On Sat, Aug 30, 2014 at 11:44 AM, Marco Aroldi <<a href="mailto:marco.aroldi@gmail.com">marco.aroldi@gmail.com</a>> wrote:<br>
>> Hi all,<br>
>> this is my first post to the list<br>
>><br>
>> My goal is to get ONE 4.8 up and running using Gluster as datastore,<br>
>> everything on CentOS 6.5<br>
>> The problem: the VM remains stuck in BOOT status<br>
>> I've found a way to boot the machines (see below), but I think is not<br>
>> the correct way to manage this setup.<br>
>><br>
>> First, let me describe what I've done until now:<br>
>> I've followed the docs at<br>
>> <a href="http://docs.opennebula.org/4.8/administration/storage/gluster_ds.html" target="_blank">http://docs.opennebula.org/4.8/administration/storage/gluster_ds.html</a><br>
>> and the post on the blog by Javier Fontan<br>
>> <a href="http://opennebula.org/native-glusterfs-image-access-for-kvm-drivers/" target="_blank">http://opennebula.org/native-glusterfs-image-access-for-kvm-drivers/</a><br>
>><br>
>> This is my Gluster volume:<br>
>> Volume Name: sys-one<br>
>> Type: Replicate<br>
>> Volume ID: f1bf1bcc-0280-46db-aab8-69fd34672263<br>
>> Status: Started<br>
>> Number of Bricks: 1 x 2 = 2<br>
>> Transport-type: tcp<br>
>> Bricks:<br>
>> Brick1: clu001:/one<br>
>> Brick2: clu100:/one<br>
>> Options Reconfigured:<br>
>> cluster.server-quorum-type: server<br>
>> cluster.quorum-type: auto<br>
>> network.remote-dio: enable<br>
>> cluster.eager-lock: enable<br>
>> performance.stat-prefetch: on<br>
>> performance.io-cache: off<br>
>> performance.read-ahead: off<br>
>> performance.quick-read: off<br>
>> storage.owner-gid: 9869<br>
>> storage.owner-uid: 9869<br>
>> server.allow-insecure: on<br>
>><br>
>> And the datastores:<br>
>> ID NAME SIZE AVAIL CLUSTER IMAGES TYPE DS TM<br>
>> 1 default 230.7G 86% - 6 img fs shared<br>
>> 2 files 230.7G 86% - 0 fil fs ssh<br>
>> 108 GLUSTER 24G 52% clussssss 2 img fs shared<br>
>> 110 new system 24G 52% clussssss 0 sys - shared<br>
>><br>
>><br>
>> DATASTORE 108 INFORMATION<br>
>> ID : 108<br>
>> NAME : GLUSTER<br>
>> USER : oneadmin<br>
>> GROUP : oneadmin<br>
>> CLUSTER : clussssss<br>
>> TYPE : IMAGE<br>
>> DS_MAD : fs<br>
>> TM_MAD : shared<br>
>> BASE PATH : /var/lib/one//datastores/108<br>
>> DISK_TYPE :<br>
>><br>
>> DATASTORE CAPACITY<br>
>> TOTAL: : 24G<br>
>> FREE: : 12.5G<br>
>> USED: : 6.3G<br>
>> LIMIT: : 12.7G<br>
>><br>
>> PERMISSIONS<br>
>> OWNER : um-<br>
>> GROUP : u--<br>
>> OTHER : ---<br>
>><br>
>> DATASTORE TEMPLATE<br>
>> BASE_PATH="/var/lib/one//datastores/"<br>
>> CLONE_TARGET="SYSTEM"<br>
>> DISK_TYPE="GLUSTER"<br>
>> DS_MAD="fs"<br>
>> GLUSTER_HOST="clu100:24007"<br>
>> GLUSTER_VOLUME="sys-one"<br>
>> LIMIT_MB="13000"<br>
>> LN_TARGET="NONE"<br>
>> TM_MAD="shared"<br>
>> TYPE="IMAGE_DS"<br>
>><br>
>> DATASTORE 110 INFORMATION<br>
>> ID : 110<br>
>> NAME : new system<br>
>> USER : oneadmin<br>
>> GROUP : oneadmin<br>
>> CLUSTER : clussssss<br>
>> TYPE : SYSTEM<br>
>> DS_MAD : -<br>
>> TM_MAD : shared<br>
>> BASE PATH : /var/lib/one//datastores/110<br>
>> DISK_TYPE : FILE<br>
>><br>
>> DATASTORE CAPACITY<br>
>> TOTAL: : 24G<br>
>> FREE: : 12.5G<br>
>> USED: : 6.3G<br>
>> LIMIT: : -<br>
>><br>
>> PERMISSIONS<br>
>> OWNER : um-<br>
>> GROUP : u--<br>
>> OTHER : ---<br>
>><br>
>> DATASTORE TEMPLATE<br>
>> BASE_PATH="/var/lib/one//datastores/"<br>
>> SHARED="YES"<br>
>> TM_MAD="shared"<br>
>> TYPE="SYSTEM_DS"<br>
>><br>
>><br>
>> Here is the mounted glusterfs:<br>
>> clu100:/sys-one on /gluster type fuse.glusterfs<br>
>> (rw,default_permissions,allow_other,max_read=131072)<br>
>><br>
>> And the symbolic links in the datastores directory:<br>
>> lrwxrwxrwx 1 oneadmin oneadmin 8 Aug 30 10:18 108 -> /gluster<br>
>> lrwxrwxrwx 1 oneadmin oneadmin 8 Aug 30 10:18 110 -> /gluster<br>
>><br>
>> I've found the culprit in the system datastore:<br>
>> Created a new system datastore ON THE LOCAL FILESYSTEM:<br>
>><br>
>> 111 system 230.7G 86% - 0 sys - shared<br>
>><br>
>> DATASTORE 111 INFORMATION<br>
>> ID : 111<br>
>> NAME : system<br>
>> USER : oneadmin<br>
>> GROUP : oneadmin<br>
>> CLUSTER : -<br>
>> TYPE : SYSTEM<br>
>> DS_MAD : -<br>
>> TM_MAD : shared<br>
>> BASE PATH : /var/lib/one//datastores/111<br>
>> DISK_TYPE : FILE<br>
>><br>
>> DATASTORE CAPACITY<br>
>> TOTAL: : 230.7G<br>
>> FREE: : 199.2G<br>
>> USED: : 1M<br>
>> LIMIT: : -<br>
>><br>
>> PERMISSIONS<br>
>> OWNER : um-<br>
>> GROUP : u--<br>
>> OTHER : ---<br>
>><br>
>> DATASTORE TEMPLATE<br>
>> BASE_PATH="/var/lib/one//datastores/"<br>
>> SHARED="YES"<br>
>> TM_MAD="shared"<br>
>> TYPE="SYSTEM_DS"<br>
>><br>
>> Deploying now puts the VM in RUNNING status but, correct me if I'm<br>
>> wrong, this setup is not compliant, right?<br>
>> Thanks for the help<br>
>><br>
>> Marco<br>
>> _______________________________________________<br>
>> Users mailing list<br>
>> <a href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a><br>
>> <a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
><br>
><br>
><br>
> --<br>
> Javier Fontán Muiños<br>
> Developer<br>
> OpenNebula - Flexible Enterprise Cloud Made Simple<br>
> <a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | @OpenNebula | <a href="http://github.com/jfontan" target="_blank">github.com/jfontan</a><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
</div></div></blockquote></div><br></div></div>