[one-users] [Docs] Problem with OpenVSwitch and some questions about Ceph
Campbell, Bill
bcampbell at axcess-financial.com
Wed Apr 10 06:04:14 PDT 2013
I know this is WAY delayed, but regarding your Ceph question:
OpenNebula is capable of doing all of your RBD image management (creation/deletion/etc.), so you shouldn't need to manually create any images on the back end, and you definitely do not need to use CephFS (not really ready for production use yet anyway).
In our particular setup we do the following:
* We change the 'system' datastore to use the 'SSH' transfer manager, so we don't have to worry about linking up the OpenNebula system to the Ceph cluster (we use a separate 10Gb network for storage). We have to modify the premigrate/postmigrate scripts for the SSH transfer manager to ensure VM deployment information is copied to each KVM host (else migrations fail).
* Have a Ceph cluster created, and copy the ceph.conf and ceph.keyring to the /etc/ceph directory of every node in the cluster (minus the OpenNebula system), and ensure that the oneadmin user you create has at a minimum read permissions on the file (we change ownership to root.oneadmin and apply 640 permissions). I know on the roadmap for 4.2 is a feature for making the integration with Ceph more secure (without relying on the keyring being on each node, and security being defined within each Libvirt deployment file directly).
* We have a dedicated system for management of images that OpenNebula connects to (as we've found connecting directly to a monitor sometimes seems to cause issues when dealing with large images, nothing catastrophic but the mon daemon sometimes stops and restarts. Still chasing that one down.) and define that dedicate system in the ceph.conf file in the DS mad.
* A oneadmin account is created as per documentation on the OpenNebula system, the KVM Hosts, and the dedicated image management system. All SSH keys are transferred to this dedicated system as well.
* The default storage pool is 'one', so you can have a datastore created without defining the pool name, create a pool named 'one' in the Ceph cluster and you should be off to the races. The nice thing about the Ceph implementation in OpenNebula is that you can define a pool name for each datastore created, so that way when you start using larger deployments of OpenNebula/Ceph, you can segment off your VMs to different Datastores/Pools (for instance, if you have an array/pool with dedicated SSDs for extremely fast I/O, you can have that in the same cluster, just defined in a separate Datastore/Pool, with your CRUSH map configured accordingly). Ceph is very extensible.
We've been using Ceph with OpenNebula since 3.8.3 with our own driver and it has been working very well for us. The testing of the integration with 4.0 has been going well for us (we made some slight modifications that we submitted to take advantage of RBD Format 2 images for copy-on-write cloning of non-persistent images) and so far have only found a couple of issues (which are already resolved).
If you have any additional questions please don't hesitate to ask!
----- Original Message -----
From: "Jaime Melis" <jmelis at opennebula.org>
To: "Jon" <three18ti at gmail.com>
Cc: "Users OpenNebula" <users at lists.opennebula.org>
Sent: Wednesday, April 10, 2013 6:37:30 AM
Subject: Re: [one-users] [Docs] Problem with OpenVSwitch and some questions about Ceph
Hi Jon
sorry for the delay in the answer.
You can always contibute to the community wiki: http://wiki.opennebula.org/
thanks!
cheers,
Jaime
On Sun, Mar 31, 2013 at 10:11 PM, Jon < three18ti at gmail.com > wrote:
Hello Jamie,
Thanks for the clarification, I didn't realize that install_novnc.sh being in two different locations was a bug.
I hadn't installed OpenNebula on anything other than a Debian / Ubuntu system, so this makes sense.
is there any userspace documentation we can contribute to? Even if not the documentation itself, perhaps a wiki? I didn't see one but that doesn't mean there isn't one.
Thanks,
Jon A
On Thu, Mar 28, 2013 at 2:56 PM, Jaime Melis < jmelis at opennebula.org > wrote:
3) Documentation
The fact that install_novnc.sh is being installed to two separate locations is a bug that has already been fixed.
With regard to the "/usr/share/opennebula" issue, I'm afraid that's because of the Debian/Ubuntu packaging policy. For other distros (CentOS and openSUSE for example) the path is the one that appears in the documentation.
To makes things easier for users I think this should be reflected in the README.debian file, and in the platform notes in the documentation. So thanks a lot for pointing this out.
I created this feature to follow this problem:
http://dev.opennebula.org/issues/1844
cheers,
Jaime
On Thu, Mar 28, 2013 at 1:06 PM, Jon < three18ti at gmail.com > wrote:
Hello All,
I've just installed OpenNebula 3.9.80 and I have to say this is
amazing. Everything works so smoothly.
Anyway, down to business.
OpenVSwitch:
I've installed and configured OpenVSwitch and am able to manually add
the OVS config using libvirt, then launch a VM,
>> <interface type='bridge'>
>> <source bridge='ovsbr0'/>
>> <virtualport type='openvswitch'>
>> </virtualport>
>> <model type='virtio'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
>> </interface>
Which creates the device in ovs:
>> system at ovsbr0:
>> lookups: hit:1346333 missed:46007 lost:0
>> flows: 8
>> port 0: ovsbr0 (internal)
>> port 1: eth0
>> port 2: br0 (internal)
>> port 7: vnet0
However, when I attempt to create a virtual network without assigning
an IP and instantiate the template I get the error:
>> [TemplateInstantiate] Error allocating a new virtual machine. Cannot get IP/MAC lease from virtual network 0.
The template of the virtual network is:
>> oneadmin at loki:~$ onevnet show testnet1
>> VIRTUAL NETWORK 0 INFORMATION
>> ID : 0
>> NAME : testnet1
>> USER : oneadmin
>> GROUP : oneadmin
>> CLUSTER : -
>> TYPE : FIXED
>> BRIDGE : ovsbr0
>> VLAN : No
>> USED LEASES : 0
>>
>> PERMISSIONS
>> OWNER : um-
>> GROUP : ---
>> OTHER : ---
>>
>> VIRTUAL NETWORK TEMPLATE
>>
>>
>> VIRTUAL MACHINES
>>
If I add an IP to the vnet, I get the following template and error
logs (full vm log attached, I think I've identified the relevant
line):
>> Thu Mar 28 10:34:05 2013 [VMM][E]: post: Command "sudo /usr/bin/ovs-ofctl add-flow ovsbr0 in_port=,dl_src=02:00:44:47:83:43,priority=40000,actions=normal" failed.
>> oneadmin at loki:~$ onevnet show testnet1
>> VIRTUAL NETWORK 0 INFORMATION
>> ID : 0
>> NAME : testnet1
>> USER : oneadmin
>> GROUP : oneadmin
>> CLUSTER : -
>> TYPE : FIXED
>> BRIDGE : ovsbr0
>> VLAN : No
>> USED LEASES : 1
>>
>> PERMISSIONS
>> OWNER : um-
>> GROUP : ---
>> OTHER : ---
>>
>> VIRTUAL NETWORK TEMPLATE
>>
>>
>> USED LEASES
>> LEASE=[ MAC="02:00:44:47:83:43", IP="192.168.0.2", IP6_LINK="fe80::400:44ff:fe47:8343", USED="1", VID="7" ]
>>
>> VIRTUAL MACHINES
>>
>> ID USER GROUP NAME STAT UCPU UMEM HOST TIME
>> 7 oneadmin oneadmin template-4-7 fail 0 0K 0d 00h00
>> root at loki:~# cat /var/log/openvswitch/ovs-vswitchd.log
>> Mar 28 10:34:04|00081|bridge|INFO|created port vnet1 on bridge ovsbr0
>> Mar 28 10:34:07|00082|netdev_linux|WARN|ethtool command ETHTOOL_GSET on network device vnet1 failed: No such device
>> Mar 28 10:34:07|00083|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on vnet1 device failed: No such device
>> Mar 28 10:34:07|00084|netdev|WARN|failed to get flags for network device vnet1: No such device
>> Mar 28 10:34:07|00085|netdev|WARN|failed to retrieve MTU for network device vnet1: No such device
>> Mar 28 10:34:07|00086|netdev|WARN|failed to get flags for network device vnet1: No such device
>> Mar 28 10:34:07|00087|bridge|INFO|destroyed port vnet1 on bridge ovsbr0
I attempted to run the command but I never set a password for the
oneadmin user, but I don't think it's a permissions / sudo access
problem.
Not really sure where to look next. Any ideas are appreciated.
CEPH:
I'm trying to use a Ceph datastore with a RBD instead of a cephFS
(it's an option).
When I try to create a Ceph datastore with a RBD type, I get a state
of "Error",
but I'm not sure where to look for relevant logs, oned.log didn't seem
to have anything,
or maybe I'm just grepping for the wrong string.
As a work around, I have been creating the directory, creating the rbd
then manually mounting it,
this seems to work in my test environment, but doesn't seem very
scalable, how are others using CEPH?
Documentation:
I've noticed some errors in the documentation, namely the location of
the install scripts,
The docs state they are in:
>> /usr/share/one/install_gems
>> /usr/share/one/sunstone/install_novnc.sh
However, I found them in:
>> /usr/share/opennebula/install_gems
>> /usr/share/opennebula/install_novnc.sh
>> /usr/share/opennebula/sunstone/install_novnc.sh
Is there some repository of the documentation somewhere that we can
contribute to?
It's a small thing, but when I'm going through the instructions, I
like to copy / paste;
I figured it out, but I know if it caused me problems, it might cause
others problems too.
Thanks again, I can't wait for the final release of OpenNebula!
Best Regards,
Jon A
_______________________________________________
Users mailing list
Users at lists.opennebula.org
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
--
Jaime Melis
Project Engineer
OpenNebula - The Open Source Toolkit for Cloud Computing
www.OpenNebula.org | jmelis at opennebula.org
--
Jaime Melis
Project Engineer
OpenNebula - The Open Source Toolkit for Cloud Computing
www.OpenNebula.org | jmelis at opennebula.org
_______________________________________________
Users mailing list
Users at lists.opennebula.org
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
NOTICE: Protect the information in this message in accordance with the company's security policies. If you received this message in error, immediately notify the sender and destroy all copies.
More information about the Users
mailing list