[one-users] virtual network cannot get out
Neil Schneider
neil at ifxonline.com
Fri May 16 11:46:17 PDT 2014
On Tue, May 13, 2014 12:01 am, Valentin Bud wrote:
> Hello Neil,
>
> Please see in line for answer and additional question that would help us
> to nail down the problem.
>
> On Mon, May 12, 2014 at 9:42 PM, Neil Schneider <neil at ifxonline.com>
> wrote:
>
>> More data points.
>>
>> I ran tcpdump on the host through the entire instantiation of a host and
>> boot up. Although somehow it appeared to get an address from the DHCP
>> server in the network, I saw no dhcp data. I could see arp conversations
>> for other hosts on the network, so I would expect it to see everything,
>> if
>> it passed out of the host interface. The only outside connection through
>> the bridge should be through em1 on the host.
>>
>> tcpdump -nnvvXXSs 1514 -w cloud1.pcap
>>
>
> I think the above would listen and capture packets on the first Ethernet
> interface
> *only*. I have tested it on my PC, a Debian Wheezy machine with 2 Ethernet
> cards
> and that's what it does. I don't know what OS are you using so your
> mileage
> may vary.
Using CentOS release 6.5. I reran tcpdump and specified the interface as
the bridge device. Although I gathered 6.5Gb of data, the result was the
same. Only saw dhcp client server connection, no further networking. I did
manage to capture the dhcp exchange. And I was able to see the leases in
the dhcp server. I may have just missed them on first pass. There are only
a few packets out of all that data. I had to split the tcpdump file, to
analyze it with wireshark.
>> I used wireshark to look through the file, but did not see any DHCP
>> conversation, in fact I saw no network traffic whatsoever from the
>> virtual
>> host. I'm beginning to think either I have the virtual network
>> misconfigured or opennebula is doing something different than I expect.
>>
>
> See my comment above about tcpdump. What bridge do you expect your
> VM to connect to? Pass -i <bridge-name> to tcpdump and recheck.
Did.
>>
>> onevnet list
>> ID USER GROUP NAME CLUSTER TYPE BRIDGE
>> LEASES
>> 0 oneadmin oneadmin management ifx-produc R
>> manageme
>> 0
>> 1 oneadmin oneadmin storage ifx-produc R storage
>> 0
>> 6 oneadmin oneadmin public ifx-produc R public
>> 0
>>
>> onevnet show 0
>> VIRTUAL NETWORK 0 INFORMATION
>> ID : 0
>> NAME : management
>> USER : oneadmin
>> GROUP : oneadmin
>> CLUSTER : ifx-production
>> TYPE : RANGED
>> BRIDGE : management
>> VLAN : Yes
>> VLAN ID : 10
>> USED LEASES : 0
>>
>> PERMISSIONS
>> OWNER : uma
>> GROUP : uma
>> OTHER : u--
>>
>> VIRTUAL NETWORK TEMPLATE
>> NETWORK_ADDRESS="10.1.4.0"
>> NETWORK_MASK="255.255.255.0"
>>
>> RANGE
>> IP_START : 10.1.4.30
>> IP_END : 10.1.4.40
>>
>> VIRTUAL MACHINES
>>
>> onevnet show 1
>> VIRTUAL NETWORK 1 INFORMATION
>> ID : 1
>> NAME : storage
>> USER : oneadmin
>> GROUP : oneadmin
>> CLUSTER : ifx-production
>> TYPE : RANGED
>> BRIDGE : storage
>> VLAN : Yes
>> VLAN ID : 20
>> USED LEASES : 0
>>
>> PERMISSIONS
>> OWNER : uma
>> GROUP : uma
>> OTHER : u--
>>
>> VIRTUAL NETWORK TEMPLATE
>> NETWORK_ADDRESS="10.1.2.0"
>> NETWORK_MASK="255.255.255.0"
>>
>> RANGE
>> IP_START : 10.1.2.30
>> IP_END : 10.1.2.40
>>
>> VIRTUAL MACHINES
>>
>> onevnet show 6
>> VIRTUAL NETWORK 6 INFORMATION
>> ID : 6
>> NAME : public
>> USER : oneadmin
>> GROUP : oneadmin
>> CLUSTER : ifx-production
>> TYPE : RANGED
>> BRIDGE : public
>> VLAN : No
>> USED LEASES : 0
>>
>> PERMISSIONS
>> OWNER : uma
>> GROUP : um-
>> OTHER : u--
>>
>> VIRTUAL NETWORK TEMPLATE
>> DNS="172.16.168.1"
>> GATEWAY="172.16.168.1"
>> NETWORK_ADDRESS="172.16.168.0"
>> NETWORK_MASK="255.255.255.0"
>>
>> RANGE
>> IP_START : 172.16.168.30
>> IP_END : 172.16.168.49
>>
>> VIRTUAL MACHINES
>>
>
> As I can see on the above onevnet output none of your virtual network have
> a VM connected to it. It should appear under VIRTUAL MACHINES section in
> one of your vnets. Have you skipped that output or wasn't any VM running
> at
> the time you've printed the output.
No vm was running. When one is running I see this.
ovs-vsctl list-ports public
em1
vnet0
ovs-vsctl iface-to-br vnet0
public
> A couple of more question regarding your setup follow.
>
> 1. Have you created the host on which you want to deploy VMs using onehost
> create
> with the proper network driver, ovswitch or ovswitch_brcompat. It could
> help the output
> of onehost list and onehost show <id>.
onehost list
ID NAME CLUSTER RVM ALLOCATED_CPU ALLOCATED_MEM STAT
0 cloud1 ifx-produ 1 100 / 1200 (8%) 2G / 31.3G (6%) on
onehost show 0
HOST 0 INFORMATION
ID : 0
NAME : cloud1
CLUSTER : ifx-production
STATE : MONITORED
IM_MAD : kvm
VM_MAD : kvm
VN_MAD : ovswitch
LAST MONITORING TIME : 05/16 10:57:34
HOST SHARES
TOTAL MEM : 31.3G
USED MEM (REAL) : 2.1G
USED MEM (ALLOCATED) : 2G
TOTAL CPU : 1200
USED CPU (REAL) : 9
USED CPU (ALLOCATED) : 100
RUNNING VMS : 1
LOCAL SYSTEM DATASTORE #102 CAPACITY
TOTAL: : 38.4G
USED: : 10G
FREE: : 6.5G
MONITORING INFORMATION
ARCH="x86_64"
CPUSPEED="1899"
HOSTNAME="cloud1"
HYPERVISOR="kvm"
MODELNAME="Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz"
NETRX="0"
NETTX="0"
VERSION="4.4.1"
VIRTUAL MACHINES
ID USER GROUP NAME STAT UCPU UMEM HOST
TIME
59 pacneil oneadmin ifx3ws1 runn 1 2G cloud1 2d
23h52
> 2. Do you want/need to use DHCP in the VM? How are you going to keep in
> sync the
> MAC - IP pair from OpenNebula with the one from your DHCP server?
I'm still setting up and testing, trying to make sure everything is
working before I put the systems in the data center. When the systems are
in the rack we will use fixed addresses. Now I'm using dhcp because it's
there and I know what addresses it will get.
> 3. What OS are you running in the hosts, what about the VMS?
Everything is CentOS 6.5.
> 4. What are the logs of the VM, on the frontend, saying? You can find them
> at /var/log/one/<VMID>.log.
This is interesting. I have the configuration in OpenNebula to run dhcp on
the bridge.
onevnet show 6
VIRTUAL NETWORK 6 INFORMATION
ID : 6
NAME : public
USER : oneadmin
GROUP : oneadmin
CLUSTER : ifx-production
TYPE : RANGED
BRIDGE : public
VLAN : No
USED LEASES : 1
PERMISSIONS
OWNER : uma
GROUP : um-
OTHER : u--
VIRTUAL NETWORK TEMPLATE
DNS="172.16.168.1"
GATEWAY="172.16.168.1"
NETWORK_ADDRESS="172.16.168.0"
NETWORK_MASK="255.255.255.0"
RANGE
IP_START : 172.16.168.30
IP_END : 172.16.168.49
USED LEASES
LEASE=[ MAC="02:00:ac:10:a8:20", IP="172.16.168.32",
IP6_LINK="fe80::400:acff:fe10:a820", USED="1", VID="59" ]
VIRTUAL MACHINES
ID USER GROUP NAME STAT UCPU UMEM HOST
TIME
59 pacneil oneadmin ifx3ws1 runn 1 2G cloud1 3d
00h04
And the logs show that it's getting it's address as it should from the
bridge.
The logs say it's happening.
Tue May 13 11:07:35 2014 [VMM][I]: Successfully execute network driver
operation: pre.
Tue May 13 11:07:39 2014 [VMM][I]: ExitCode: 0
Tue May 13 11:07:39 2014 [VMM][I]: Successfully execute virtualization
driver operation: deploy.
Tue May 13 11:07:40 2014 [VMM][I]: post: Executed "sudo ovs-ofctl add-flow
public
in_port=14,arp,dl_src=02:00:ac:10:a8:20,priority=45000,actions=drop".
Tue May 13 11:07:40 2014 [VMM][I]: post: Executed "sudo ovs-ofctl add-flow
public
in_port=14,arp,dl_src=02:00:ac:10:a8:20,nw_src=172.16.168.32,priority=46000,actions=normal".
Tue May 13 11:07:40 2014 [VMM][I]: post: Executed "sudo ovs-ofctl add-flow
public in_port=14,dl_src=02:00:ac:10:a8:20,priority=40000,actions=normal".
Tue May 13 11:07:40 2014 [VMM][I]: post: Executed "sudo ovs-ofctl add-flow
public in_port=14,priority=39000,actions=drop".
Tue May 13 11:07:40 2014 [VMM][I]: ExitCode: 0
Tue May 13 11:07:40 2014 [VMM][I]: Successfully execute network driver
operation: post.
Tue May 13 11:07:40 2014 [LCM][I]: New VM state is RUNNING
However that's not what ifconfig on the virtual host shows. Can't cut and
paste, because the only access I have to the VM is through virt-manager.
Even VNC doesn't work in Sunstone for this host.
inet addr:172.16.168.154 Bcast:172.16.168.255 Mask:255.255.255.0
So one thinks it's getting an address from the public bridge and it's
actually getting it from the network dhcp server where the host lives.
> 5. What are the Open vSwitch logs saying on the hosts your VM get deployed
> onto?
2014-05-13T18:07:37.150Z|00190|bridge|INFO|bridge public: added interface
vnet0 on port 14
2014-05-13T18:07:49.990Z|00191|ofproto|INFO|public: 4 flow_mods 10 s ago
(4 adds)
It appears there that ONE thinks the VM is getting an address from ONE,
while it's actually getting another from the external dhcp server. I'm not
sure how this effects them internally, or how openvswitch will try to
route them. I checked the routing table and there was no default route. I
manually added it, but it made no difference, the VM still can't ping any
other hosts on the network.
I have instantiated hosts on an identical host cloud2 running openvswitch,
but using virsh or virt-manager, and they start and run fine. They connect
to the network and can route traffic fine. I have 6 VMs running on cloud2
and have tested them and they all work fine. I can ssh to them, they
network fine. When I instantiate a VM using ONE on cloud1, the host I'm
working with, network fails to work.
--
Neil Schneider
Systems Administrator
IFX
12750 High Bluff Drive, Suite 460
San Diego, CA 92130
Phone 858-724-1024 | Fax 858-724-1043 | http://www.ifxonline.com
More information about the Users
mailing list