Hello Sean,<div><br></div><div>well, as you say it would be better to ask this to some iptables guru.</div><div><br></div><div>I'm assuming that the rule that isn't working is the PREROUTING rule, since the forward rule can be double checked by changing the default policy to ACCEPT.</div>
<div><br></div><div>If it's the PREROUTING rule I think the problem might be the '-d' parameter. Were you specifying it correctly?</div><div><br></div><div>Also, did you have the static route in your host to reach the NAT network?</div>
<div><br></div><div>cheers,<br>Jaime<br><br><div class="gmail_quote">On Mon, May 14, 2012 at 5:19 PM, Sean Abbott <span dir="ltr"><<a href="mailto:seabbott@akamai.com" target="_blank">seabbott@akamai.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This works!<br>
<br>
Ok, so I didn't need to add the route, because that was already there,<br>
presumably placed there by libvirt because in order for their default<br>
route to work it needs to be there.<br>
<br>
This setup is only minorly different than a number of different things I<br>
tried while playing with this last week, and although I know this may<br>
not be the correct forum, I would *REALLY* like to understand why this<br>
worked and what I did last week did not. (Unfortunately, I did not keep<br>
as good of record as I should have on what I did do)<br>
<br>
I've been doing iptables by using iptables-apply (and the one time I<br>
didn't I was trying to add some rules to log traffic and managed to add<br>
a tcp DROP that caused my machine to become totally unreachable.<br>
<br>
So, I've been iptables-save > somefile.iptables; cp somefile.iptables<br>
newfile.iptables, modding newfile.iptables, and then doing an<br>
# iptables-apply -t 60 newfile.iptables.<br>
<br>
Last week, I tried a number of variations. There were never any rules<br>
in the nat table, so last week I would add lines like:<br>
<br>
*nat<br>
:PREROUTING ACCEPT [2449896:<a href="tel:2486671730" value="+12486671730">2486671730</a>]<br>
-A PREROUTING -p tcp -d <external IP> --dport 4545 -j DNAT --to<br>
<a href="http://192.168.122.10:4545" target="_blank">192.168.122.10:4545</a><br>
<br>
then for the filter tables, in the FORWARD chain, before any other<br>
FORWARD rueles, I was adding things like:<br>
:FORWARD ACCEPT [5957629:2882482096]<br>
-A FORWARD - tcp -d <a href="http://192.168.122.10/32" target="_blank">192.168.122.10/32</a> -o virbr0 -d 192.168.122.10<br>
--dport 4545 -m state --state NEW -j ACCEPT<br>
<br>
(and, understand, RELATED and established I thought would be covered by<br>
this existing rule, placed automatically by libvirt:)<br>
-A FORWARD -d <a href="http://192.168.122.0/24" target="_blank">192.168.122.0/24</a> -o virbr0 -m state --state<br>
RELATED,ESTABLISHED -j ACCEPT<br>
<br>
And there are numerious other varitions I've tried.<br>
<br>
Now, this time, per your recommendation, I DID use -I instead of -A in<br>
both rules. And, for the FORWARD rule, I used the whole network rather<br>
than the specific IP address.<br>
<br>
Can you or someone else please try and explain WHY your rules worked,<br>
and, using the examples that I've provided, why my old attempts did not?<br>
<br>
Again, I know this is quite possibly the wrong forum for these questions<br>
and if so, I apologize and I'll try and write something to an iptables<br>
forum. I just really need to be able to understand what's happening so<br>
I can be more able to accomplish this on my own/faster next time.<br>
<br>
Thanks so much for your help! It's great to get this working!<br>
<br>
(I actually removed my ability to connect to my onemaster with my early<br>
ill-placed DROP command, so I won't be able to work on getting the hooks<br>
or custom network drivers until I get that back, and I'm relying on some<br>
rather busy remote lab minions to recover that machine, so it may be a<br>
day or two.)<br>
<br>
Thanks again!<br>
<br>
sean<br>
<div class="im"><br>
On 05/14/2012 07:21 AM, Jaime Melis wrote:<br>
> Hello Sean,<br>
><br>
> ok, I think we understand what you mean.<br>
><br>
> There are two things here: (1) Getting the host to forward the port with<br>
> iptables (2) doing it automatically with OpenNebula using customized network<br>
> drivers:<br>
><br>
> Let's start by achieving this manually and we can move on later on to doing it<br>
> automatically with OpenNebula:<br>
><br>
> To do this you need to do the following in the host that's running the gateway VM:<br>
><br>
> * add an iptables rule to forward to the desired vm:<br>
> iptables -t nat -I PREROUTING -p tcp --dport <PORT> -j DNAT --to-destination<br>
> <guest_vm_ip>:<PORT><br>
><br>
> * allow forwarding:<br>
> echo "1" > /proc/sys/net/ipv4/ip_forward<br>
> iptables -I FORWARD -m state -d <libvirt_nat_network>/24 --state<br>
> NEW,RELATED,ESTABLISHED -j ACCEPT<br>
><br>
> * add a static route:<br>
> route add -net <libvirt_nat_network> netmask 255.255.255.0 dev <BRIDGE><br>
><br>
> Can you try this and let us know if it works?<br>
><br>
> Cheers,<br>
> Jaime<br>
><br>
> On Fri, May 11, 2012 at 7:08 PM, Sean Abbott <<a href="mailto:seabbott@akamai.com">seabbott@akamai.com</a><br>
</div><div class="im">> <mailto:<a href="mailto:seabbott@akamai.com">seabbott@akamai.com</a>>> wrote:<br>
><br>
> The hosts connect to each other through a backend network. They're all<br>
> on the same switch, and they use a bridge to that backend network to<br>
> provide inter-vm connectivity.<br>
><br>
> So, in the templates below, fullbackend is connected to a bridge that<br>
> goes to a private network. natnet0 is connected to virbr0, which does<br>
> IP masquerading out to the world via libvirt's default nat with<br>
> libvirt's default iptables rules. As I said, all that's working fine.<br>
> So regardless of whether opennebula truly SUPPORTS it, it does work. :-)<br>
><br>
> All I need to go is get one port to forward to one vm and my setup will<br>
> work wonderfully.<br>
><br>
> On 05/11/2012 11:01 AM, Jaime Melis wrote:<br>
> > Dear Sean,<br>
> ><br>
> > OpenNebula does not support Libvirt's NAT-network(<a href="http://192.168.122.0/24" target="_blank">192.168.122.0/24</a><br>
</div>> <<a href="http://192.168.122.0/24" target="_blank">http://192.168.122.0/24</a>>).<br>
<div class="im">> > You have to create a network environment as described here:<br>
> > <a href="http://opennebula.org/documentation:rel3.4:plan#networking" target="_blank">http://opennebula.org/documentation:rel3.4:plan#networking</a><br>
> ><br>
> > How are your hosts connected to each other?<br>
> ><br>
> > Regards,<br>
> > Jaime<br>
> ><br>
> > On Fri, May 11, 2012 at 4:40 PM, Sean Abbott <<a href="mailto:seabbott@akamai.com">seabbott@akamai.com</a><br>
</div><div><div class="h5">> <mailto:<a href="mailto:seabbott@akamai.com">seabbott@akamai.com</a>>> wrote:<br>
> >> Hello,<br>
> >><br>
> >> I have a setup intended for selenium grid. I'll have a bunch of<br>
> >> machines using the default NAT for internet access and a management<br>
> >> network for communication between each other. The template for these<br>
> >> VMs is:<br>
> >><br>
> >> NIC=[<br>
> >> MODEL=virtio,<br>
> >> NETWORK="natnet0",<br>
> >> NETWORK_UNAME=oneadmin ]<br>
> >> NIC=[<br>
> >> MODEL=virtio,<br>
> >> NETWORK="fullbackend",<br>
> >> NETWORK_UNAME=oneadmin ]<br>
> >><br>
> >><br>
> >> That's all working. I need one VM to be able to communicate with the<br>
> >> outside world on a single port (4444). I've set up template as follows:<br>
> >><br>
> >> NIC=[<br>
> >> MODEL=virtio,<br>
> >> NETWORK="natnet0",<br>
> >> NETWORK_UNAME=oneadmin,<br>
> >> IP=192.168.122.100,<br>
> >> WHITE_PORTS_TCP=4444 ]<br>
> >> NIC=[<br>
> >> MODEL=virtio,<br>
> >> NETWORK="fullbackend",<br>
> >> NETWORK_UNAME=oneadmin ]<br>
> >><br>
> >> The resulting VM can still use the internet and it can still communicate<br>
> >> with the other VMs. I can communicate with it on port 4444 from the<br>
> >> host that it's on, but not from outside.<br>
> >><br>
> >> Is this what WHITE_PORTS_TCP is intended for? Am I doing something wrong?<br>
> >><br>
> >> Thanks!<br>
> >><br>
> >> sean<br>
> >> _______________________________________________<br>
> >> Users mailing list<br>
</div></div>> >> <a href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a> <mailto:<a href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a>><br>
<div class="im">> >> <a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
> ><br>
> ><br>
><br>
><br>
><br>
><br>
> --<br>
> Jaime Melis<br>
> Project Engineer<br>
> OpenNebula - The Open Source Toolkit for Cloud Computing<br>
</div>> <a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> <<a href="http://www.OpenNebula.org" target="_blank">http://www.OpenNebula.org</a>> | <a href="mailto:jmelis@opennebula.org">jmelis@opennebula.org</a><br>
> <mailto:<a href="mailto:jmelis@opennebula.org">jmelis@opennebula.org</a>><br>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Jaime Melis<br>Project Engineer<br>OpenNebula - The Open Source Toolkit for Cloud Computing<br><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:jmelis@opennebula.org" target="_blank">jmelis@opennebula.org</a><br>
</div>