I have recently been toying with multiple ethernet interfaces on completely different vlans for security purposes, and have been struggling to figure this out. I found millions of articles on the web of how to play with iptables and routes, but virtually none that address this simple problem. Consider this example:
Eth0 10.10.1.10/24 gw 10.10.1.10
Eth1 10.1.51.200/24 gw 10.1.51.1
The default gateways for both lans are the same router.
$ netstat -anr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.10.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.1.51.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
0.0.0.0 10.10.1.1 0.0.0.0 UG 0 0 0 eth0
By default, Linux allows you to specify ONE default gateway on your primary nic, in this case eth0. This is a huge problem in this scenario, because you cannot be assured that traffic intended for eth1 will come to eth1. Everyone gets confused and trys to send the data to eth0 which they either can’t see or they are forced to use the router to translate the vlans and subnets. Epic Fail. For all intents, this server is using the one default route on eth0.
How to fix it.
Borrowing the method from this website, we create a policy route. I am using Ubuntu 10.04 Server 64bit in this example which has kernel support for “policy routing” If you are on a modern Debian or Ubuntu system (k 2.6.26 or newer), this should be there already. Don’t know about the other distros, but they should be similar.
sudo vi /etc/iproute2/rt_tables
add “1 internal” to the bottom. This creates a policy table for the internal network on eth0.
This table starts out empty so we need to give it some information about the network eth0 is on. My eth0 is 10.10.1.10 and my gateway is 10.10.1.1.
ip route add 10.10.1.0/24 dev eth0 src 10.10.1.10 table internal
ip route add default via 10.10.1.1 dev eth0 table internal
Great. We have created the route table for eth0. Now we need to tell the kernel how to use this policy with an ip rule
Ubuntu seems to forget them on restart, and I want these to come back every time the computer is restarted so I put them in /etc/rc.local
sudo vi /etc/rc.local
Add the following lines before exit:
ip rule add from 10.10.1.10/32 table internal
ip rule add to 10.10.1.10/32 table internal
Nothing has happened yet because we told these rules to be created and applied at bootup, so restart. You should now be able to see the rules with ip rule show
Congratulations! You’re system should now properly route traffic. Repeat the process as necessary.