using advanced routing to control traffic across your interfaces

by nc on November 21, 2005 · 2 comments

[This page originally lived at I am working on migrating all content over to WordPress, which is why this post exists. This document is mostly up-to-date; please leave a comment with any changes!]

One of my tasks at work has been to set up Nagios to monitor all of our critical services. In the process of setting this up, I’ve ran into a very interesting issue related to the way Linux does ARP with a “strange” routing table. This article details what the problem I ran into was, and what I did to resolve it with Advanced Routing.

Last modified: 11/21/2005 Nate Carlson

As an aside, this article could also be very useful for people who have two separate ISP’s, with a separate IP range from each ISP. The gist of what I end up doing is setting up source routes to guarantee that traffic will go back out the proper interface, which can be necessary to get the expected behavior out of your network.

First of all, I need to explain a bit about our network layout. For each of our public-facing boxes, we have two network interfaces – “front” and “back”. Let’s call the front interface eth0, and the back interface eth1. Front is used to serve actual data to the world, and back is supposed to be used for management purposes. Assume that is our front network, and is our back network. Our routing table looks something like this:

Destination	Gateway		Genmask		Flags Metric Ref    Use Iface   UG    0      0        0 eth1     U     0      0        0 eth1     U     0      0        0 eth0         UG    0      0        0 eth0 and are the uplink “internal” routers; is the load balancer that these boxes are behind. is a management network at our main office, which is where the Nagios server is located that monitors this box. Let’s say that the local IP’s on this box are and

On the Nagios server, I am only monitoring (front) network at this point. I should probably be monitoring both, but hadn’t set that up yet; this is rather fortunate, as if I was monitoring both interfaces, I wouldn’t see the strange behavior. What is this behavior, you ask? In times of low load (IE, no traffic going to/from the box besides the Nagios monitoring), the box would occasionally become unreachable. I could verify this by trying to ping it’s address on the network – I wasn’t able to reach it. However, the second I ping the interface, the interface becomes reachable again. I worked with the network guy on and off for a few weeks to try to figure out what was causing this behavior, and finally we figured out that it’s the way that the Linux kernel sends ARP requests. What happens is that the ARP entry for times out on the Linux box (because of the lack of traffic), and it tries to re-resolve it. However, since the address we’re trying to connect to from the Nagios is in the network, the Linux box sends an arp entry out the eth1 interface that looks like:

“Who has Tell”

The Cisco router we’re using denies this request, as the IP asking for the ARP entry is not part of the network it’s asking for. In the ARP debug logs on the Cisco, we got an error like:

“IP ARP req filtered src , dst wrong cable, interface

So, what can we do to get around this problem? I can see three solutions, any of which would work:
1) Add a static ARP entry for the router on the Linux box
2) Set up advanced routing on the Linux box, so traffic will go back out the same interface it came in
3) Figure out a way to get the router to answer the filtered ARP requests, and/or mangle the ARP request with iptables to “appear” to come from the right IP.

I really didn’t like either #1 or #3, so I went with #2. Here’s what the rules I added end up looking like:

## Table 100 – Traffic in/out of eth0, front
$ ip route add table 100 via dev eth0
$ ip route add table 100 default via dev eth0

## Table 101 – Traffic in/out of eth1, back
$ ip route add table 101 via dev eth1

## Main table; default routes. Default to using the “back” interface for comms to HQ.
$ ip route add table main via dev eth1

$ ip route add table main via dev eth1

## Make our traffic follow these rules
$ ip rule add from lookup 100
$ ip rule add from lookup 101

With these rules in place, everything’s working great – traffic’s flowing in and out of the interfaces, as expected. Now, when the box tries to reply to traffic that hit it at, it will go back out the eth0 interface, and ARP for, which works just fine. All by the wonders of source routing.

If you have any comments on this document, please feel free to drop me an e-mail at:

{ 2 comments… read them below or add one }

Bernd December 22, 2009 at 8:47 am

I had exactly the same problem, but solved it on the cisco router (your method #3). On the interface in question, I added the config line: “ip mobile arp”, and the arp now works across interfaces.



nc December 22, 2009 at 9:50 am

Excellent – thanks for the note, good to know how to do it that way! ;)


Leave a Comment

Previous post:

Next post: