Random Developments

So, take a firewall module for a large switch made by a five-letter manufacturer that shall not be named (on the other hand, why not? naming it will help people Googling for it. It’s Cisco). Start to plan and test a migration path for all the equipment you already have, so that you end up with everything behind the firewall. Obviously, you don’t want to move everything on the same day.

Next, add to the mix a DHCP server that provides IP addresses to most things on the net, according to a set of rules (each MAC address is allocated the same IP address every time). This server, like everything else, is on the “outside” of the firewall.

Then, just to make sure that you did things right and that you can get traffic to flow across the firewall, you move one PC to the “inside” and try to make it talk to the net. It won’t, because it can’t get an IP address from the DHCP server. You enable “dhcprelay” just as the manual tells you to, correcting for some syntax errors in the examples in the manual, and it still won’t work. So you give the PC a fixed address and try again. Still no go.

After some tweaking with routes and OSPF configuration, voilà, it works! Pings go through, DNS works, even web browsing. So, ok, let’s try DHCP again. But, still nothing. Read manuals, tweak things, read some more. Still nothing. Look at the server, it sees the requests. Turn debug on in the firewall, it claims to be forwarding the replies to the PC. But the PC never sees them.

Ok, let’s bring the big guns out. Mirror a switch port, start ethereal, and look at the network traffic. Ok, there goes a DHCPDISCOVER, but where is the DHCPOFFER? Hmm, and what is this ARP request doing here? The firewall is sending an ARP request to the “inside” network asking who has A.B.C.D, where A.B.C.D is the address of the default gateway for the firewall. Can you see what is wrong with this? Two things: one, the default gateway is in the “outside” network; two, why does it need to ARP the gateway to forward a DHCPOFFER packet? It’s not even addressed to an IP address!

So, quick experiment: take the default route out, and try again. And it works!

Where are we, then? DHCP relaying works, as long as there is no default route set. If a default route is set, the firewall seems to try to do the right thing, but it does it in the wrong way. Seriously wrong.

Time to get support, then. We contact our provider, tell them about the issue and, after a few e-mails back and forth, they seem to understand the problem. The next day they report back saying that there is a confirmed bug that seems to match our situation. The description of the bug is “DHCP relay does not work in customer setting, not reproducible in the lab”. No mention of interaction with default routes. And the software versions in which the bug is confirmed do not match ours. But they do recommend an upgrade to a version that is 0.0.0.1 higher than ours. Ok, we’ll give it a shot. Go to cisco.com, locate the software for the firewall module and, surprise surprise, the latest version available is the one we’re running; the one they recommended is not available.

Another contact with support later, we find out that the version is indeed not available, and they don’t know when it will be. Well, isn’t that great?

In short: dhcprelay does not seem to work in Cisco firewall modules when a default route is set. A bug fix may or may not exist, but we will only know for sure when Cisco decides to release the new version of the software.

If anyone knows of anything related to this issue, please let me know. Meanwhile, I’ll be setting up a new DHCP server inside the firewall.

Subscribe to the comments through RSS Feed