ramblings of the village idiot

For those of you who have contacted me regarding my last post on cheap colocation, I have space available again, since I’m upgrading to a full rack. The gist: 1U of colo space in Minneapolis for $50/mo, in a good datacenter. For more info, see my colocation page.

When I purchased my laptop a little bit over a year ago, I added Dell’s CompleteCare service to it. This was advertised (at the time) as “3-year Next Business Day Onsite warranty with CompleteCare”. To me, this implied that the CompleteCare service would be rendered on-site on the next-day. However, a couple weeks ago, I pulled a stupid, and spilled hot chocolate all over my laptop. After talking to many manager and such, Dell demanded that my laptop be sent in to the depot for service. I was rather irritated (to say the least), but finally found a CompleteCare terms and conditions sheet online that did specify that CompleteCare service is generally rendered at a depot.

Dell did do a pretty quick turn-around — they had someone from DHL out with a box to pick it up on a Thursday, and I received the laptop the next Wednesday. However, the hard drive was still not fixed — the laptop included a note saying the depot didn’t have the hard drive in stock, and they would ship me one out. I called support, and ensured that they were shipping it out – I got it the next day. It also turned out that they didn’t include the PCMCIA card blank (no big deal), and the Bluetooth card.. they also shipped those out without a problem. I was also a little annoyed that they shipped the laptop back second-day delivery (it shipped on Monday) when the manager had promised me that they would expedite it as much as possible, but I guess that’s ok.

I waited a week or two to post this, because I was *really* annoyed that they wouldn’t do the service onsite, and wanted some time to calm down before posting a nasty rant. Looking back, they still did a pretty decent job – their warranty service still beats what any other company I’ve seen would have done (“You spilled hot chocolate on your laptop? Buy a new one.”) I certainly will not buy another laptop from Dell Home, though — any future purchases will be from Dell Small Business. That way, I’ll at least talk to someone who has a reasonable grasp of the English language.

Our printer (Epson Stylus Photo R200) died on us Monday night. I talked things over with Tiff, and we decided to go with an all-in-one printer, so she could do copying and such, too (she often could use that for work). One of my requirements was also to get a network-connected printer, so we don’t have to have a computer on to be able to print from one of the other PC’s in the house. We ended up picking up an HP Photosmart 3210, which seemed to be a decent mix of price and features. Price was $260 (at Best Buy), minus a $60 rebate, and a free $20 Best Buy gift card with the purchase. I also had $130 in Best Buy cards already, so it won’t be a very high out-of-pocket expense.

In any case, got the printer home, and plugged it into the network. It went through a self-initialization cycle, which took about 5 minutes. During this, I started the driver install under both Linux and Windows. For Windows, I just popped the CD in, and let it do it’s thing. For Linux, I used the HPLIP driver. Surprisingly (not my experience working with Epson printers under Linux), I had the HP printer fully working about 2 minutes after an “apt-get install hplip hpijs-ppds”. All I had to do was run “hp-makeuri “, plug the URI into cups, and it worked! It also automatically installed a scanner driver, so I can fire up Kooka and scan over the network.. surprisingly simple.

On the Windows box, on the other hand, the install took about 20 minutes total – 10 minutes of installing, 2-3 minutes for a reboot, and about 5-10 more minutes of installing after the reboot. The software works fine, but still, that took forever!

I’m glad to see that HP’s doing some serious work on getting better support for their printers in Linux. I think I’ll be pretty happy with this printer – the output is pretty nice, and it is *fast*.

Well, or not. With the storms tonight, we had a ~20 second brownout.. and, of course, I don’t have most of my servers on UPS’s yet. I’ve got two APC Back-UPS NS 1250’s, and a APC Back-UPS Pro 1100.. but I’ve only got two of the UPS’s hooked up, and like three boxes on them. I figured I’d wait until I got all the boxes racked up, and then do it.. stupid decision, eh?

Ah well, at least now I have a reason to get the boxes on the UPS’s.. at least I didn’t lose any drives like last time.

Today, I ran into this link again:

http://www.kondra.com/circuit/circuit.html

It discusses how to set up monitoring of your circuit panels to get detailed information on how much each circuit in your house is using. I’ve been looking to implement something similar at home, and I fired the author (Dave) an e-mail asking where he purchased his gear. He was very helpful (we’ve actually been having quite a long mail thread – seems like quite a nice guy!), and suggested that I e-mail “someone” at Trend Point, which is the company he developed the monitoring software for.

I fired “someone” an e-mail, saying I was interested in getting pricing on their new logic board (which looks very cool – supports an ethernet transport, with XML for the data). I got a reply back pretty quicky, just stating:

“I’m sorry but, we do not provide the software or hardware for home sales.”

I replied back, saying:

“Well, does it count that I run a small business out of my home? :) (Or is it a certification thing?)”

His reply was that their products run many thousands of dollars, so he can’t help me.

I *hate* it when sales people give you this crap. How does he know that I don’t want to spend a couple grand to monitor power at my house? If he took a second to peruse my web side, he’d quickly see that I’m a real geek, and willing to spend cash on good solutions. I sent him a short rant back saying that, and his reply again was just “Sorry, I can’t help you.”

All I really have to say is after that experience with a sales guy, this is one company I won’t be going to when I look at implementing circuit monitoring at work, and won’t suggest it to any of my clients. Their loss, I guess.

Update:
Dave called “someone”, and requested that “someone” shoot me another message. “someone” sent a good message, saying that they are currently over allocated for production, and that they are trying not to take any small orders at the moment. So, it wasn’t actually anything against a “home user”; it was just that he doesn’t want any small orders at the moment. I just wish that he had said that in the first place.. but I did also probably overreact a bit.

For those of you using WordPress and Google Analytics, you’ve got to check out this plugin:

http://www.oratransplant.nl/uga

It’ll track just about everything you can do on your blog with Google Analytics. Much larger feature set than any of ther others I’ve seen. Also much simpler than hacking the themes to add the code, which I had done previously. :)

OK, it’s not the usual geeky fare that gets posted here.. but Tiff and I got a new puppy. His name is Chimo, and he’s a 4 month old Malemute/Lab mix. We’ve decided to make him a geeky dog, and set up a blog for him.. check it out!.

As those of you who use Xen on the i386 arch know, the libc6 stuff can be rather annoying. The Debian libc6 developers have finally released a test glibc that includes xen compatibility — no more moving /lib/tls out of the way and losing performance!

You can grab the packages from:
http://people.debian.org/~aurel32/xen/

Hopefully these will be mainline soon.

Well, I used to swear by XMMS for a music player.. then I decided I wanted some nifty features like better KDE integration and last.fm updates (that’s where the last listened links came from on the right), so I decided to try out Amarok again. It’s great! It has all sorts of nifty features, like automatically trying to figure out what music (within your collection) you’d like based on what you’re listening to, automatic lyric and cover downloading, etc. In any case, if you’re still an old-fashioned XMMS user (like I was up until a few days ago), and a KDE user, give it a shot!

After a few very tiresome weeks working night shifts for the Olympics (started at midnight on Friday the 10th, and had the midnight to 9am (well, 8:30 most days) shift every day until the last shift started at midnight on the 25th), I’m finally back to my normal hours, and only working Monday through Friday again. What a change! In any case, I’m trying to get caught up on many missed e-mails, household stuff (we closed on our new townhouse on the 8th, and moved in the same day – the Olympics really threw off our ability to get the house into order), and sleep.. if I’ve missed your e-mail, feel free to fire me another one, and I’ll try to get back to you a bit quicker.

Oh, and if you’re bored, check out all the people who were involved in bringing you the NBCOlympics site.

Well, this weekend, we’re going to be trying a new experiment.. we’ll be putting a live stream of the gold metal hockey game between Sweden and Russia or Finland up on the NBC Olympics web site (streaming video is fun!). It’ll be interesting to see how it goes, and how well Akamai can handle the load. Feel free to check it out – the stream should start at 7am Central Time. *crosses fingers*. There hasn’t been much news coverage of this – the only article I’ve seen is at Broadcasting Cable. Unfortunately, because of some restrictions that are imposed upon broadcasters, we have to have DRM enabled on the stream, and lock it down to US IP addresses.. this pretty much makes it impossible to view this with anything but Microsoft Windows Media Player. Have to get Crossover Office running on my laptop again so I can watch it.. (scratch that; cxoffice’s supported version of Media Player won’t play it, either.)

Update:This is active as of *right now*; http://www.nbcolympics.com/streaming.

On another note, I’m almost done with these really annoying night shifts for the Olympics – after tonight, one more night. Woohoo! It’ll be great to actually get to sleep in my own bed in the new townhouse with my wife again.. and not be dead tired all the time! After this, I can finally say for sure that I have no desire to work nights.. just not for me.

Seems like pings aren’t working properly with WordPress 2.0… odd. I’ll have to keep playing with it. Appreciate any comments if other people are having this issue.

I’m trying svn now, to see if it fixes anything..

I’ve been using eFax Free for awhile, but have been having problems with a pile of spam faxes filling up the “free” 20 faxes a month, and having them cancel accounts on me. I finally found a decent replacement – k7. They offer a free DID in the Seattle area, with fax-/voicemail-to-email service. Seems to work pretty well.. nice for those occasional fax users like me! In any case, I recommend it – not sure I’d send anything private through it (or any other external fax-to-email gateway), though.. I’ll have to post further once I’ve actually gotten some use out of it.

The company that I work for is responsible for the production (both the hosting environment and the actual design) of the NBC Olympics web site, and we’re finally starting to see some serious media attention focusing on the site – it’s nice. A local TV station interviewed a few people here, and put the report online. Doing a Google News search for the site also returns quite a few hits, although most of them don’t mention us.

With the current version of the Linux kernel (2.6.15), some patches, and ATI’s fglrx driver version 8.20.8, software suspend/resume *finally* works, with full accelerated graphics support. Woohoo! Uptime on my laptop is now 6 days, and that involves many trips between home and work, and many suspend/resumes.

If you’d like a copy of the scripts I’m using to get suspend working properly, let me know. I can either suspend with a FN-ESC, or by running a script (usually the method I use if I want to shut down the network card and restart it when I resume.)

Tim (one of my co-workers) and I have been messing around with mt-daapd for the last couple days, and I gotta say, the software *rocks*!

For those of you not familiar with DAAP, it’s the protocol that Apple uses for iTunes’s music operations over the network. mt-daapd is an open-source product that runs on Linux, and lets you set up an iTunes server. The current stable version is pretty nifty, but if you’re willing to go bleeding edge, you get lots of sweet features, like the ability to set up smart playlists via the web interface, and the ability to transcode pretty much any audio format on the fly, so iTunes can play ogg vorbis files over the network. If you’d like to try out mt-daapd’s current CVS version, I’ve got Debian packages built for it – give me a holler.

It’s also fairly trivial to set up a SSH tunnel to a remote iTunes server, and share that iTunes share on your local network.. assuming that you have a Linux box with mDNSResponder set up, at least. It’s pretty simple – just set up your SSH connection:

ssh username@remote-linux-box -N -f -L *:3690:ip-of-itunes-or-mt-daapd-server:3689

Then, in your /etc/mdns/mDNSResponder.conf file:

“Nate’s Music at Home” _daap._tcp. local. 3690

With this, I can easily stream music from my mt-daapd server at home to any box running iTunes on my subnet at work.

Like I said, teh cool!

Just got back from the new Narnia: The Lion, The Witch, and the Wardrobe movie. Man, that movie rules! I just read the book (well, all of them) a couple weeks ago (gotta re-read the books before seeing the movies, you know), and the movie is darn close to the book. I can only think of a few things that were left out, I didn’t see any major (but plenty of minor) differences in the primary parts of the movie. In fact, this is one of the few movies I’ve seen where they add a fair bit of content that isn’t in the book – it seemed to fill out the story a bit, since you can’t see what people are thinking on-screen like you can when you read the books.

In any case, I highly recommend it – this is one of the few movies I’ve seen where I don’t come away feeling that they’ve missed out on majorly important parts of the book. It reminds me a lot of the Lord of the Rings movies.

Oh, yeah – Lucy doesn’t annoy the crap out of you in this movie, like the older ones. They got that part right, for sure!

Virtual Private Networks, or VPNs, are a way of securely accessing resources on your network from untrusted points on the internet. This page describes some of the various types of open-source VPN solutions that are available on Linux systems, with benefits and drawbacks for each solution. I’m not making a VPN comparison, per se, but I hope I provide enough information that you can draw your own conclusions, and make a decision on what type of VPN best fits your need. Feel free to mail me (ipsec@natecarlson.com) with any questions.

Contents:
IPSec VPNs (Openswan, KAME)
SSL-Based VPNs (OpenVPN)
PPTP-Based VPNs (PoPToP)
Commercial VPN software

IPSec VPNs (Openswan, KAME)

IPSec is one of the older VPN standards, and is still very secure and useful when properly configured. There are two major separate implementations of IPSec under Linux. The first is the project that was originally called FreeS/WAN, but has now forked into Openswan and Strongswan. This implementation provides its own IPSec kernel stack, and it can also use the code included in recent kernels. The second is a port of KAME from BSD. KAME can only use the kernel stack. The main IPSec specification itself does not provide a virtual IP for the remote host on the local network, but there are various extensions that offer this. You can also run L2TP over IPSec, which is well supported by Microsoft’s recent operating systems.

Pros: IPSec is a very established protocol, and is well supported by pretty much anything that supports VPN connections (routers, smartphones, operating systems, you name it!) The Openswan implementation works with the proprietary XAUTH extension, and can work as a client to Cisco, Nortel, and many other VPN concentrators. IPSec makes it reasonably easy to secure what can and cannot go over a tunnel, at the kernel level, without having to set up extra firewall rules. Very flexible for subnet<->subnet configurations, host<->subnet configurations, and so on.

Cons: IPSec can be difficult to get set up and working. It also does not work behind some types of NAT gateways, although this has improved with NAT-Traversal support.

Links:

Openswan
IPSec-tools, KAME port for Linux
My article on configuring Openswan with X.509 and Windows XP’s client
My article on configuring Openswan with L2TP and Windows XP’s client
Consulting information, if you need help

SSL-Based VPNs (OpenVPN)

Recently, SSL-based VPN have been gaining popularity. The big benefit to SSL VPN’s is that you only require a single TCP or UDP port to tunnel your traffic on, so you can easily traverse most firewalls. There are many implementations of SSL VPN’s; many of them are commercial, and support both a web-based interface (which only allows you to browse web pages on the remote network, but works on any browser on any platform.. it is essentially a browser-based proxy server) and a full tunneled implementation. As far as open source implementations go, the most mature by far is OpenVPN. OpenVPN is fairly mature, very feature-rich, and has been ported to most major operating systems. As of yet, there is not a “clientless” (ie, web-browser-based) version available that I am aware of.

Pros: Trivial firewall configuration; just needs a single TCP or UDP port. Uses SSL, which is a very mature protocol. Available for most operating systems, including Windows. Flexible configuration options.

Cons: Requires more firewall configuration that IPSec to control access to internal resources. OpenVPN is not supported in most commercial VPN concentrators; however, they usually provide their own implementation.

Links:
OpenVPN’s home page; very informative.

PPTP-Based VPNs (PoPToP)

PPTP is the protocol that Microsoft originally supported somewhere around Windows 95. It’s been used for a long time, but there are many questions about the security of it. Basically it tunnels a PPP connection over the GRE protocol. PoPToP is the main PPTP server for Linux. If you think you need PPTP, I would highly advise you to look at L2TP over IPSec instead — L2TP over IPSec is more secure, offers all the same features, plus a few extras.

Pros: Easy configuration under Windows, supported by many commercial routers/firewalls.

Cons: Questionable security, firewall/NAT problems similar to IPSec. Requires kernel patches to offer encryption.

Links:
PoPToP home page.
Analysis of the MSCHAP-v2 protocol

[ad name=”Chitika 728×90 Leaderboard”]

Commercial VPN software

There are various commercial VPN clients available for Linux, but as far as I am aware, there are not any commercial servers. If you are aware of any commercial servers for Linux, please let me know. Below is a list of some of the better-known commercial clients.

Links:
vpnc — open-source client to connect to Cisco VPN concentrators with IPsec
OpenConnect — open-source client to connect to Cisco “AnyConnect” SSL VPN’s
Cisco Client — link is to reference manual; if you have a CCO login you can download the client.
Nortel VPN Client
Apani VPN client for Nortel

Just want to issue a blanket apology to people who have e-mailed me for help with IPSec that I have not had time to respond to – I’ve been very busy lately, and unfortunately this has fallen on my priority list somewhat.

I’ve been considering posting a forum on my site, to help me better track requests for help – would this be useful to anyone? I’ve historically preferred the e-mail method, but many people don’t want to mail the Openswan list for some reason (or don’t get help there); I’m thinking that doing forum posts instead may help me get responses back to people who I’d otherwise miss (since the forum post will sit there forever, while my mailbox gets filled with other clutter.) Feel free to leave a comment and let me know if this would be helpful for you!

[This page originally lived at http://www.natecarlson.com/linux/advanced-routing-in-out.php. I am working on migrating all content over to WordPress, which is why this post exists. This document is mostly up-to-date; please leave a comment with any changes!]

One of my tasks at work has been to set up Nagios to monitor all of our critical services. In the process of setting this up, I’ve ran into a very interesting issue related to the way Linux does ARP with a “strange” routing table. This article details what the problem I ran into was, and what I did to resolve it with Advanced Routing.

Last modified: 11/21/2005 Nate Carlson

As an aside, this article could also be very useful for people who have two separate ISP’s, with a separate IP range from each ISP. The gist of what I end up doing is setting up source routes to guarantee that traffic will go back out the proper interface, which can be necessary to get the expected behavior out of your network.

First of all, I need to explain a bit about our network layout. For each of our public-facing boxes, we have two network interfaces – “front” and “back”. Let’s call the front interface eth0, and the back interface eth1. Front is used to serve actual data to the world, and back is supposed to be used for management purposes. Assume that 10.100.0.0/16 is our front network, and 10.101.0.0/16 is our back network. Our routing table looks something like this:

Destination	Gateway		Genmask		Flags Metric Ref    Use Iface
10.0.0.0	10.101.0.254	255.255.255.0   UG    0      0        0 eth1
10.101.0.0	0.0.0.0		255.255.0.0     U     0      0        0 eth1
10.100.0.0	0.0.0.0		255.255.0.0     U     0      0        0 eth0
0.0.0.0		10.100.0.1	0.0.0.0         UG    0      0        0 eth0

10.100.0.254 and 10.101.0.254 are the uplink “internal” routers; 10.100.0.1 is the load balancer that these boxes are behind. 10.0.0.0/24 is a management network at our main office, which is where the Nagios server is located that monitors this box. Let’s say that the local IP’s on this box are 10.100.0.100 and 10.101.0.100.

On the Nagios server, I am only monitoring 10.100.0.100 (front) network at this point. I should probably be monitoring both, but hadn’t set that up yet; this is rather fortunate, as if I was monitoring both interfaces, I wouldn’t see the strange behavior. What is this behavior, you ask? In times of low load (IE, no traffic going to/from the box besides the Nagios monitoring), the box would occasionally become unreachable. I could verify this by trying to ping it’s address on the 10.100.0.0 network – I wasn’t able to reach it. However, the second I ping the 10.101.0.0 interface, the 10.100.0.0 interface becomes reachable again. I worked with the network guy on and off for a few weeks to try to figure out what was causing this behavior, and finally we figured out that it’s the way that the Linux kernel sends ARP requests. What happens is that the ARP entry for 10.101.0.254 times out on the Linux box (because of the lack of traffic), and it tries to re-resolve it. However, since the address we’re trying to connect to from the Nagios is in the 10.100.0.0 network, the Linux box sends an arp entry out the eth1 interface that looks like:

“Who has 10.101.0.254? Tell 10.100.0.100”

The Cisco router we’re using denies this request, as the IP asking for the ARP entry is not part of the network it’s asking for. In the ARP debug logs on the Cisco, we got an error like:

“IP ARP req filtered src 10.100.0.100 , dst 10.101.0.254 wrong cable, interface “

So, what can we do to get around this problem? I can see three solutions, any of which would work:
1) Add a static ARP entry for the router on the Linux box
2) Set up advanced routing on the Linux box, so traffic will go back out the same interface it came in
3) Figure out a way to get the router to answer the filtered ARP requests, and/or mangle the ARP request with iptables to “appear” to come from the right IP.

I really didn’t like either #1 or #3, so I went with #2. Here’s what the rules I added end up looking like:

## Table 100 – Traffic in/out of eth0, front
$ ip route add table 100 10.0.0.0/24 via 10.100.0.254 dev eth0
$ ip route add table 100 default via 10.100.0.1 dev eth0

## Table 101 – Traffic in/out of eth1, back
$ ip route add table 101 10.0.0.0/24 via 10.101.0.254 dev eth1

## Main table; default routes. Default to using the “back” interface for comms to HQ.
$ ip route add table main 10.0.0.0/24 via 10.101.0.254 dev eth1

$ ip route add table main 172.16.4.0/24 via 10.19.0.254 dev eth1

## Make our traffic follow these rules
$ ip rule add from 10.100.0.0/16 lookup 100
$ ip rule add from 10.101.0.0/16 lookup 101

With these rules in place, everything’s working great – traffic’s flowing in and out of the interfaces, as expected. Now, when the box tries to reply to traffic that hit it at 10.100.0.100, it will go back out the eth0 interface, and ARP for 10.100.0.254, which works just fine. All by the wonders of source routing.

If you have any comments on this document, please feel free to drop me an e-mail at: natecars@natecarlson.com