ramblings of the village idiot » Geek Stuff

Configuring IPMP on NexentaStor 3

nc — Fri, 10 Sep 2010 18:38:23 +0000

At my day job, we have recently started rolling out NexentaStor 3 for our VM image storage as a trial. If all goes well, our long-term plans are to eventually migrate all storage from Netapp to NexentaStor. As we started rolling out our NexentaStor trial, one missing feature we quickly ran across is the lack of IPMP (IP Multipathing) support. The network configuration interface that they provide can currently configure aggregated interfaces with the LACP protocol, but they do not have any mechanism to configure IPMP to aggregate interfaces from multiple switches. We were able to work out an approach to configure IPMP manually, and received Nexenta’s blessing to use it in our environment. (Important note: if you are going to try this on a licensed copy of NexentaStor, please check with your support team to ensure that they are ok with you making these changes.)

Updated 2010-Sep-16 — Added information on how to add static routes to configure ipmp ping targets

Server hardware configuration

First of all, I should detail what we are trying to configure. Our production machines are quite similar to the SuperMicro build I documented earlier, with a few varying specs. Here’s what is in it:

2x Intel E5620 CPUs
48G memory (6x 8G modules)
10x 1TB Seagate Constellation ES SAS drives
6x Intel 32gb X25-E drives
2x Crucial 128gb RealSSD drives
2x Intel Gigabit ET quad-port NICs

The machine has an 8TB license, with 2 of the disks configured as hot spares. The Intel SSDs are configured as 3 log-mirrors, and the RealSSDs are configured as cache devices.

Caveats

The only major caveat that I’ve hit with this configuration is that the ipmp interfaces will not be viewable via the Nexenta utilities. You can still see all the underlying interfaces; just not the ipmp ones. It’s mostly cosmetic, but is distracting and annoying.

Of course, YMMV – this worked for me, but no guarantees that it will work for you! :)

Network configuration

Desired configuration

Here is the network configuration we desire:

LACP Aggregate #1 – 4 gigabit links to Switch-1
LACP Aggregate #2 – 4 gigabit links to Switch-2
IPMP Interface #1 (balancing LACP1 and LACP2) – Native VLAN on NICs; access to our management/backend network
IPMP Interface #2 (balancing LACP1 and LACP2) – VLAN ID 100; VM storage network
IPMP Interface #3 (balancing LACP1 and LACP2) – VLAN ID 200; NFS storage network

General configuration steps

As far as I know, you cannot do VLAN tagging on top of an IPMP trunk on Solaris, which means that we need to create the VLAN interfaces on each of the aggregate interfaces, and then create three separate IPMP interfaces – one per vlan. Here are the basic configuration steps:

Via NMC: Create individual LACP aggregates (aggr1 and aggr2), with igbX interfaces as members.
Via NMC: Create VLAN interfaces ’100′ and ’200′ on top of both ‘aggr1′ and ‘aggr2′. This will create the interfaces ‘aggr100001′, ‘aggr100002′, ‘aggr200001′, and ‘aggr200002′.
Via NMC: Configure an IP address from within each VLAN on each of these six interfaces. This will allow IPMP to use ICMP probes in addition to link detection to try to find failed links.
Via the console: Configure the three IPMP interfaces, and add the six aggr interfaces to the proper IPMP groups.

NMC – Create LACP aggregates

This assumes that whatever interface you configured during installation is *not* one of the interfaces you desire to be part of the aggregate. If that is not true, you will need to be on the system console (via IPMI hopefully!), and destroy that interface first. Here is an example of how to create the aggregates (output from NMC; so this is how it ends up being configured):

nmc@nexenta:/$ setup network aggregation create
Links to aggregate  : igb2,igb3,igb4,igb5
LACP mode           : active
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr1           L3,L4    auto                 active        short       -----
nmc@nexenta:/$ setup network aggregation create
Links to aggregate  : igb6,igb7,igb8,igb9
LACP mode           : active
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr2           L3,L4    auto                 active        short       -----

NMC – Create vlan interfaces on each aggregate interface

nmc@nexenta:/$ setup network interface aggr1 vlan create
VLAN Id : 100
aggr100001: flags=201000842 mtu 9000 index 39
        inet 0.0.0.0 netmask 0
        ether 0:1b:21:6c:3c:de

nmc@nexenta:/$ setup network interface aggr2 vlan create
VLAN Id : 100
aggr100002: flags=201000842 mtu 9000 index 40
        inet 0.0.0.0 netmask 0
        ether 0:1b:21:6c:3d:de

nmc@nexenta:/$ setup network interface aggr1 vlan create
VLAN Id : 200
aggr200001: flags=201000842 mtu 9000 index 41
        inet 0.0.0.0 netmask 0
        ether 0:1b:21:6c:3e:de 

nmc@nexenta:/$ setup network interface aggr2 vlan create
VLAN Id : 200
aggr200002: flags=201000842 mtu 9000 index 42
        inet 0.0.0.0 netmask 0
        ether 0:1b:21:6c:3f:de

NMC – Assign IP addresses

This assumes the following IP ranges:
Native VLAN: 10.100.0.0/24
VLAN 100: 10.100.100.0/24
VLAN 200: 10.100.200.0/24

It also assumes that aggregate 1 should be assigned .2 within each /24, and aggregate 2 should be assigned .3. The ipmp shared interface will be assigned .1.

nmc@nexenta:/$ setup network interface vlan aggr1 static
aggr1 IP address: 10.100.0.2
aggr1 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr1 as 10.100.0.2/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr2 static
aggr2 IP address: 10.100.0.3
aggr2 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr2 as 10.100.0.3/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr100001 static
aggr100001 IP address: 10.100.100.2
aggr100001 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr100001 as 10.100.100.2/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr100002 static
aggr100002 IP address: 10.100.100.3
aggr100002 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr100002 as 10.100.100.3/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr200001 static
aggr200001 IP address: 10.100.200.2
aggr200001 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr200001 as 10.100.200.2/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr200002 static
aggr200002 IP address: 10.100.200.3
aggr200002 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr200002 as 10.100.200.3/255.255.255.0 ... OK.

Console – Set up IPMP

First, we need to get into expert mode.

nmc@nexenta:/$ options expert_mode=1                                                                          

nmc@nexenta:/$ !bash
You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). CAUTION: NexentaStor
appliance is not a general purpose operating system: managing the appliance via Unix shell is NOT
recommended. This management console (NMC) is the command-line interface (CLI) of the appliance,
specifically designed for all command-line interactions. Using Unix shell without authorization of your
support provider may not be supported and MAY VOID your license agreement. To display the agreement,
please use 'show appliance license agreement'.
Proceed anyway? (type No to return to the management console)  Yes

root@nexenta:/volumes#

Next step is to set up the hostname files for each of the IPMP interfaces. I will name the interfaces as follows:

ipmp0 – ipmp interface for aggr1 and aggr2
ipmp100000 – ipmp interface for aggr100001 and aggr100002
ipmp200000 – ipmp interface for aggr200001 and aggr200002

These files also set the IP address that we would like the system to apply to these.

root@nexenta:/etc# cat hostname.ipmp0
ipmp group ipmp0 10.100.0.1 netmask 255.255.255.0 up
root@nexenta:/etc# cat hostname.ipmp100000
ipmp group ipmp100000 10.100.100.1 netmask 255.255.255.0 up
root@nexenta:/etc# cat hostname.ipmp200000
ipmp group ipmp200000 10.100.200.1 netmask 255.255.255.0 up

Next, the groups need to be configured in the hostname. files. We need to add ‘group -failover up’ to each of these files, before the ‘up’ at the end. Here are the files after the changes are made:

root@nexenta:/etc# for i in /etc/hostname.aggr* ; do echo $i ; cat $i ; done
/etc/hostname.aggr1
10.100.0.2 netmask 255.255.255.0 mtu 9000  broadcast +  group ipmp0 -failover up
/etc/hostname.aggr2
10.100.0.3 netmask 255.255.255.0 mtu 9000  broadcast + group ipmp0 -failover up
/etc/hostname.aggr100001
10.100.100.2 netmask 255.255.255.0  broadcast +  group ipmp100000 -failover up
/etc/hostname.aggr100002
10.100.100.3 netmask 255.255.255.0  broadcast + group ipmp100000 -failover up
/etc/hostname.aggr200001
10.100.200.2 netmask 255.255.255.0  broadcast +  group ipmp200000 -failover up
/etc/hostname.aggr200002
10.100.200.3 netmask 255.255.255.0  broadcast +  group ipmp200000 -failover up

Now that all the interface configs are in place, we can apply it.. here’s the easiest way I figured out; if anyone knows a better way I’d love to hear it!

# svcadm disable svc:/network/physical:default
# for i in aggr1 aggr2 aggr100001 aggr100002 aggr200001 aggr200002 ipmp0 ipmp100000 ipmp200000 ; ifconfig $i unplumb ; done
# svcadm enable svc:/network/physical:default

At this point, all of your interfaces should be up, and all the IP addresses should be pingable. Make sure that you can ping the individual interface IPs, and the IPMP IPs. You should be able to use the ‘ipmpstat’ command to see information about your groups; IE:

root@nexenta:/volumes# ipmpstat -a
ADDRESS                   STATE  GROUP       INBOUND     OUTBOUND
nexenta-vl100        up     ipmp100000    aggr100001    aggr100002 aggr100001
nexenta-vl200        up     ipmp200000    aggr200001    aggr200002 aggr200001
nexenta                up     ipmp0       aggr1       aggr2 aggr1

Note that this configuration provides failover and outbound load balancing, but it does not provide inbound load balancing. If you would like inbound load balancing, you need to configure an IP alias on each of the ipmp interfaces, and then mix up which IP you use from the host that is connecting to your Nexenta machine (or use multipathing if it’s iSCSI!)

One last thing – once everything is configured, you will probably want to define your own ping targets. You can view the ones that ipmp picked automatically by running ‘ipmpstat -t’. On our configuration, on one VLAN, two Nexenta nodes picked each other.. so when we took machine two down (intentionally), machine one marked that interface down, and then when we booted machine two back up, it could not reach machine one’s interface, and marked its interface on that vlan down. Nice race condition. Oddly, mpathd (the daemon that does the checking) does not use a configuration file for ping targets, but instead relies on host routes. What we’ve done is added routes to the individual IP addresses that we would like to have it monitor by using the NMC command ‘setup network routes add’, and specifying the IP address to monitor as both the ‘Destination’ and the ‘Gateway’. We picked four to five IPs on each VLAN that were stable hosts (routers, Xen domain-0′s and the like), and added them on both hosts.. this will give more consistent results, as multiple core machines would have to go down before the interface would be disabled incorrectly.

I hope this helps! Please feel free to leave a comment if you run into any trouble getting it working.

I think I found a fellow geek..

nc — Sat, 28 Aug 2010 03:27:17 +0000

I was walking the dogs this morning, and ran across this car up the street.. Minnesota license plate ’1RQ’.. I don’t know anything that this could mean besides ‘IRQ’.. a rather geeky concept.. what d’ya think? Is there another meaning? Inquiring minds and all!

New puppy!

Android: AT&T Captivate’s wifi networking is broken

nc — Fri, 27 Aug 2010 17:54:23 +0000

I’ve finally made the switch from Windows Mobile to Android by way of an AT&T Captivate (which is one of the many Samsung Galaxy S phones.) So far, I’ve been fairly happy – app selection is good, it’s great to have the ability to tinker on it (no, have not rooted it or flashed a custom rom yet), and the screen is great. The only major downfalls I’ve run into so far are the lack of a hardware keyboard (I am so used to being able to type on my phone without looking — I know that there are Android options that have a full keyboard, but I wanted the combination of a full keyboard with a large high-res screen and a fast processor.. the only options available right now are on carriers that I don’t want to go with. However, the T-Mobile G2 looks like it would have fit the bill — too bad I already bought!) and the weak integrated mail client — fortunately K-9 Mail took care of that one.

I’ve only run into one thing which really causes problems that I haven’t been able to fix yet — Samsung’s Android build seems to use old-fashioned classful addressing instead of honoring modern CIDR, and then hides the fact that it is doing that from the user. This means that if you are on a Wifi network that assigns an IP in 10.0.0.0/8, it will assign your netmask as 255.0.0.0 at the OS level, and you will not be able to reach any other subnets in 10.0.0.0/8. This won’t be a problem for the majority of users out there, but for people who have advanced home networks (me!), or connect to a corporate/campus network with their phone, it is likely to completely break the ability to use the network.

Oddly, if you look at the networking on the phone using a tool such as Network Info II, it will always show your netmask as the netmask that the DHCP server sent you. This is what really confused me for awhile — Network Info showed the proper netmask, but I couldn’t reach anything on 10.0.0.0/8 outside of the /24 that my phone’s IP was in. Finally, I figured out how to get the Android SDK working, and accessed the phone using adb. Here’s what I found:

nc@nc-desktop:~$ adb shell
$ ifconfig eth0
eth0: ip 10.0.55.201 mask 255.0.0.0 flags [up broadcast running multicast]
$

*Ahem* – no, my netmask is *not* 255.0.0.0. ;) At this point I assumed that the apps that were displaying 255.255.255.0 at the UI level were just assuming that was my netmask, and figured that it was a problem with the way that my DHCP server was configured. I started to dig around the filesystem, and found the /system/etc/dhcpcd directory — the dhcpcd configuration. The file /system/etc/dhcpcd/dhcpcd-hooks/95-configured looked interesting – it appeared to dump out the ip configuration that it received from the dhcp server into the system properties:

case "${reason}" in
BOUND|INFORM|REBIND|REBOOT|RENEW|TIMEOUT|IPV4LL)
    setprop dhcp.${interface}.ipaddress "${new_ip_address}"
    setprop dhcp.${interface}.gateway   "${new_routers%% *}"
    setprop dhcp.${interface}.mask      "${new_subnet_mask}"
    setprop dhcp.${interface}.leasetime "${new_dhcp_lease_time}"
    setprop dhcp.${interface}.server    "${new_dhcp_server_identifier}"

    setprop dhcp.${interface}.result "ok"
    ;;

I figured that I should be able to use ‘getprop’ to see what was set there; I gave it a shot..

[dhcp.eth0.result]: [ok]
[init.svc.dhcpcd]: [running]
[dhcp.eth0.pid]: [26234]
[dhcp.eth0.reason]: [BOUND]
[dhcp.eth0.dns1]: [8.8.8.8]
[dhcp.eth0.dns2]: [8.8.4.4]
[dhcp.eth0.dns3]: []
[dhcp.eth0.dns4]: []
[dhcp.eth0.ipaddress]: [10.0.55.201]
[dhcp.eth0.gateway]: [10.0.55.254]
[dhcp.eth0.mask]: [255.255.255.0]
[dhcp.eth0.leasetime]: [7200]
[dhcp.eth0.server]: [10.0.55.254]

What do you know – it was receiving the info from the DHCP server and processing it correctly! So this means that whatever utility dhcpcd is using to set up the IP interface must not be working properly, and assuming classful networking.

There are a few posts around the ‘net on this; first, a forum post at AT&T, and then something I posted at xda-developers; however, nobody’s had any luck figuring out why this is yet, or how to fix it (short of rooting the phone.) One workaround is to configure a static IP, but that’s a lame solution.

If anyone knows how to get this working properly I’d love to hear about it! :)

Review: SuperMicro’s SC847 (SC847A) 4U chassis with 36 drive bays

nc — Fri, 07 May 2010 07:20:01 +0000

[Or "my quest for the ultimate home-brew storage array."] At my day job, we use a variety of storage solutions based on the type of data we’re hosting. Over the last year, we have started to deploy SuperMicro-based hardware with OpenSolaris and ZFS for storage of some classes of data. The systems we have built previously have not had any strict performance requirements, and were built with SuperMicro’s SC846E2 chassis, which supports 24 total SAS/SATA drives, with an integrated SAS expander in the backplane to support multipath to SAS drives. We’re building out a new system that we hope to be able to promote to tier-1 for some “less critical data”, so we wanted better drive density and more performance. We landed on the relatively new SuperMicro SC847 chassis, which supports 36 total 3.5″ drives (24 front and 12 rear) in a 4U enclosure. While researching this product, I didn’t find many reviews and detailed pictures of the chassis, so figured I’d take some pictures while building the system and post them for the benefit of anyone else interested in such a solution.

Updates:
[2010-05-19 Some observations on power consumption appended to the bottom of the post.]
[2010-05-20 Updated notes a bit to clarify that I am not doing multilane or SAS - thanks for reminding me to clarify that Mike.]
[2011-12-20 Replacing references of 'port multiplier' with 'SAS Expander' to reflect the actual technology in use.. thanks commenters Erik and Aaron for reminding me that port multiplier is not a generic term, and sorry it took me so long to fix the terminology!]

In the systems we’ve built so far, we’ve only deployed SATA drives since OpenSolaris can still get us decent performance with SSD for read and write cache. This means that in the 4U cases we’ve used with integrated SAS expanders, we have only used one of the two SFF-8087 connectors on the backplane; this works fine, but limits the total throughput of all drives in the system to 4 3gbit/s channels (on this chassis, 6 drives would be on each 3gbit channel.) On our most recent build, we built it with the intention of using it both for “nearline”-class storage, and as a test platform to see if we can get the performance we need to store VM images. As part of this decision, we decided to go with a backplane that supports full throughput to each drive. We also decided to use SATA drives for the storage disks, versus 7200rpm SAS drives (which would support multilane, but with the backplane we’re using it doesn’t matter), or faster SAS disks (as the SSD caches should give us all the speed we need.) For redundancy, our plan is to use replication between appliances versus running multi-head stacked to the same storage shelves; for an example of a multi-head/multi-shelf setup, see this build by the local geek Mike Horwath of ipHouse.

When purchasing a SuperMicro chassis with a SAS backplane, there are a few things you should be aware of..

There are different models of the chassis that include different style backplanes:

‘A’ style (IE – SC847A) – This chassis includes backplanes that allow direct access to each drive (no SAS expander) via SFF-8087 connectors. In the SC847 case, the front backplane has 6 SFF-8087 connectors, and the rear backplane has 3 SFF-8087 connectors. This allows full bandwidth to every drive, and minimizes the number of cables as much as possible. Downside, of course, is that you need enough controllers to provide 9 SFF-8087 connectors!
‘TQ’ style – not available for the SC847 cases, but in the SC846 chassis an example part number would be ‘SC846TQ‘. This backplane provides an individual SATA connector for each drive — in other words, you will need 24 SATA cables, and 24 SATA ports to connect them to. This will be a bit of a mess cable-wise.. with the SFF-8087 option, I don’t know why anyone would still be interested in this – if you have a reason, please comment! This is quite a common option on the 2U chassis – it can actually be difficult to purchase a 2U barebones “SuperServer” that includes SFF-8087 connectors.
‘E1′ style (IE – SC847E1) – This chassis includes backplanes with integrated 3gbit/s SAS expander, without multipath support. Each backplane has one SFF-8087 connector, so you only need two SFF-8087 ports in a SC847E1 system. The downside is that you are limited to 3gbit/s per channel – so you’d have a total of 6 drives on each 3gbit/s channel for the front backplane, and 3 drives on each channel for the rear backplane. SuperMicro also has a ‘E16′ option (IE – SC847E16) which is upcoming, and supports SATA3/SAS2, for a total of 6gbit/s per channel.
‘E2′ style (IE – SC847E2) – Similar to the SC847E1, this includes a SAS expander on the backplane, but also supports multipath for SAS drives. Each backplane has two SFF-8087 connectors. Same caveats as the E1 apply. They also have a ‘E26′ version coming out soon (IE – SC847E26) which will include SAS2 (6gbit/s) expanders.

I do wish that SuperMicro would offer a “best of both worlds” option – it would be great to be able to get a high amount of bandwidth to each drive, and also support multipath. Maybe something like a SAS2 backplane which only put two or three drives on each channel instead of six drives? If they did two drives per channel with a SAS expander, and supported multipath, it should be possible to get the same amount of total bandwidth to each drive (assuming active/active multipath), and still keep a reasonable number of total SFF-8087 connectors, plus support multipath with SAS drives, and get the bonus of controller redundancy. If anyone knows of an alternate vendor or of plans at SuperMicro to offer this, by all means, comment!

You can also choose the type of expansion slots you would like to support on the motherboard tray; you will need to match the tray to the type of motherboard that you purchase. Note that these are the same options available on their 2U chassis – the concept of the SC847 chassis essentially makes your motherboard choices the same as the 2U systems.

‘UB’ option (IE, SC847A-R1400UB) – this option supports SuperMicro’s proprietary UIO expansion cards. It uses a proprietary riser card to mount the cards horizontally, and will support 4 full-height cards and 3 low-profile cards in the SC847. They get the card density by mounting the components for one (or more) UIO cards on the opposite site of the PCB than you usually see – the connector itself is still PCI-E x8, but the bracket and components are all on the opposite side. I have not ordered a chassis that uses UIO recently, so I’m not sure if the sample part number would include riser cards or not. Note that you will need to purchase a SuperMicro board that supports UIO for this chassis.
‘LPB’ option (IE, SC847A-R1400LPB) – this option supports 7 low-profile expansion slots. If you do not have any need for full-height cards, this gives you the maximum number of high-speed slots. This is the option you will need to go with if you want to use a motherboard from a vendor other than SuperMicro.

For the system I’m building, we went with the following components:

SuperMicro SC847A-R1400LPB chassis – 36-bay chassis with backplanes that offer direct access to each drive via SFF-8087 connectors. 7 low-profile expansion slots on the motherboard tray.
SuperMicro X8DTH-6F motherboard – Intel 5520 chipset; supports Intel’s 5500- and 5600- series Xeon CPUs. Has an integrated LSI 2008 SAS2 controller, which supports 8 channels via two SFF-8087 ports. 7 PCI-E 2.0 x8 slots. 12 total memory slots. IPMI with KVMoIP integrated. Two Gig-E network ports based on Intel’s newest 82576 chipset. This board is great.. but what would make it perfect for me would be a version of the board that had 18 memory slots and 4 integrated Gig-E ports instead of two. Ah well, can’t have it all!
2x Intel E5620 Westmere processors
24gb DDR3 memory; PC3-10600, registered/ecc.
4x LSI 9211-8i PCI-E SAS-2 HBA – 2 SFF-8087 ports on each controller; same chipset (LSI 2008) as the onboard controllers. This gives me a total of 10 SFF-8087 SAS2 ports, which is one more than needed to supports all the drive bays. I should also note that we haven’t had any problems with the LSI2008-based controllers dropping offline with timeouts under OpenSolaris; with our other systems, we started with LSI 3081E-R controllers, and had no end of systems failing due to bug ID 6894775 in OpenSolaris, which as far as I’m able to tell has not yet been resolved. Swapping the controllers out with 9211-8i’s solved all the issues we were having.
Variety of SuperMicro and 3ware SFF-8087 cables in various lengths to reach the ports on the backplanes from the controller locations.
2x Seagate 750gb SATA hard drives for boot disks.
18x Hitachi 2TB SATA hard drives for data disks.
2x Intel 32gb X25-E SATA-2 SSD’s; used in ZFS for a mirrored Zero Intent Log (ZIL); write cache. (Note: 2.5″ drives; needs a SuperMicro MCP-220-00043-0N adapter to mount in the hot-swap bays.)
1x Crucial RealSSD C300 128gb SSD; used in ZFS for a L2ARC read cache. (Also a 2.5″ drive; see note above.)

We purchased the system from CDW, with our own customer-specific pricing. I’m not allowed to share what we paid, but for your reference, I’ve whipped up a shopping cart at Provantage with (essentially) the same components. There is no special pricing here; this is just the pricing that their web site listed as of May 8 2010 at 11:18am central time. Note: I have no affiliation with Provantage. I have ordered from them previously, and enjoyed their service, but cannot guarantee you will have a good experience there. The prices here may or may not be valid if you go to order. You may be able to get better pricing by talking to a customer service rep there. I also had to change a few components for parts that Provantage did not have available – namely some of the various lengths of SFF-8087 cables. I error’d on the side of ‘long’, so it should work, but I haven’t built a system with those exact cables, so can’t guarantee anything.

As you can see, the total price for this system came out at just under $8500, or $8717.14 shipped. Not bad at all for a high-performance storage array with 18 2tb data drives and the ability to add 13 more.

If we do decide that this is the route to go for our VM image storage, the config would be similar to above, with the following changes at minimum:

More memory (probably 48gb) using 8gb modules to leave room for more expansion without having to replace modules.
Switch from desktop HDDs to enterprise or nearline HDDs (6gb SAS if they are economical); probably also go with lower capacity drives, as our VMs would not require the same amount of total storage, and NexentaStor is priced by the terabyte of raw storage.
Add more (either 4x or 6x total, still used in pairs of 2) X25-E’s for ZIL/SLOG, possibly also go with 64gb instead of 32gb. (More total drives should mean more total throughput for synchronous writes. If Seagate Pulsars are available, also consider those.
Add additional RealSSD C300′s for cache drives; the more the better.
Add additional network capacity in the form of PCI-E NIC cards – either 2x 4-port Gig-E or 2x 10-GigE. This will allow us to make better use of IPMP and LACP to both distribute our network load among our core switches and use more than 2gbit total bandwidth.

In any case, on to some pictures of the chassis and build.

Chassis in shipping box – includes good quality rackmount rails and the expected box of screws, power cables, etc. First SuperMicro chassis I’ve ordered that is palletized.

Front of the chassis – 24 drive bays up front.

Rear of the chassis – 12 drive bays, and a tray for the motherboard above them. Also shows the air shroud to direct airflow over the CPUs; the only part of the chassis that feels cheap at all.. but it serves its purpose just fine.

System with the motherboard tray removed. Note that as far as the mounting is concern the tray is pretty much the same as a standard SuperMicro 2U system. You’ll need to order heatsinks, cards, etc that would work in a 2U.

View of the system from the back with the motherboard and four front fans removed. You can see a bit of the front backplane in the upper right; two of the SFF-8087 connectors are visible. All cable routing goes underneath the fans; there is plenty of room under the motherboard for cable slack. You can also see the connectors that the power supplies slide into on the upper left hand corner, and a pile of extra power cables that are unneeded for my configuration underneath that.

Another shot of the front backplane. You can see the five of the six SFF-8087 connectors (the other is on the right-hand side of the backplane which is not visible.) Also note the fans that I’ve removed to get better access to the backplane.

One of the power connectors that the fans slide into (white four-pin connector near the center of the picture); the SFF-8087 connector that is not visible in the picture above is highlighted in red.

Motherboard tray before installing the motherboard. This tray uses a different style screw system than I’ve seen before; instead of having threaded holes that you screw standoffs into, they have standoffs coming up off the bottom (one highlighted in blue), which you screw an adapter onto (highlighted in red) which the motherboard rests on and is secured to.

A partial view of the rear backplane on the system; also the bundle of extra power cables and the ribbon cable connected to the front panel.

Labels on one of the power supplies. This system includes a pair of ‘PWS-1K41P-1R’ power supplies, which output 1400W at 220V or 1100W at 120V.

Motherboard installed on tray, with the four LSI SAS HBAs in their boxes.

One of the two Intel E5620 ‘Westmere’ Xeon processors set in motherboard but not secured yet.

Both processors and 24gb of memory installed. No heatsinks yet.

Motherboard tray complete and ready to be installed in the system. Heatsinks and LSI controllers have been installed. Note the two SFF-8087 connectors integrated on the motherboard, and eight more on the four controllers.

Prep work on the rear backplane; the chassis shipped with the power cables pre-wired; I connected the SFF-8087 cable.

Motherboard tray installed back in the system; SFF-8087 cables connected to three of the four LSI controllers. I ended up moving one controller over for ease of cabling – notice the gap in the middle of the four controllers.

(Note: The pictures of the finished system below this point were taken on 5/7/2010; thanks to my coworker Colleen for letting me borrow her camera since I #natefail‘d to bring mine!)

The seven cooling fans to keep this system running nice and cool.

HBAs with all cables connected.

Finished system build with the top off. One power supply is slightly pulled out since I only have a single power cable plugged in.. if you have one cable plugged in but both power supplies installed, alas, the alarm buzzer is loud.

Front hard drive lights after system is finished – note that we don’t have every drive bay populated yet.

Rear drive lights while system is running.

The build-out on the system went fine for the most part; the only problem I ran into is that the motherboard did not have a BIOS installed which supported the relatively new Westmere processors. Fortunately I had a Nehalem E5520 I could borrow from another system to get the BIOS upgraded.. I wish the BIOS recovery procedure would work for unsupported processors, but ah well. I was pleased with the way the motherboard tray slides out; it makes it easy to get the cabling tucked underneath and routed so that they will not interfere with airflow. There also seems to be plenty of airflow to keep the 36 drives cooled.

I currently have NexentaStor 3.0 running on the system; we have not yet landed on what operating system we will run on this long-term.. but it will likely either be NexentaCore or NexentaStor. If we deploy this solution for our VM images (with some upgrades as mentioned above), we will almost certainly use NexentaStor and the VMDC plugin, but we’ll cross that bridge if we get there!

Here’s the disk configuration I have running at the moment with NexentaStor:

‘syspool’: Mirrored ZFS zpool with 2x750gb Seagate drives.
‘NateVol1′: ZFS zpool with..

2 RaidZ3 arrays with 8 2TB disks each
2 2TB disks set as spares
2 36gb Intel X25-E SSDs as a mirrored log device
1 128gb Crucial RealSSD C300 as a cache device

..and the obligatory screenshot of the data volume config:

This nets 18T usable space, and would allow for a simultaneous failure of any three data disks before there is any risk of data loss. (Each of the sub-arrays in ‘NateVol1′ have 3 parity disks – so I could also lose 3 disks from each of the sub-arrays without any issues.)

Again, this system only has two Gig-E NICs at the moment.. I’ve done I/O tests with NFS across one NIC and iSCSI across the other NIC, and can max out the bandwidth on both cards simultaneously with multiple runs of Bonnie++ 1.96 without the system breaking a sweat. I like! I should also note that this is with both deduplication and compression enabled.

Another note – before putting this into production, I did some simple “amp clamp” power usage tests on the box, with one power supply unplugged. The other power supply was plugged into 120V. While idling, it consumed 3.3A, and while running multiple copies of Bonnie in the ZFS storage pool (with all active disks lighting up nicely), it consumed 4.1A. Not bad at all for the amount of disk in this machine! I’d estimate that if the 13 additional drive bays were occupied with 2TB disks, and all those disks were active, the machine would consume about 5.5A – maybe slightly more. When we racked it up at the data center (in one of our legacy racks that is still 120V), the power usage bumped up by 3.2A combined across the A+B power, which matches nicely with my clamped readings. I’m very impressed – under 500 watts while running full out.. wow.

I will update this post once we decide on a final configuration “for real” and put this into production, but so far I’d highly recommend this configuration! If you’ve used the SC847 chassis, I’d love to hear what you’ve thought. I’d also love to try out the 45-bay storage expansion version of this chassis at some point – talk about some dense storage! :)

Mediacom, stop messing with my packets!

nc — Sat, 05 Dec 2009 03:55:52 +0000

Mediacom keeps getting worse and worse. As I’ve mentioned before. Mediacom is my ISP at home. Unfortunately, it’s the only service of reasonable speed at a decent price available for my address.

For the past few months, I’ve noticed that I seemingly randomly get redirected to a “Mediacom Assist” page, located at http://assist.mediacomcable.com. I assumed that they were being evil, and inspecting dns packets for NXDOMAIN (I don’t use their DNS servers, so it couldn’t just be Sitefinder-type junk.) However, I seemed to randomly hit links to working sites (ie, google.com) that also returned this error – just figured it was a fluke in their mechanism, and moved on. I just had a kid, after all, I’m too busy to be digging into what stupid things my ISP is doing! ;)
Tonight, I was reading the fun thread about MacPadd at Tom’s Hardware, and they have a link to the LiveJournal blog that MacPadd was posting at for awhile. The given URL was macpadd.livejournal.com. When I hit this page, Mediacom sends me over to “Assist”. I know that LiveJournal has a wildcard set up, so this should not be possible. For the fun of it, I set up a tcpdump, and checked what happens when I try to hit that page.. here are the results:

Request:

GET / HTTP/1.1
Host: macpadd.livejournal.com
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.5) Gecko/20091109 Ubuntu/9.10 (karmic) Firefox/3.5.5 FirePHP/0.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflatehttp://www.natecarlson.com/wp-admin/plugins.php
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

Reply:

HTTP/1.1 200 OK
The Search Guide redirection service has been enabled to provide helpful searches from browser queries. You entered a non-existent url and your browser attempted to redirect you with Javascript. To enable this please update your browser preferences. To turn off this feature please click this here

Ahem. The same query from a non-mediacom machine returns the expected 404 page from LiveJournal. So, apparently Mediacom is doing deep packet inspection, and rewriting 404′s under certain circumstances. The page that they redirect you to also has a pile of ‘Sponsored Links’ that are going to make them cash. Chances are this is all about the cash for them. They offer an opt-out, but who knows what else they are hijacking? This is just wrong and EVIL. For all I know, they could intercepting ads for Comcast or satellite TV and replacing them with their own. Or just about anything else, as long as money is involved.

Could we please get a net neutrality bill passed that says that ISP’s are not allowed to muck with our packets in this way? Well, maybe not – any net neutrality bill that would actually get passed would probably contain more harm than good. ;)

After figuring out what they were doing, I found lots of other posts about this.. anyone else find this to be pathetic?

why must mediacom be so terrible?

Google launches new public DNS resolver service, OpenDNS competitor

nc — Thu, 03 Dec 2009 19:30:10 +0000

Google recently launched a new Public DNS resolution service. I’m very excited to see this.. the most commonly-used public resolver up until now has been OpenDNS, which works fairly well, but has some features that I do not likely — namely, it will (by default) redirect any requests for non-existing domains to a host on their network, which will then reply to web pages with a pretty “This doesn’t exist” page. However, this can cause havoc for other services, such as SMTP. They will also attempt to auto-correct mistyped domains, which is handy, but just gives me the creeps. All of this can be disabled, but to do that, you have to sign up for an account with them, and register whatever IP you happen to be on. Pain in the rear. And it also gives them the ability to track your queries to a username, although you can ‘disable’ that feature too.

I did some quick tests against Google’s service, and it looks great so far – excellent response times, and not doing any of the “naughty stuff”. It seems that the “naughty stuff” was actually one of the things they wanted to prevent, which I completely agree with.

If you’ve been beta testing this or trying it out since the announcement, let me know how it’s going for you!

under construction..

nc — Sat, 07 Nov 2009 04:54:41 +0000

You may notice a few changes around here; I’m changing my entire site to be managed by WordPress. If you find any links that are broken, hate the new design, or just want to tell me I’m crazy, please leave a comment on this post. Thanks!

No related posts.

nexenta/opensolaris on a dell 2850 with drac 4 – virtual media issues

nc — Tue, 03 Nov 2009 17:30:20 +0000

I recently tried installing both OpenSolaris and Nexenta Core Platform on a Dell PowerEdge 2850 with a DRAC4 card. The installation process goes fine, but when rebooting into the installed OS the kernel goes into a loop of trying to reset an ATA device (‘Bus Reset’ errors — if I have a chance, I’ll repeat the process and get the exact errors.) I finally just disabled virtual media (go into the DRAC setup menu by hitting Ctrl-D on bootup, hit page-down, and use the ‘E’ option to turn it off), and after doing that, it worked just fine.

Annoying error (it’s odd that the installer works fine, but then the actual OS doesn’t; possibly because there’s no CD ‘inserted’?), but at least it’s not too hard to work around.

colocation post updated

nc — Tue, 03 Nov 2009 16:04:53 +0000

I’ve gotten quite a few questions about cheap colocation in Minneapolis via my old post here, where I was soliciting people to help fill my rack at ipHouse. Unfortunately, I no longer offer colocation services, but I have updated the post with the providers I am currently using and recommend.

This brings to mind another thing I’ve been thinking about doing for awhile. I’ve always had trouble finding definitive information about the quality of colocation providers, including information about their bandwidth providers, “real” costs (ie – cost per megabit/terabyte, cost for IP addresses, cost for power), etc. I’ve been debating putting up a review-style site where users can submit their opinions of providers, including pricing, which would then be aggregated into a searchable archive. Is this something that would be interesting to you? If so, please comment.. if there seems to be interest, I will try to put something together “real soon now”. ;)

don’t we all love airlines?

nc — Thu, 09 Jul 2009 03:22:25 +0000

via @weierd.. airlines blow. “Let’s smash a guitar, then refuse to pay for it!”. I’m glad I never fly United. Not that any of the airlines I *do* fly would be better, but hey, it was United in this case. ;)

No related posts.

odesk – online melting pot for consultants/contractors

nc — Wed, 02 Apr 2008 05:11:09 +0000

The other day, I ran across oDesk, which is a site where contractors (“Providers”) can offer their services and clients (“Buyers”) can find them. Providers can take free tests to show their skills. The site also facilitates payments, etc.

Quite interesting – I’ll have to try it out at some point. One thing I notice is that many of the providers are from other countries, and their rates are dirt cheap – it’ll be interesting to see what the quality of their work is. It would certainly be tempting to hire someone for $10/hr to do some menial labor..

colocation post updated

mosso – “the hosting cloud”?

nc — Fri, 14 Mar 2008 16:33:29 +0000

A few weeks ago, a coworker pointed out Mosso, which bills itself as “The Hosting Cloud”. Looks like a really interesting service – basically like shared hosting that will scale forever; no need to worry about individual servers, etc.

If you are just hosting a generic php/rails/etc site that gets a lot of traffic, it looks like this would be a pretty sweet (and affordable) solution. Unfortunately, most of the stuff I do (both personally and at work) is too complicated to host on this – ie, it requires third-party software like solr or other java apps.

If anyone has tried it out, let me know how it went for you!

No related posts.

Bust your phone? Need a temp replacement quick? AT&T customer? You’re in luck!

nc — Mon, 25 Feb 2008 03:36:43 +0000

Last week, my HTC Kaiser dropped off my belt in a cold, cold parking lot, and sat there for about four hours before I was able to come back and get it. Gah! It worked OK, except for a big crack on the bottom half of the screen. Gah! After a couple days, the crack had spread, so I can’t see anything on the display. This makes the phone pretty much useless. Sigh.

I put an order in for a new phone yesterday, but because it’s a Saturday, I won’t get a new phone delivered until Tuesday at the earliest. All my spare phones have been sold or eaten (my dogs + electronics != bliss), so that means I can’t make calls or see who’s calling me until a new phone comes. I figured I could probably deal with this.. but today I was at Target, and noticed that they have AT&T Gophones, which are pre-paid cell phones. On a whim, I picked up a $20 GoPhone (Motorola c168i), figuring I might be able to pop my SIM card in and make it work. Got home, tried it out, and what’dya know, it works!

It’s a junk phone (it’s only got GPRS – not even EDGE networking, horrible interface, etc), but at least I can send and receive calls, texts, etc. ;) I’d expect that this would also work for any other company that is GSM (I guess in the US that would be, er, T-Mobile).. if this helps you out, let me know!

¡unɟ sı ǝpoɔıun ‘ʍoʍ

nc — Mon, 11 Feb 2008 15:39:14 +0000

ok, this site is cool:

http://www.revfad.com/flip.html

No related posts.

interesting concept in recruiting

nc — Wed, 23 Jan 2008 05:15:55 +0000

A friend invited me to a new site – Notchup. Basically, you set up a profile, post some details, and in theory employers who are looking for employees like you will actually pay to interview you. Interesting! I might be a little bit more interested in listening to recruiters if they are willing to pay me for my time. ;)

No related posts.

at&t is just rocking! 3g in minneapolis/st.paul

nc — Thu, 01 Nov 2007 13:25:27 +0000

AT&T (ex-Cingular) has now turned 3g on in the Minneapolis/St. Paul area! What a pleasant surprise after getting the WM6 treatment yesterday. ;)

They’ve made me a happy customer.

at&t finally releases wm6 for their 8525

nc — Thu, 01 Nov 2007 13:24:44 +0000

Windows Mobile 6 is *finally* available for the (HTC-built) AT&T 8525! Yeehaw!

It’s available from here:
http://www.america.htc.com/support/8525/software-downloads.html

First impressions:

It’s fast
The GPS sharing feature works nicely
Looks about the same. Menus are very similar.

Upgrade was easy [just back your stuff up!]; highly recommended so far!

bandwidth.com outage

nc — Wed, 24 Oct 2007 16:41:56 +0000

Bandwidth.com had an outage last night, from around 4:10pm central to around 6:30 central. It affected all of their SIP trunks. Interesting! Apparently, it was attributed to a Level3 outage, but it doesn’t appear that other VoIP trunks with other L3 resellers went down. We’ll see…

No related posts.

cisco 2801s use standard laptop memory

nc — Tue, 17 Jul 2007 22:56:06 +0000

This one is purely for the geeks, and I’m just posting it because I couldn’t find a definitive answer on the web: if you have a Cisco 2801 and want to add more memory (there is 128mb soldered on the board), standard pc133 sodimm memory (ie, the same kind you use in older laptops) will work just fine! Yippie!

Cisco 2801 (revision 5.0) with 356352K/36864K bytes of memory.

inexpensive dvd+rw drive for your laptop

new resume format

nc — Sat, 07 Apr 2007 05:59:39 +0000

I’ve gotten tired of my custom-written MySQL app to handle my resume, so I decided to scrap it and try out xml-resume-library to maintain it. It’s quite interesting, and I think the results are pretty good.. my resume page has been converted over to it, check it out!

One of the really cool things that this library lets you do is have a single XML file that you can transform (with stylesheets) into plain-text, pdf, html, etc. Right now, the page is static; however, in the future, I plan to try out PHP5′s XSL library to do the transformation for me. That way, I should just be able to update a XML file, copy it to my web server, and have the resume auto-update – including links to download a pdf or text version. Cool, eh?

[Edit 2007/04/07] OK, I went and made everything dynamic now – there’s just an XML file sitting on disk that all the pages get generated from. Hurrah! Check it out.