ramblings of the village idiot » Virtualization

XenServer Networking: VLANs and MAC broadcast packets

nc — Mon, 27 Dec 2010 18:59:52 +0000

I recently ran into an issue with newer versions of XenServer that had me really really confused. I recently moved a VM that is acting as a router from a XenServer 5.0 machine to a XenServer 5.6 machine. This machine runs both OSPF and a DHCP server on top of VLAN interfaces in XenServer, and oddly, both of these services stopped working after I moved the VM.

The network configuration was pretty much identical on the old and new XenServer machines — two NICs, each NIC with a native VLAN of ’1′ (internal network), and a pile of VLANs on top of each NIC for actual networks I care about. A ‘tcpdump’ on the XenServer 5.6 dom0 would show both the DHCP and OSPF packets, but a ‘tcpdump’ within the VM would not show anything. It seemed *odd!*.

After doing some digging, I ran across a forum post at Citrix which described the same issue. The basic issue appears to be that Linux’s bridging implementation in newer kernels eats the 802.1q packets for broadcast addresses (for example, DHCP packets are addressed to the mac ‘FF:FF:FF:FF:FF:FF’).. in my case, I had the VMs on tagged interfaces on top of ‘eth1′, which XenServer had put into a bridge called ‘xenbr1′. I did not have any VMs using the raw interface for eth1 (ie – adding a VM virtual interface to xenbr1), so I just did a ‘ifconfig xenbr1 down ; brctl delbr xenbr1′ to get rid of that bridge, and everything started working.

This obviously isn’t a long-term solution; just a ‘works-for-now’ hack.. I need to look into a method to disable the native VLAN bridge, and also look into the possibility of using OpenVSwitch to handle the networking, which is supposed to resolve this issue. If you’ve run into this issue, I’d love to hear what you did to get around it!

Configuring IPMP on NexentaStor 3

nc — Fri, 10 Sep 2010 18:38:23 +0000

At my day job, we have recently started rolling out NexentaStor 3 for our VM image storage as a trial. If all goes well, our long-term plans are to eventually migrate all storage from Netapp to NexentaStor. As we started rolling out our NexentaStor trial, one missing feature we quickly ran across is the lack of IPMP (IP Multipathing) support. The network configuration interface that they provide can currently configure aggregated interfaces with the LACP protocol, but they do not have any mechanism to configure IPMP to aggregate interfaces from multiple switches. We were able to work out an approach to configure IPMP manually, and received Nexenta’s blessing to use it in our environment. (Important note: if you are going to try this on a licensed copy of NexentaStor, please check with your support team to ensure that they are ok with you making these changes.)

Updated 2010-Sep-16 — Added information on how to add static routes to configure ipmp ping targets

Server hardware configuration

First of all, I should detail what we are trying to configure. Our production machines are quite similar to the SuperMicro build I documented earlier, with a few varying specs. Here’s what is in it:

2x Intel E5620 CPUs
48G memory (6x 8G modules)
10x 1TB Seagate Constellation ES SAS drives
6x Intel 32gb X25-E drives
2x Crucial 128gb RealSSD drives
2x Intel Gigabit ET quad-port NICs

The machine has an 8TB license, with 2 of the disks configured as hot spares. The Intel SSDs are configured as 3 log-mirrors, and the RealSSDs are configured as cache devices.

Caveats

The only major caveat that I’ve hit with this configuration is that the ipmp interfaces will not be viewable via the Nexenta utilities. You can still see all the underlying interfaces; just not the ipmp ones. It’s mostly cosmetic, but is distracting and annoying.

Of course, YMMV – this worked for me, but no guarantees that it will work for you! :)

Network configuration

Desired configuration

Here is the network configuration we desire:

LACP Aggregate #1 – 4 gigabit links to Switch-1
LACP Aggregate #2 – 4 gigabit links to Switch-2
IPMP Interface #1 (balancing LACP1 and LACP2) – Native VLAN on NICs; access to our management/backend network
IPMP Interface #2 (balancing LACP1 and LACP2) – VLAN ID 100; VM storage network
IPMP Interface #3 (balancing LACP1 and LACP2) – VLAN ID 200; NFS storage network

General configuration steps

As far as I know, you cannot do VLAN tagging on top of an IPMP trunk on Solaris, which means that we need to create the VLAN interfaces on each of the aggregate interfaces, and then create three separate IPMP interfaces – one per vlan. Here are the basic configuration steps:

Via NMC: Create individual LACP aggregates (aggr1 and aggr2), with igbX interfaces as members.
Via NMC: Create VLAN interfaces ’100′ and ’200′ on top of both ‘aggr1′ and ‘aggr2′. This will create the interfaces ‘aggr100001′, ‘aggr100002′, ‘aggr200001′, and ‘aggr200002′.
Via NMC: Configure an IP address from within each VLAN on each of these six interfaces. This will allow IPMP to use ICMP probes in addition to link detection to try to find failed links.
Via the console: Configure the three IPMP interfaces, and add the six aggr interfaces to the proper IPMP groups.

NMC – Create LACP aggregates

This assumes that whatever interface you configured during installation is *not* one of the interfaces you desire to be part of the aggregate. If that is not true, you will need to be on the system console (via IPMI hopefully!), and destroy that interface first. Here is an example of how to create the aggregates (output from NMC; so this is how it ends up being configured):

nmc@nexenta:/$ setup network aggregation create
Links to aggregate  : igb2,igb3,igb4,igb5
LACP mode           : active
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr1           L3,L4    auto                 active        short       -----
nmc@nexenta:/$ setup network aggregation create
Links to aggregate  : igb6,igb7,igb8,igb9
LACP mode           : active
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr2           L3,L4    auto                 active        short       -----

NMC – Create vlan interfaces on each aggregate interface

nmc@nexenta:/$ setup network interface aggr1 vlan create
VLAN Id : 100
aggr100001: flags=201000842 mtu 9000 index 39
        inet 0.0.0.0 netmask 0
        ether 0:1b:21:6c:3c:de

nmc@nexenta:/$ setup network interface aggr2 vlan create
VLAN Id : 100
aggr100002: flags=201000842 mtu 9000 index 40
        inet 0.0.0.0 netmask 0
        ether 0:1b:21:6c:3d:de

nmc@nexenta:/$ setup network interface aggr1 vlan create
VLAN Id : 200
aggr200001: flags=201000842 mtu 9000 index 41
        inet 0.0.0.0 netmask 0
        ether 0:1b:21:6c:3e:de 

nmc@nexenta:/$ setup network interface aggr2 vlan create
VLAN Id : 200
aggr200002: flags=201000842 mtu 9000 index 42
        inet 0.0.0.0 netmask 0
        ether 0:1b:21:6c:3f:de

NMC – Assign IP addresses

This assumes the following IP ranges:
Native VLAN: 10.100.0.0/24
VLAN 100: 10.100.100.0/24
VLAN 200: 10.100.200.0/24

It also assumes that aggregate 1 should be assigned .2 within each /24, and aggregate 2 should be assigned .3. The ipmp shared interface will be assigned .1.

nmc@nexenta:/$ setup network interface vlan aggr1 static
aggr1 IP address: 10.100.0.2
aggr1 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr1 as 10.100.0.2/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr2 static
aggr2 IP address: 10.100.0.3
aggr2 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr2 as 10.100.0.3/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr100001 static
aggr100001 IP address: 10.100.100.2
aggr100001 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr100001 as 10.100.100.2/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr100002 static
aggr100002 IP address: 10.100.100.3
aggr100002 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr100002 as 10.100.100.3/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr200001 static
aggr200001 IP address: 10.100.200.2
aggr200001 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr200001 as 10.100.200.2/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr200002 static
aggr200002 IP address: 10.100.200.3
aggr200002 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr200002 as 10.100.200.3/255.255.255.0 ... OK.

Console – Set up IPMP

First, we need to get into expert mode.

nmc@nexenta:/$ options expert_mode=1                                                                          

nmc@nexenta:/$ !bash
You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). CAUTION: NexentaStor
appliance is not a general purpose operating system: managing the appliance via Unix shell is NOT
recommended. This management console (NMC) is the command-line interface (CLI) of the appliance,
specifically designed for all command-line interactions. Using Unix shell without authorization of your
support provider may not be supported and MAY VOID your license agreement. To display the agreement,
please use 'show appliance license agreement'.
Proceed anyway? (type No to return to the management console)  Yes

root@nexenta:/volumes#

Next step is to set up the hostname files for each of the IPMP interfaces. I will name the interfaces as follows:

ipmp0 – ipmp interface for aggr1 and aggr2
ipmp100000 – ipmp interface for aggr100001 and aggr100002
ipmp200000 – ipmp interface for aggr200001 and aggr200002

These files also set the IP address that we would like the system to apply to these.

root@nexenta:/etc# cat hostname.ipmp0
ipmp group ipmp0 10.100.0.1 netmask 255.255.255.0 up
root@nexenta:/etc# cat hostname.ipmp100000
ipmp group ipmp100000 10.100.100.1 netmask 255.255.255.0 up
root@nexenta:/etc# cat hostname.ipmp200000
ipmp group ipmp200000 10.100.200.1 netmask 255.255.255.0 up

Next, the groups need to be configured in the hostname. files. We need to add ‘group -failover up’ to each of these files, before the ‘up’ at the end. Here are the files after the changes are made:

root@nexenta:/etc# for i in /etc/hostname.aggr* ; do echo $i ; cat $i ; done
/etc/hostname.aggr1
10.100.0.2 netmask 255.255.255.0 mtu 9000  broadcast +  group ipmp0 -failover up
/etc/hostname.aggr2
10.100.0.3 netmask 255.255.255.0 mtu 9000  broadcast + group ipmp0 -failover up
/etc/hostname.aggr100001
10.100.100.2 netmask 255.255.255.0  broadcast +  group ipmp100000 -failover up
/etc/hostname.aggr100002
10.100.100.3 netmask 255.255.255.0  broadcast + group ipmp100000 -failover up
/etc/hostname.aggr200001
10.100.200.2 netmask 255.255.255.0  broadcast +  group ipmp200000 -failover up
/etc/hostname.aggr200002
10.100.200.3 netmask 255.255.255.0  broadcast +  group ipmp200000 -failover up

Now that all the interface configs are in place, we can apply it.. here’s the easiest way I figured out; if anyone knows a better way I’d love to hear it!

# svcadm disable svc:/network/physical:default
# for i in aggr1 aggr2 aggr100001 aggr100002 aggr200001 aggr200002 ipmp0 ipmp100000 ipmp200000 ; ifconfig $i unplumb ; done
# svcadm enable svc:/network/physical:default

At this point, all of your interfaces should be up, and all the IP addresses should be pingable. Make sure that you can ping the individual interface IPs, and the IPMP IPs. You should be able to use the ‘ipmpstat’ command to see information about your groups; IE:

root@nexenta:/volumes# ipmpstat -a
ADDRESS                   STATE  GROUP       INBOUND     OUTBOUND
nexenta-vl100        up     ipmp100000    aggr100001    aggr100002 aggr100001
nexenta-vl200        up     ipmp200000    aggr200001    aggr200002 aggr200001
nexenta                up     ipmp0       aggr1       aggr2 aggr1

Note that this configuration provides failover and outbound load balancing, but it does not provide inbound load balancing. If you would like inbound load balancing, you need to configure an IP alias on each of the ipmp interfaces, and then mix up which IP you use from the host that is connecting to your Nexenta machine (or use multipathing if it’s iSCSI!)

One last thing – once everything is configured, you will probably want to define your own ping targets. You can view the ones that ipmp picked automatically by running ‘ipmpstat -t’. On our configuration, on one VLAN, two Nexenta nodes picked each other.. so when we took machine two down (intentionally), machine one marked that interface down, and then when we booted machine two back up, it could not reach machine one’s interface, and marked its interface on that vlan down. Nice race condition. Oddly, mpathd (the daemon that does the checking) does not use a configuration file for ping targets, but instead relies on host routes. What we’ve done is added routes to the individual IP addresses that we would like to have it monitor by using the NMC command ‘setup network routes add’, and specifying the IP address to monitor as both the ‘Destination’ and the ‘Gateway’. We picked four to five IPs on each VLAN that were stable hosts (routers, Xen domain-0′s and the like), and added them on both hosts.. this will give more consistent results, as multiple core machines would have to go down before the interface would be disabled incorrectly.

I hope this helps! Please feel free to leave a comment if you run into any trouble getting it working.

Review: SuperMicro’s SC847 (SC847A) 4U chassis with 36 drive bays

nc — Fri, 07 May 2010 07:20:01 +0000

[Or "my quest for the ultimate home-brew storage array."] At my day job, we use a variety of storage solutions based on the type of data we’re hosting. Over the last year, we have started to deploy SuperMicro-based hardware with OpenSolaris and ZFS for storage of some classes of data. The systems we have built previously have not had any strict performance requirements, and were built with SuperMicro’s SC846E2 chassis, which supports 24 total SAS/SATA drives, with an integrated SAS expander in the backplane to support multipath to SAS drives. We’re building out a new system that we hope to be able to promote to tier-1 for some “less critical data”, so we wanted better drive density and more performance. We landed on the relatively new SuperMicro SC847 chassis, which supports 36 total 3.5″ drives (24 front and 12 rear) in a 4U enclosure. While researching this product, I didn’t find many reviews and detailed pictures of the chassis, so figured I’d take some pictures while building the system and post them for the benefit of anyone else interested in such a solution.

Updates:
[2010-05-19 Some observations on power consumption appended to the bottom of the post.]
[2010-05-20 Updated notes a bit to clarify that I am not doing multilane or SAS - thanks for reminding me to clarify that Mike.]
[2011-12-20 Replacing references of 'port multiplier' with 'SAS Expander' to reflect the actual technology in use.. thanks commenters Erik and Aaron for reminding me that port multiplier is not a generic term, and sorry it took me so long to fix the terminology!]

In the systems we’ve built so far, we’ve only deployed SATA drives since OpenSolaris can still get us decent performance with SSD for read and write cache. This means that in the 4U cases we’ve used with integrated SAS expanders, we have only used one of the two SFF-8087 connectors on the backplane; this works fine, but limits the total throughput of all drives in the system to 4 3gbit/s channels (on this chassis, 6 drives would be on each 3gbit channel.) On our most recent build, we built it with the intention of using it both for “nearline”-class storage, and as a test platform to see if we can get the performance we need to store VM images. As part of this decision, we decided to go with a backplane that supports full throughput to each drive. We also decided to use SATA drives for the storage disks, versus 7200rpm SAS drives (which would support multilane, but with the backplane we’re using it doesn’t matter), or faster SAS disks (as the SSD caches should give us all the speed we need.) For redundancy, our plan is to use replication between appliances versus running multi-head stacked to the same storage shelves; for an example of a multi-head/multi-shelf setup, see this build by the local geek Mike Horwath of ipHouse.

When purchasing a SuperMicro chassis with a SAS backplane, there are a few things you should be aware of..

There are different models of the chassis that include different style backplanes:

‘A’ style (IE – SC847A) – This chassis includes backplanes that allow direct access to each drive (no SAS expander) via SFF-8087 connectors. In the SC847 case, the front backplane has 6 SFF-8087 connectors, and the rear backplane has 3 SFF-8087 connectors. This allows full bandwidth to every drive, and minimizes the number of cables as much as possible. Downside, of course, is that you need enough controllers to provide 9 SFF-8087 connectors!
‘TQ’ style – not available for the SC847 cases, but in the SC846 chassis an example part number would be ‘SC846TQ‘. This backplane provides an individual SATA connector for each drive — in other words, you will need 24 SATA cables, and 24 SATA ports to connect them to. This will be a bit of a mess cable-wise.. with the SFF-8087 option, I don’t know why anyone would still be interested in this – if you have a reason, please comment! This is quite a common option on the 2U chassis – it can actually be difficult to purchase a 2U barebones “SuperServer” that includes SFF-8087 connectors.
‘E1′ style (IE – SC847E1) – This chassis includes backplanes with integrated 3gbit/s SAS expander, without multipath support. Each backplane has one SFF-8087 connector, so you only need two SFF-8087 ports in a SC847E1 system. The downside is that you are limited to 3gbit/s per channel – so you’d have a total of 6 drives on each 3gbit/s channel for the front backplane, and 3 drives on each channel for the rear backplane. SuperMicro also has a ‘E16′ option (IE – SC847E16) which is upcoming, and supports SATA3/SAS2, for a total of 6gbit/s per channel.
‘E2′ style (IE – SC847E2) – Similar to the SC847E1, this includes a SAS expander on the backplane, but also supports multipath for SAS drives. Each backplane has two SFF-8087 connectors. Same caveats as the E1 apply. They also have a ‘E26′ version coming out soon (IE – SC847E26) which will include SAS2 (6gbit/s) expanders.

I do wish that SuperMicro would offer a “best of both worlds” option – it would be great to be able to get a high amount of bandwidth to each drive, and also support multipath. Maybe something like a SAS2 backplane which only put two or three drives on each channel instead of six drives? If they did two drives per channel with a SAS expander, and supported multipath, it should be possible to get the same amount of total bandwidth to each drive (assuming active/active multipath), and still keep a reasonable number of total SFF-8087 connectors, plus support multipath with SAS drives, and get the bonus of controller redundancy. If anyone knows of an alternate vendor or of plans at SuperMicro to offer this, by all means, comment!

You can also choose the type of expansion slots you would like to support on the motherboard tray; you will need to match the tray to the type of motherboard that you purchase. Note that these are the same options available on their 2U chassis – the concept of the SC847 chassis essentially makes your motherboard choices the same as the 2U systems.

‘UB’ option (IE, SC847A-R1400UB) – this option supports SuperMicro’s proprietary UIO expansion cards. It uses a proprietary riser card to mount the cards horizontally, and will support 4 full-height cards and 3 low-profile cards in the SC847. They get the card density by mounting the components for one (or more) UIO cards on the opposite site of the PCB than you usually see – the connector itself is still PCI-E x8, but the bracket and components are all on the opposite side. I have not ordered a chassis that uses UIO recently, so I’m not sure if the sample part number would include riser cards or not. Note that you will need to purchase a SuperMicro board that supports UIO for this chassis.
‘LPB’ option (IE, SC847A-R1400LPB) – this option supports 7 low-profile expansion slots. If you do not have any need for full-height cards, this gives you the maximum number of high-speed slots. This is the option you will need to go with if you want to use a motherboard from a vendor other than SuperMicro.

For the system I’m building, we went with the following components:

SuperMicro SC847A-R1400LPB chassis – 36-bay chassis with backplanes that offer direct access to each drive via SFF-8087 connectors. 7 low-profile expansion slots on the motherboard tray.
SuperMicro X8DTH-6F motherboard – Intel 5520 chipset; supports Intel’s 5500- and 5600- series Xeon CPUs. Has an integrated LSI 2008 SAS2 controller, which supports 8 channels via two SFF-8087 ports. 7 PCI-E 2.0 x8 slots. 12 total memory slots. IPMI with KVMoIP integrated. Two Gig-E network ports based on Intel’s newest 82576 chipset. This board is great.. but what would make it perfect for me would be a version of the board that had 18 memory slots and 4 integrated Gig-E ports instead of two. Ah well, can’t have it all!
2x Intel E5620 Westmere processors
24gb DDR3 memory; PC3-10600, registered/ecc.
4x LSI 9211-8i PCI-E SAS-2 HBA – 2 SFF-8087 ports on each controller; same chipset (LSI 2008) as the onboard controllers. This gives me a total of 10 SFF-8087 SAS2 ports, which is one more than needed to supports all the drive bays. I should also note that we haven’t had any problems with the LSI2008-based controllers dropping offline with timeouts under OpenSolaris; with our other systems, we started with LSI 3081E-R controllers, and had no end of systems failing due to bug ID 6894775 in OpenSolaris, which as far as I’m able to tell has not yet been resolved. Swapping the controllers out with 9211-8i’s solved all the issues we were having.
Variety of SuperMicro and 3ware SFF-8087 cables in various lengths to reach the ports on the backplanes from the controller locations.
2x Seagate 750gb SATA hard drives for boot disks.
18x Hitachi 2TB SATA hard drives for data disks.
2x Intel 32gb X25-E SATA-2 SSD’s; used in ZFS for a mirrored Zero Intent Log (ZIL); write cache. (Note: 2.5″ drives; needs a SuperMicro MCP-220-00043-0N adapter to mount in the hot-swap bays.)
1x Crucial RealSSD C300 128gb SSD; used in ZFS for a L2ARC read cache. (Also a 2.5″ drive; see note above.)

We purchased the system from CDW, with our own customer-specific pricing. I’m not allowed to share what we paid, but for your reference, I’ve whipped up a shopping cart at Provantage with (essentially) the same components. There is no special pricing here; this is just the pricing that their web site listed as of May 8 2010 at 11:18am central time. Note: I have no affiliation with Provantage. I have ordered from them previously, and enjoyed their service, but cannot guarantee you will have a good experience there. The prices here may or may not be valid if you go to order. You may be able to get better pricing by talking to a customer service rep there. I also had to change a few components for parts that Provantage did not have available – namely some of the various lengths of SFF-8087 cables. I error’d on the side of ‘long’, so it should work, but I haven’t built a system with those exact cables, so can’t guarantee anything.

As you can see, the total price for this system came out at just under $8500, or $8717.14 shipped. Not bad at all for a high-performance storage array with 18 2tb data drives and the ability to add 13 more.

If we do decide that this is the route to go for our VM image storage, the config would be similar to above, with the following changes at minimum:

More memory (probably 48gb) using 8gb modules to leave room for more expansion without having to replace modules.
Switch from desktop HDDs to enterprise or nearline HDDs (6gb SAS if they are economical); probably also go with lower capacity drives, as our VMs would not require the same amount of total storage, and NexentaStor is priced by the terabyte of raw storage.
Add more (either 4x or 6x total, still used in pairs of 2) X25-E’s for ZIL/SLOG, possibly also go with 64gb instead of 32gb. (More total drives should mean more total throughput for synchronous writes. If Seagate Pulsars are available, also consider those.
Add additional RealSSD C300′s for cache drives; the more the better.
Add additional network capacity in the form of PCI-E NIC cards – either 2x 4-port Gig-E or 2x 10-GigE. This will allow us to make better use of IPMP and LACP to both distribute our network load among our core switches and use more than 2gbit total bandwidth.

In any case, on to some pictures of the chassis and build.

Chassis in shipping box – includes good quality rackmount rails and the expected box of screws, power cables, etc. First SuperMicro chassis I’ve ordered that is palletized.

Front of the chassis – 24 drive bays up front.

Rear of the chassis – 12 drive bays, and a tray for the motherboard above them. Also shows the air shroud to direct airflow over the CPUs; the only part of the chassis that feels cheap at all.. but it serves its purpose just fine.

System with the motherboard tray removed. Note that as far as the mounting is concern the tray is pretty much the same as a standard SuperMicro 2U system. You’ll need to order heatsinks, cards, etc that would work in a 2U.

View of the system from the back with the motherboard and four front fans removed. You can see a bit of the front backplane in the upper right; two of the SFF-8087 connectors are visible. All cable routing goes underneath the fans; there is plenty of room under the motherboard for cable slack. You can also see the connectors that the power supplies slide into on the upper left hand corner, and a pile of extra power cables that are unneeded for my configuration underneath that.

Another shot of the front backplane. You can see the five of the six SFF-8087 connectors (the other is on the right-hand side of the backplane which is not visible.) Also note the fans that I’ve removed to get better access to the backplane.

One of the power connectors that the fans slide into (white four-pin connector near the center of the picture); the SFF-8087 connector that is not visible in the picture above is highlighted in red.

Motherboard tray before installing the motherboard. This tray uses a different style screw system than I’ve seen before; instead of having threaded holes that you screw standoffs into, they have standoffs coming up off the bottom (one highlighted in blue), which you screw an adapter onto (highlighted in red) which the motherboard rests on and is secured to.

A partial view of the rear backplane on the system; also the bundle of extra power cables and the ribbon cable connected to the front panel.

Labels on one of the power supplies. This system includes a pair of ‘PWS-1K41P-1R’ power supplies, which output 1400W at 220V or 1100W at 120V.

Motherboard installed on tray, with the four LSI SAS HBAs in their boxes.

One of the two Intel E5620 ‘Westmere’ Xeon processors set in motherboard but not secured yet.

Both processors and 24gb of memory installed. No heatsinks yet.

Motherboard tray complete and ready to be installed in the system. Heatsinks and LSI controllers have been installed. Note the two SFF-8087 connectors integrated on the motherboard, and eight more on the four controllers.

Prep work on the rear backplane; the chassis shipped with the power cables pre-wired; I connected the SFF-8087 cable.

Motherboard tray installed back in the system; SFF-8087 cables connected to three of the four LSI controllers. I ended up moving one controller over for ease of cabling – notice the gap in the middle of the four controllers.

(Note: The pictures of the finished system below this point were taken on 5/7/2010; thanks to my coworker Colleen for letting me borrow her camera since I #natefail‘d to bring mine!)

The seven cooling fans to keep this system running nice and cool.

HBAs with all cables connected.

Finished system build with the top off. One power supply is slightly pulled out since I only have a single power cable plugged in.. if you have one cable plugged in but both power supplies installed, alas, the alarm buzzer is loud.

Front hard drive lights after system is finished – note that we don’t have every drive bay populated yet.

Rear drive lights while system is running.

The build-out on the system went fine for the most part; the only problem I ran into is that the motherboard did not have a BIOS installed which supported the relatively new Westmere processors. Fortunately I had a Nehalem E5520 I could borrow from another system to get the BIOS upgraded.. I wish the BIOS recovery procedure would work for unsupported processors, but ah well. I was pleased with the way the motherboard tray slides out; it makes it easy to get the cabling tucked underneath and routed so that they will not interfere with airflow. There also seems to be plenty of airflow to keep the 36 drives cooled.

I currently have NexentaStor 3.0 running on the system; we have not yet landed on what operating system we will run on this long-term.. but it will likely either be NexentaCore or NexentaStor. If we deploy this solution for our VM images (with some upgrades as mentioned above), we will almost certainly use NexentaStor and the VMDC plugin, but we’ll cross that bridge if we get there!

Here’s the disk configuration I have running at the moment with NexentaStor:

‘syspool’: Mirrored ZFS zpool with 2x750gb Seagate drives.
‘NateVol1′: ZFS zpool with..

2 RaidZ3 arrays with 8 2TB disks each
2 2TB disks set as spares
2 36gb Intel X25-E SSDs as a mirrored log device
1 128gb Crucial RealSSD C300 as a cache device

..and the obligatory screenshot of the data volume config:

This nets 18T usable space, and would allow for a simultaneous failure of any three data disks before there is any risk of data loss. (Each of the sub-arrays in ‘NateVol1′ have 3 parity disks – so I could also lose 3 disks from each of the sub-arrays without any issues.)

Again, this system only has two Gig-E NICs at the moment.. I’ve done I/O tests with NFS across one NIC and iSCSI across the other NIC, and can max out the bandwidth on both cards simultaneously with multiple runs of Bonnie++ 1.96 without the system breaking a sweat. I like! I should also note that this is with both deduplication and compression enabled.

Another note – before putting this into production, I did some simple “amp clamp” power usage tests on the box, with one power supply unplugged. The other power supply was plugged into 120V. While idling, it consumed 3.3A, and while running multiple copies of Bonnie in the ZFS storage pool (with all active disks lighting up nicely), it consumed 4.1A. Not bad at all for the amount of disk in this machine! I’d estimate that if the 13 additional drive bays were occupied with 2TB disks, and all those disks were active, the machine would consume about 5.5A – maybe slightly more. When we racked it up at the data center (in one of our legacy racks that is still 120V), the power usage bumped up by 3.2A combined across the A+B power, which matches nicely with my clamped readings. I’m very impressed – under 500 watts while running full out.. wow.

I will update this post once we decide on a final configuration “for real” and put this into production, but so far I’d highly recommend this configuration! If you’ve used the SC847 chassis, I’d love to hear what you’ve thought. I’d also love to try out the 45-bay storage expansion version of this chassis at some point – talk about some dense storage! :)

Sun’s Unified Storage 7210 – designed to disappoint?

nc — Tue, 23 Feb 2010 18:49:59 +0000

At my day job, we purchased 3 Sun (er, Oracle) 7210′s (part of their ‘Unified Storage’ platform; the machines are X4540′s with their specialized Unified Storage OS) on the recommendation of one of our vendors. The models we purchased contained 48 250gb disks and 32gb memory — no SSD’s. Our intent was to use these for our VM image storage (VM infrastructure is Xen, using LVM-over-iSCSI to the 7210′s.) We planned to install two units in production, one in our DR environment, and to use ZFS replication to have VM disaster recovery bliss.

It’s been a disaster.

Update [2010/02/26]: Sun has really stepped up to the plate and gotten the right people on this issue. I’m not sure if it’s what I posted here, the numerous ex-Sun contacts that I had who pinged people they knew, or the president of our VAR hammering at some of his contacts within Sun/Oracle. I will keep updating as we work towards RCA’s and a resolution.

Update [2010/08/27]: Thanks to some hard work from our reseller and people at Oracle, we were able to return these units a few months ago. I wish we had been able to work through the issues with Oracle, but needed to get something that we could trust online ASAP. For the record, I do think that we probably did receive a “batch” of bad units; I still have not heard from anyone else who has had multiple independent units fail simultaneously as we did. I will also keep comments open here, and encourage anyone who has had great deployments to post a followup – I do believe that there are people that have had great success with these units out there, or else they wouldn’t be selling so well! :) And again, thank you to the hard work from our reseller, and for all the people from Oracle (and those who used to work for Sun but did not move to Oracle) who did their best to help us!

Here’s the pre-sales issues we had with the unit..

ZFS does not use traditional raid controllers, and uses your system memory as cache. This is great for reads (massive cache!), but for writes, it means that it’s extremely dangerous to turn write caching on. The memory used for write caching (which could be up to 32G in our system, or more in one with more memory) is not battery-backed, so if the system fails in any way (power supply failure, crash, power failure, etc.), any data in that cache is gone. On a “normal” ZFS configuration with SSD’s for the ZIL, there wouldn’t be a need for write caching in memory – data is cached on the write-optimized SSD’s, and if your system crashes, it will still be there. However, the model we ended up with does not contain a SSD, and the Sun Oracle-supported SSD for this unit is around $6000 (18gb ‘Logzilla’.) YIKES.
Without SSD or write cache (we need the write cache disabled for the reasons described above), the maximum streaming synchronous write speed I have been able to get over iSCSI is ~35MB/s – that’s with triple-mirrored disks. With this configuration, if I have multiple virtual machines running and start a write-heavy process, the other VMs will slow to a crawl with high I/O wait. Not acceptable in production.
For the reasons above, I firmly believe that Sun Oracle should NOT sell any ZFS-based storage appliances without SSDs – at least not without a big warning sticker that says “Make sure you can afford to either lose any data in cache (in some environments this would not be a huge issue), or work with the speeds described above.
Sun Oracle no longer offers Try-and-Buy programs. We’ve been told that once you open the unit, it’s yours. Not a strong sign that they are confident in their technology. Oh, and we were told this after we had purchased and installed the units.

Now, for the technical problems we’ve been having..

The DR unit has had problems since a few days after we installed it. It’s been throwing fan errors, which Sun Oracle has told us is a ‘firmware bug’ with no fix (even though it only happens on one of three nodes) – the workaround is to restart the service processor. It also has had a bad CPU, and worst of all, it started rebooting every night. Sun Oracle support had us jump through a zillion hoops for over a month to try to debug the nightly reboot issue, replaced various components (including the motherboard), and never came up with a workable solution. Then, a few weeks ago, after 4-5 days of not touching the system (I had been assigned to other tasks and the coworker who was taking it over was on vacation, plus a weekend), the reboots suddenly stopped. We were never able to get the reboots to either alert Sun Oracle via the phone home mechanism, or trigger the alerts we had set up via the web GUI – even though it was rebooting unexpectedly. There was no easy way to figure out that it had rebooted, in fact – I ended up having to go to the shell (naughty!) and run ‘last’ to see the reboot times. If we were relying on the GUI, and didn’t have monitoring set up of the system (which is what alerted us to the reboots), we wouldn’t have known until we put a production load on it — which, since this is a DR unit, would only have been when we failed to our DR site. Sun Oracle support has not been able to offer a RCA, except to say that whenever it rebooted the mpt driver threw errors. Since it ‘fixed itself’, Sun Oracle considered the case closed. The combination of the machine being flaky from the start combined with the fact that Sun Oracle couldn’t debug the issue has made us push for a replacement unit, but it seems nobody in Sun Oracle support has the authority to replace a chunk of hardware that can’t be trusted.
A few weeks into debugging the issues with the DR unit, both of the production units (which were taking a test workload) developed problems at the same time. One of them stopped answering on the VLAN that we were running iSCSI on (which was trunked on top of the management VLAN; that VLAN continued to work fine), and the other turned itself off. The best part is this happened within 10 minutes of each other. Again, Sun Oracle support is unable to give us a root cause for either issue. The systems are completely isolated from each other physically (separate circuits, separate racks in different areas of the data center, etc) — the only shared component is the network (both are dual-homed to two switches.) Nothing else in the data center exhibited any issues during this period – the switches didn’t show any oddities, other systems and PDU’s on the same UPS branch did not show power issues, etc. Again, Sun Oracle’s response was “if it happens again let us know.” Even though we would have taken a full production outage of _everything_ because of this. Again, neither of the units triggered the phone-home or self-alert functionality when these issues occurred.
When troubleshooting with Sun Oracle, their first step is to reboot the system into a debug mode. These are production storage appliances – having to reboot to get a RCA for a previous issue is *not* acceptable! Again, this makes it feel more like “this a server that should be in a redundant zero-shared config” more than a storage appliance.

Really, I can understand technical issues; they happen with every product. However, it appears that the way the specialized version of Solaris is set up on these, they are incapable of getting useful logs. On top of that, the support has been worse than useless. They have chewed through a ton of our time (we’ve probably spent at least 4 man-weeks working with them on this), and have not come back with anything useful. After escalating the tickets, nobody has the authority to either replace a box or allow us to return it. The escalation process is slow and doesn’t actually really seem to do anything; after requesting escalations, we don’t get higher-level contacts (the only way I’ve managed to do that so far is by blind-calling the service center when our regular tech wasn’t available, and requesting a callback), and the people it’s escalated to still have no power to resolve our issues. Every time we request escalation it also seems to slow the process by at least a few days – it’s crazy. The support people all seem to want to help, but the system is stacked against them.

So, right now I’m stuck with 3 7210′s that have all had issues, that I don’t trust in production, and I can’t return. Thanks a lot Sun Oracle! What happened to the old-fashioned Sun, where the gear was way too expensive (and yes, these 7210′s were expensive), but the support made up for it?

My advice to you if you are looking at a Unified Storage appliance is to run far away – the ones with SSDs included would in all likelyhood perform as expected, but the quality of the support and lack of customer service will scare me away from ever making a Sun Oracle purchase again.

Update [2010/02/24] – After chatting with many contacts that I used to have within Sun, I have changed most of the relevant ‘Sun’ names to ‘Oracle’. It seems that when Oracle purchased Sun, many, many good people were let go. Bummer.

Web interface with Java console for XenServer (and Xen Cloud Platform)

nc — Fri, 04 Dec 2009 22:36:12 +0000

One of the things that is nice with the release of Xen Cloud Platform (which I mentioned previously) is Project XVP. As the site mentions, it has four major components – xvp (a VNC proxy to allow you to connect to the console of a VM hosted on XenServer or Xen Cloud Platform), xvpviewer (TightVNC’s Java client extended to support XenServer-specific functions), xvpweb (a simple web interface to manage your VM’s), and xvpdiscover (an app to build a config file suitable for xvpweb and xvp.)

The executive summary is that there is now an open-source project available to get console access to XenServer machines – without the need for XenCenter on Windows. I believe this is also the only method to get access to the console on Xen Cloud Platform machines, but would love to be proven wrong. ;)

I followed the directions provided at the xvp page to build the RPM’s.. I was building on a CentOS 5.0 x64 system (I’m usually a Debian user and this was a spare VM I had laying around), and while building libxenserver, ran into the following errors:

/usr/bin/ld: skipping incompatible /usr/lib/libgssapi_krb5.so when searching for -lgssapi_krb5 /usr/bin/ld: skipping incompatible /usr/lib/libgssapi_krb5.a when searching for -lgssapi_krb5

It looks like older versions of RHEL/CentOS5 on 64-bit put the wrong development libraries in /usr/lib, gah! This page has one method of fixing this; the other is to remove the devel packages, and re-install only the 64-bit versions.. RHEL/CentOS bugs me on 64-bit since they install both 32-bit and 64-bit versions of stuff in the same areas, sigh.

In any case, after fixing the dev libraries, the xvp SRPM built as expected. Be sure that you have the CentOS ‘Extras’ repo enabled (needed for php-mcrypt), and install the RPM’s generated.

To generate a config file, you can run the ‘xvpdiscover’ command. It will prompt you for a hostname, username, and password (these are creds for a XenServer host), vnc password (this is a password needed to VNC in), and output file. Save the output to /etc/xvp.conf, and make sure the permissions are set so that your web server can access it. Once this is done, fire up xvp with ‘/etc/init.d/xvp start’.

After doing this, when I tried to hit the web page. I got a blank page back, and the Apache error log shows:

PHP Parse error: syntax error, unexpected T_STRING in /var/www/html/xvpweb/index.php on line 35

Turns out that CentOS leaves short_open_tags turned on by default. Edit /etc/php.ini, and change ‘short_open_tag = On’ to ‘short_open_tag = Off’, and restart Apache.

After doing that, I hit the web page again, and after a few moments, a list of all the VM’s along with their status popped up. There is a small Console icon by each VM, click it, and you’ve got the console. The console gives you simple operations such as shutting down the VM, rebooting the VM, etc. Nice!

The web interface is actually not necessary at all if you just want to use a raw VNC client and don’t care about those extra functions – you can check /var/log/xvp.log once xvp is started, and get the VNC port number for each VM. Then you can use your favorite VNC client to connect.

Things I’d love to see in the future:

Automatic reconfiguration — if you add or remove a VM right now, you will need to regenerate the config file. You can do this via cron, but it’d be nice if it could happen dynamically.
Ability to authenticate both the web page and the VNC passwords against the XenServer database. I guess VNC password doesn’t matter if you never manually connect to VNC, but it would be great to not have to configure authentication sources in two different places.
Ability to see tags, descriptions, etc in the web interface, and sort in various ways (next item down covers this, though.)
The web interface is very, very basic. It’d be nice to see other simple features integrated – IE, reboot, live migrate, etc. Shouldn’t be hard at all to do, just needs someone to do it.
It would also be nice to see the web interface extended to fully replace XenCenter, at least for monitoring functions (ie, graphing, etc.). That will take a lot of development. ;) I’d love to be able to delete XenCenter (and my Windows VM).
Debian or Ubuntu packages. ;) This would not be difficult at all.

In any case, this means I no longer have to fire up a Windows VM just to access the consoles of my XenServer VM’s, which is very much appreciated!

Xen Cloud Platform

nc — Tue, 03 Nov 2009 13:58:24 +0000

Wow, that was fast! As I mentioned yesterday, Citrix has open sourced the server-side components of XenServer. It came in the form of Xen Cloud Platform. As I expected, they open-sourced xapi, but not the client or the Windows components.

In any case, I will update once I play with it.. but this is great!

Update: Another post on the xen.org blog details the release of the xapi toolstack. Sounds pretty much the same as the version included in XenServer.. only thing I don’t recall being available in the XenServer version is the Java applet for console feature, and the Javascript XenAPI client – I’m guessing those were added to allow users to get console access without needing the Windows client. Nice!

citrix to open-source xenserver

nc — Mon, 02 Nov 2009 20:18:32 +0000

Well, I didn’t see this one coming. It appears that Citrix is about to open-source XenServer. There are few details so far, but I’d expect that what they are open-sourcing are the server-side components — IE, their management framework for Xen. I’d be surprised if they open-sourced their .Net XenServer client, but by open sourcing the server side of the stack, I wouldn’t be surprised to see other clients pop up. (The client apparently is purely API-based — IE, it doesn’t have any special access to xapi that other API consumers wouldn’t have.) I’d love to see a OS-independent client pop up. ;)

In any case, I’m glad to see this, and look forward to playing with the bits!

virtuozzo on nehalem systems at softlayer

nc — Fri, 24 Apr 2009 22:55:49 +0000

As I mentioned in my previous post, I discussed how to get networking working under Debian Lenny on the new Nehalem systems at SoftLayer. However, it seems like there are lots of users who want to get Virtuozzo on RHEL/CentOS running on these boxes. A guy named Shikhir got ahold of me via a Web Hosting Talk thread, and was able to provide me with a system to build on, and some motivation to get it done for him. I was able to build Intel’s driver RPM and get it working for Virtuozzo.

So, on to the instructions!

1) Set up your system with RHEL5.3 64-bit.

2) Download the Virtuozzo installer; run through it as usual, but -do not- reboot at the end.

3) Ensure that the Virtuozzo installer installed the kernel vzkernel-2.6.18-028stab059.6 by running:

[root@nehalem ~]# rpm -q vzkernel
vzkernel-2.6.18-028stab059.6

If the kernel version is anything else, please download and install the kernel version above.

4) Download the driver RPM from:
http://www.natecarlson.com/downloads/virtuozzo-nehalem/igb-1.3.19.3-1.x86_64.rpm
..and install it with:

rpm -Uvh igb-1.3.19.3-1.x86_64.rpm

5) Reboot. If all went properly, your system will boot up with the Virtuozzo kernel and have full networking!

Hope this helps!

-Nate

xenenterprise 4.1

nc — Mon, 31 Mar 2008 13:45:42 +0000

As many of you know, I’m a big fan of XenEnterprise as a virtulization platform, as it’s based on open source software, and just makes it a lot nicer to manage. For some reason, it just seems to make a lot more sense to me than VMware ESX when I’m trying to debug a problem between the two – I think it’s just because I know the OSS Xen hypervisor very well, and can understand what’s going on behind the scenes a little bit more.

In any case, Citrix just released a new version of XenEnterprise – 4.1. There are lots of new features (like very tight Netapp integration, which is cool), but the biggest one for me has to be support for live migration on fibre channel storage, which is huge. I’ve done extensive testing of it starting with 4.1 beta 1, and am pleased to report that it works quite well in my environment. Highly recommended – if you haven’t tried this yet, go give it a shot!