Initial write of 500mb on 5 threads (total of 5 500mb files)
Initial Write: 1,292.01 MB/s
Re-Write: 1,308.76 MB/s
Read: 3,560.75 MB/s
Re-Read: 3,655.25 MB/s
Reverse Read: 3,136.23 MB/s
Stride Read: 2,804.91 MB/s
Random Read: 2,499.73 MB/s
Mixed workload: 1,981.39 MB/s
Random Write: 369.06 MB/s
Fwrite: 1,248.06
Fread: 3,365.72

We used the following parts in our build:
1x Intel S5520HC Server board with two 5620 CPUs and 24GB of memory
1x Supermicro SC847A-1400LPB Storage Chassis
4x Adaptec 5805Z RAID Controllers with ZMM super caps
1x Intel Quad Port ET2 Gigabit NIC
36x Western Digital RE4 500GB drives (Storage drives)
2x Hitachi 2.5″ SATA HDDs (for running our OS in a mirrored array)
8x Tripplite SFF-8087 to SFF-8087 cables (We didn’t have enough controller ports to connect all 9 backplane connectors to RAID controllers)
OS: Sun Solaris 11 Express

One thing we attempted (and failed at) was to see if the 4 additional onboard (motherboard) SATA ports would detect and run the last 4 drives on the rear backplane. We only have 5 PCIe slots on our motherboard and we needed one for the NIC. Since the only low profile SAS/SATA cards I could find only had a max of 2 SFF-8087 ports on them we knew we would only be able to use a max of 32 drives via RAID controllers. I was hoping the onboard SATA ports (via a reverse breakout cable) would run the final 4 drives for a pair of ZIL drives and a pair of L2ARC drives so we tried initially to get 4 standard SATA WD RE4 drives to work but the onboard controllers wouldn’t detect any of the drives in that backplane. Not a huge deal as those 4 drives aren’t going to break the bank on capacity (we ended up with 12 TB). Adaptec said they are coming out with a low profile card that has up to 6 ports on it and when they finally do I’ll switch to that so I can use all the drive bays. As of right now I’m not seeing a need for a ZIL or L2ARC drive setup unless I’m missing something.

http://nexentastor.org/boards/1/topics/4400

I will use a similar case with ASUS P8B-C/SAS/4L motherboard.
It have 8 SAS 6Gb/s, 2 SATA 6Gb/s, 4 SATA 3Gb/s ports, but only SATA connectors (total 14 SATA connectors!).
Can this motherboard be used with “A” case? I know – exist SFF8087 to 4xSATA cables, but I’m not sure – can this cable to be used here?

It’s unfortunate that there is such a price-premium between SATA & SAS drives, especially in the “post-flood” market.

We’re looking at getting about 5 servers and have had to totally rethink our plans with the cost of 3TB SAS drives going from 11.6c/GB to 30+c/GB.

In the end we’ve decided to buy a server with a crapload of Seagate GoFlex 3TB drives (and rip out the Barracuda XT from inside) and make do with that, initially, until there is some more price stability. We’ll relegate that to “ZFS Backup” utility when we can afford to get the SAS drives for our main storage server pair (now delayed in purchasing).

I have not used them with interposers, but I’ve heard similar things for any sata device on an interposer.. that it just causes problems in the long run. ;( I can’t really comment specifically though..

Differences in our real build from the above:
* We went with 48G of memory instead of 24G.
* We went with 1tb/2tb 7200rpm SAS2 drives instead of SATA.
* We went with 2x 300gb 15K SAS2 drives for the boot disks instead of SATA.
* We went with 2x 128GB RealSSD C300′s for the L2ARC (read cache), and 6x 32GB Intel SLC SSDs in three mirrors for the ZIL.
* We added 2x Intel quad-port gigabit ethernet cards to the build, for a total of 10 GigE ports.

For networking, we are terminating these into our Cisco 6509 switches.. they have a SUP720-10G, but have limited 10GB ports – so we went dual port channels of 4-ports each.

I should note that if I were doing this today, I wouldn’t go with the Intel SSDs anymore – find a SLC SSD with a 6gbit interface. The Micron 50gb P300 drives *rock*, and are somewhere around $600/ea.

I will definitely take your suggestions into consideration and make some sort of blog post of my own to document my build assuming i go this route.

are you still using the onboard 1gb NIC’s to connect to your switch? are you using LACP? what kind of switch did you end up going with?

im assuming we will be adding a dual 10gbe adapter to our build (its to house datastores for ESX) but curious to know what others did.

Things I would have done differently..

• As I’ve mentioned previously, look into the multipath SAS2 expanders that are available now; do some bandwidth calculations – if the bandwidth will be enough, go that route instead. Less cabling, and dual paths to each SAS2 disk. Note that you’d also either need to get SAS SSDs ($$), or find interposers that will work in the drive sleds.
• Possibly consider going with the model that supports full-height cards, and use the new LSI-9201adapters, with 4 SAS2 ports instead of the 2 SAS2 ports on the 9211′s.
• Make sure *not* to enable dedupe at any layer – right now, it seems to cause more problems than the space that it saves is worth. Nexenta’s working hard to fix this, and I have not yet tried it again on 3.1.1.. but in 3.0.5 it caused us no ends of stability issues. Just to be clear – this is not a Nexenta-specific thing; from what I’ve heard dedupe is still a risky proposition on the Unified Storage nodes from Sun. Oh — if you are only running a single node with no replication, and have plenty of memory, you can probably enable dedupe; just don’t use it if you use replication.
• Consider going with 15k SAS disks instead of the 7200rpm SAS disks. We haven’t had any issues with general performance with the 7200rpm drives, but when a drive fails, a resilver takes *forever*. If you don’t need to huge amount of space that you can get with 7200rpm drives, please, go 15k. ;)

Really, that’s about it. Quite happy with the solution overall.. just took a little bit to figure everything out, and to realize that dedupe should be *off*! Also – we did have some problems with support at Nexenta in the early days, but we were always able to escalate and get the issues addressed. However, this appears to have been growing pains.. things are far better now, including a real support portal where all registered geeks in the org can view and update cases together, etc.

Any thoughts on this build a year-ish later? How is the reliability/stability? How is the supermicro stuff holding up? anything you would have done different? (i am considering something similar and interested in how yours ended up panning out)

Appreciate any thoughts.

Well, SAS expanders + SATA disks can give problems. In that case SATA cables might be better, maybe?
http://gdamore.blogspot.com/2010/12/update-on-sata-expanders.html

Thanks for an interesting post. I’ve also looked at this as a ZFS storage option but I was put off the E-series chassis after seeing reports of problems with SAS expanders and SATA disks. See here for example:
http://gdamore.blogspot.com/2010/12/update-on-sata-expanders.html
Have you had any related problems?

I was also put off going to the 6Gbps SAS2008 because it reports the disk WWN, resulting in the very long ‘t’ numbers. I presume the main nuisance is being able to map disk WWN to chassis slot number, and also when you replace a failed disk do you have to find the different ID of the replacement disk to be able to run zfs replace?
I was therefore considering using the older SAS3081E-R cards, but you’ve mentioned a bug with them. unfortunately bugs.opensolaris.org now seems to have died (thanks Oracle!). Do you know any other link for details of this bug and its status?

Cheers
Zaeem

Thanks

I’ve got a HP DL120G6 with dual core idle in the server room.
I’ve also got some 2GB harddisks and a few x25M’s laying around.
So basically I’ve got the ingredients to give this a try.
The goal would be to create NFS server for backing up nightly VMWare snapshots.

my questions:
Aren’t you worried about wearing out you SSD cache drive?
Does solaris/nexenta support trim?
Does the dedup/compression feature require a lot of cpu power and memory?
Where do I install the O.S. Do I need a separate drive for this or can this also be on the zfs raid?
I solaris any good for NFS when comparing to linux?

What is community experience with scrubbing / resilvering of such data volumes? We have 11-drives RAIDZ2 vdevs, and a few of them in one volume. Drives are 1-2TB 7200kRPM SAS drives. And we experience huge performance degradation during these operations. We are running snv_111b release.