Matt Pson Just another dead sysadmin blog

18Aug/11Off

On using Cisco’s UCS servers as normal servers

"So you are using the whole Cisco UCS system ...and how?"

I have gotten the above question in some different flavors quite a few times lately. And the answer is: no we do not, we use them as normal, single, rackservers just as you would with a rackserver from any other vendor.

The Cisco C200 M2 comes with a decent CPU (has 2 CPU sockets) and a few slots for RAM (up to 192GB), 3 PCI Express slots (one is low-profile) and you can install up to 4 standard 3,5" SATA disks of  your choice (no more having to stick with what other vendors can supply) plus a very decent out-of-band management card as standard (compare DRAC or iLo) and all that at quite a lower price than a comparable system from say Dell. Sure, the standard included warranty and support is far from what Dell or HP offers but I'm sure that Cisco will happily supply that for an additional fee if you want. We recently saw a increase in baseprice of this system but it is still 25-50% off from the prices that Dell keeps flooding our mailboxes with (which I know is not the prices we would pay after phoning our Dell representative).

The Cisco C210 M2 is much the same basic system but still quite a bit different as it has 16 2,5" diskslots in a 2U chassis where you have to get the disks from Cisco (which can take quite some time if unlucky) and 5 PCI Express slots. Apart from those two things it seems that the C200 and C210 shares all other properties.

I have earlier reported about a newer system, the C260 monster (64 DIMM slots and 16 drives in a 2U chassis with the new Intel Xeon family processors!), but I have yet to see it in real life (and more importantly in a pricelist).

But yes, these are usable as normal rackservers without any modifications and provides good value in the low-end range of rackservers. I would recommend them to anyone just looking for a decent rackserver any day.

18Jul/11Off

Proving a point – building a SAN (part 4 – the neverending story?)

Finally after weeks and weeks of waiting all parts for my SAN project had arrived and I unpacked the last cables and installed them and they sure did work just fine. All 16 disks jumped online and agreed to my configuration. Victory!

(Well, one of the Seagate Constellation 500GB 2,5" disks failed quite early when I started to transfer some data onto the filesystem. No way I would wait another couple of weeks for Cisco to ship me a new so I went by the local PC dealer and picked up a Seagate Momentus 500GB which worked perfectly.)

I decided to spread disks evenly among the 2 physical controllers and the channels so I got:

Card 1, channel 1: 1 SAS disk (system) and 3 SATA (storage pool)
Card 1, channel 2: 1 SSD disk (zil) and 3 SATA (storage pool)
Card 2, channel 1: 1 SAS disk (system) and 3 SATA (storage pool)
Card 2, channel 2: 1 SSD disk (zil) and 3 SATA (storage pool)

Brilliant, now to install Nexentastor CE and make some zpools. I was happy to see that during my wait for the hardware to arrive a new version had been released 3.0.5 which hopefully addressed some evil bugs I had read about.

I set up my zpool with two mirrored logdevices and two raidz2 pools with 6 disks each to be safe even if more than one disk would fail later on. The end result looks like this:

config:

        NAME         STATE     READ WRITE CKSUM
        stor         ONLINE       0     0     0
          raidz2-0   ONLINE       0     0     0
            c0t1d0   ONLINE       0     0     0
            c0t2d0   ONLINE       0     0     0
            c0t3d0   ONLINE       0     0     0
            c1t10d0  ONLINE       0     0     0
            c1t5d0   ONLINE       0     0     0
            c1t6d0   ONLINE       0     0     0
          raidz2-1   ONLINE       0     0     0
            c0t4d0   ONLINE       0     0     0
            c0t8d0   ONLINE       0     0     0
            c0t6d0   ONLINE       0     0     0
            c1t7d0   ONLINE       0     0     0
            c1t8d0   ONLINE       0     0     0
            c1t9d0   ONLINE       0     0     0
        logs
          mirror-2   ONLINE       0     0     0
            c0t0d0   ONLINE       0     0     0
            c1t4d0   ONLINE       0     0     0

errors: No known data errors

 

Sure, there was alot of diskspace 'wasted' with this configuration as I ended up with approximately 3.5TB of usable space from 6TB (12x500GB) of disks. Better safe than sorry when it comes to data and in this case the data would be virtual disks for a bunch of servers and if there is one thing I hate it's sitting in the middle of the night and restoring terabytes of data in case of an emergency (it's a fact that disks never break during daytime).

Enough labtesting, everything seemed to work just fine. Down to the datacenter to rack the server so it could be put into pre-production before my summer vacation in a couple of days. Took the 4 gigabit ports made a nice LACP of them in order to make no single NFS-client be able to use up all network bandwidth.

Made a NFS share and mounted it in our VMware ESXi cluster and transfered some uncritical VMs to the new SAN. and...

"Damn! That's fast..."

The difference when using proper a proper hardware setup (read: SSD as ZIL in a zpool) was like day and night compared to our old, SSD-less, Sun 7210. Average write latency reports in VMware was in the single digits compared to the 3 digit numbers we were used to.

Edit: now a few weeks later we have done even more test and even transfered some critical system to the new SAN and it just works without a single hickup so far (knock on wood). Next up is to transfer all data off the Sun 7210 system so we can upgrade the firmware and re-install the system with the latest updates.

13Jun/11Off

Proving a point – building a SAN (part 3)

Just a short update on the SAN building activities. All items finally arrived the other day in very much Cisco-style, around 30 boxes with each item packaged individually. After some assembly it became painfully obvious that I had forgotten to order SAS-cables long enough to reach from the raidcards to the disk backplane. Bummer!

Well, after some studying of the C210 Server Maintenance Manual and the quick-installation sheet for the LSI 3081E-R raidcard I figured it should work with a standard SFF-8087 cable. I found a local dealer that could have some Promise SAS cables delivered the following day with SFF-8087 connectors in both ends and long enough to ensure proper cabling inside the chassis.

The next day I connected everything and spent quite some time making sure that the cables would not prevent proper airflow etc. etc. Problem was, when powering up the server no disks could be found. Not a single one, no SAS disks, no SATA disks and not the SSD disks either (though they have SATA interface). Troubleshooting commenced, reseating cables, checking each disk and so on but nothing worked.

Using the Promise cable connected to the onboard SATA controller should really let me see the SATA disks connected so I did try that. The result was no disks at all found and my USB keyboard didn't work either in this combination - really odd. Uh-huh. Using the original cable that came with the server, that is much too short to reach to raidcards with, to the onboard SATA controller works just fine and lets me see the SATA disks.

So, obviously, you need Cisco cables to make this work and my belief that SFF-8087 was a SAS-cable standard is now gone and shattered.

The really bad thing is that getting those Cisco cables seems to add another "few weeks" (according to our Cisco dealer) until I can have this up and running.

To be continued...

7May/11Off

Proving a point – building a SAN (part 2)

So, lets make a little update on the last post as I have now put together (and ordered) a bunch of hardware in order to build a better SAN than my last experiment. This is what I ended up with:

  • Cisco C210 M2 server with 24GB RAM ( a nice 2U server with 16 x 2,5" diskslots, starting off with one 2,4GHz quadcore Intel E5620 CPU  )
  • a Intel Quad Gigabit network card ( I plan to aggregate these ports into a 4Gb NFS port using LACP )
  • two LSI MegaRAID SAS3081E-R RAID cards ( to get 16 channels to connect disks to )
  • 2 x 146GB 10000rpm SAS disks ( mirrored for os )
  • 2 x 120GB OCZ Vertex 3 Max IOPS SSD ( mirrored for ZFS logging )
  • 12 x 500GB 7200rpm SATA disks ( to create the storage )
  • NexentaStor ( storage os based on solaris )

That would be it. It will probably be a few weeks until all parts arrive which gives me time to think about the setup.

When it comes to disklayout I'm still deciding between using 6 mirror sets of 2 disks ( effectively a raid-10 giving a total of 3TB usable disk but with what should be the best performance possible. creating the mirrors between controllers, disk 1 on controller one mirrored with disk 1 on controller two, would give quite high resilience when it comes to faulting disks - if the 'right' disks (or controller) fail it could survive 6 disk failures, in theory ) or using 11 disks in a raidz with one spare ( like raid-5, giving a total of 5TB raw disk but with less resilience to faults and the usual write penalty when using stripes/distributed parity ). It all depends on if the SSD logdisks is enough to achive good write performance, which I'm told they will.

The logdisk, the mirrored SSD, is another thing I'm pondering as they come in 120GB size and I guess that ZFS will use a fraction of that. Some tell me that partitioning each disk in 2 partition and use one mirror set of partitions for the write log and the other 2 mirror partitions for read cache is a viable solution. Wonder if that will be needed - our problems so far has not been about the read performance, just the write performance.

One person even told me that since I'm using SSD drives with MLC technology I could partition them into 4 30GB partitions each creating a 8-way mirror for the ZFS log in order to cover for cellfailures on the disk. I'm not that convinced as that will lead to 4 times more write operations and if a disk fails I still have to replace the whole disk.

Well, well, when the hardware arrives I'll have to do some testing I think. I'll write some updated when I have something to report 🙂

25Apr/11Off

Installing Solaris via CIMC – nope, not possible

Well, actually it very much possible to install Sun Oracle Solaris via the CIMC on the C200/C210 servers as long as one does not rely on the 'Virtual Media' function to mount a iso-file to install from. The installation just grinds to a halt when it tries to figure out where the CD-Rom is. From the look of it the installer finds the physical drive and tries to attach itself to that for the rest of the installation and just ignores the virtual one.

Now if I had known that before I left work as going there, to the data centre,  just to insert a CD and go back home again seems like a lot of time for very little work (erhm... no, I'm not working from home again, am I?).