Category Archives: server

WD Red Drives – Inconsistent Performer

I have two servers, each with WD Red 2.5 1tb hard drives. The servers run 4 drives in RAID 10 configuration.

I had previously benchmarked them with ATTO’s Disk Benchmark software on two different occasions. I thought it was strange to get different results. Today I was doing another benchmark and again got a 3rd set of results.

Here are the dates and different results I have received.

August 29, 2014

Maximum Read: 162MB/Sec
Maximum Write: 135MB/Sec

raid-10

April 7, 2015

Maximum Read: 174MB/Sec
Maximum Write: 58MB/Sec

RAID 10 - Slow

September 5, 2015

Maximum Read: 143MB/Sec
Maximum Write: 70MB/Sec

raid10-bench

Why so much variance in the results? Really strange…

I suspect the results from April 7, 2015 are probably ‘off’ as the write speed sorta flat lining around 55MB/Sec seems too consistent.

If you just go with the average (maximums) from the other two you get:

Maximum Read: 152MB/Sec
Maximum Write: 102MB/Sec

Windows fails to see Autounattend.xml from ISO

I’m working on setting up a bunch of servers, so I used the Windows System Image Manager to create an image with answers to all the questions of a basic install.

I used VMWare to boot the ISO image I was creating to test.

Placing the autounattended.xml on the root of the DVD ISO gets the image in VMWare fully setup.

Try it out on the actual server and nothing works. I am prompted just like a normal install. It is not reading the autounattended.xml file at all.

Turns out the autounattended.xml file will only work if the BIOS considers the device removable, like a USB thumb drive or something. The BIOS I am using is mounting the ISO as a fixed disk so the Windows installer will not use the autounatteended.xml that lives in the root of the ISO.

Good news, there is a solution to this problem of the Autounattended.xml not being picked up by the installer. The process inegrates the Autounattended.xml into the boot.wim file so it is always picked up.

You need the following items beforehand:

  • WAIK
  • The USB or ISO disk you plan to make your install disk

Then do these steps:

Mount the boot.wim image located on your USB HDD disk (in this example H: is the USB disk) using ImageX from WAIK

imagex /mountrw H:\Sources\boot.wim 2 C:\temp

(assumes you have a folder “C:\Temp”)

(the number 2 stands for Index 2 within the boot.wim image)

Fire up your Windows Explorer and navigate to C:\Temp. You will see your boot.wim image mounted and all. Drop your “autounattend.xml” file you created directly into this folder (right next the the Setup.exe file)

Close Windows Explorer and unmount the image:

imagex /unmount /commit c:\temp

Benchmark Amazon

Amazon AWS LogoAmazon gave me $100 gift card for AWS services a couple months back. As much as I love the concept of cloud the numbers have never worked out for me. My gift card is going to expire soon, so lets burn up some credits by bench-marking some of their AWS EC2 instances.

I’m most interested in CPU and disk performance of various instances since servers do not need high end graphics. All tests were performed on Windows 2012 R2 Standard Edition (64 bit) using PerformanceTest 8.0 software.

The Amazon pricing per hour is for the N. Virginia area, which I believe is cheapest zone they offer.

Instance Type:

c4.large – 8 ECU – 2 vCPU – 3.75gb RAM – $0.193 per Hour

CPU Mark: 2,708
Disk Mark: 787 (EBS Storage)

c4.2xlarge – 31 ECU – 8 vCPU – 15gb RAM – $0.773 per Hour

CPU Mark: 9,485
Disk Mark: 1,017 (EBS Storage)

c4.4xlarge – 30 ECU – 16 vCPU – 30gb RAM – $1.546 per Hour

CPU Mark: 15,680
Disk Mark: 998 (EBS Storage)

c3.xlarge – 14 ECU – 4 vCPU – 7.5gb RAM – 0.376 per Hour

CPU Mark: oops… have to redo this one
Disk Mark: 910 (2 x 40 SSD)

Non Amazon Systems:

Supermicro – XEON L5420x2 – 16gb RAM

CPU Mark: 4,445
Disk Mark: 1,385 (Samsung 840 EVO 120gb)

Crunching The Numbers

Xeon L5420I want to compare prices of running in the cloud to the Dual Xeon L5420 processors, which are available for very cheap on eBay. Perfectly good used servers, slap some new SSD into them stick them in a datacenter and run them until they die.

The closest match offered by Amazon is the c4.2xlarge class machine, which has a CPU mark of 9,485 vs the dual Xeon’s score of 6,734.

The cost to run in the Amazon cloud would cost you $556.56 per month. That is just the machine, it does not include extras such as a load balancer, VPN or bandwidth.

The cost to run a 1/4 rack (10 machines) would be $5,556.60 per month. If you need and entire rack it would cost you $23,375.50 per month.

You can get your cost down quite a bit if you are willing to commit to a long term agreement of 1, 2 or 3 years with Amazon. Once you commit to a specific instance you can’t change, so calculate your usage requirements before committing.

Another cheap rack of compute power

I think it was about 1 year ago that I setup a rack at a datacenter, filled with servers that had come off lease. You can get them cheap, real cheap vs. a new server.

I read the other day that Intel has not made any ‘major’ improvements to their processors since 2011. Sure there have been some improvements to SATA, SSD’s etc. But when you can buy a server for 10% – 20% of the price of purchasing new, new just does not seem worthwhile.

Last year we used 1U twin Supermicro servers using the the X7DWT-INF motherboard. They came equipped with 2x Intel Xeon L5420 quad core processors and 16GB ram. I looked up the price paid last year, they were $450.

They work fine, way more ram than we need. The only downside is the IPMI management is not always the greatest but we have managed. We even bought extra servers that just sit in the rack, to be used as parts in the event of a failure of any of the old servers. So far the parts machines are just sitting there, no issues with parts.

Now 2015, we want to build another rack – at another datacenter (additional redundancy). Would like to find computers with X8 based motherboards as the IPMI is supposed to be better.

Unfortunately they are still too costly, so we are looking at the exact same model of server that we bought last year. The good news is the price has dropped for $450 per 1U down to $250. Imagine, there are two full servers in 1U for $250. That is really $125 per server, since there are two per 1U. Simply blows my mind, since a new machine would cost you $2,000+ for a server and you don’t get anywhere near the price/performance boost.

Say we put 45 1U units in a rack (that is 90 servers) for a cost without hard drives of $11,250. If we could find new servers, twin models for $2000 (without hard drives) the cost would be $90,000. I doubt you could find servers for $2000 new.

There are no hard drives included with the servers, so SSD will be purchased for most.

A couple servers will be used as database servers, last year we used WD Red drives and attempted to implement read caching using Fusion-IO cards. The caching concept did not work very well, the performance improvement seen was not worth the effort.

Seagate Laptop SSHDSo this year, rather than WD Red (currently $69.99) we are going to try using Seagate Laptop SSHD (currently $79.68).

Now according to benchmarks over at UserBenchmark these 2.5 inch drives do not perform well vs. 3.5 inch drives or SSD (of course). However, if you benchmark them WD Red 2.5 vs Seagate Laptop SSHD they actually perform 58% better overall than WD Red and 161% better on 4K random writes.

Since the workload on the database servers are 90% write, we are going to give these laptop drives a chance.

We still have a Fusion-IO card sitting here unused as well from last year. So we can stick that in one of the DB servers to increase the read side of things. Would not go out and buy one just for this purpose but since it is just sitting here on the shelf, might as well put it in.

Boosting Old Technology – Can Caching Help?

I have written a few times about some older Supermicro servers I am working with recently.

Since the setup I built has 4 x 1tb hard drives (spinning kind), and no space for an SSD to use as cache I bought a used 160gb Fusion-IO ioDrive on eBay.

I had been running Debian on the server. The drivers/software for the Fusion-IO under Linux were a bit restrictive, requiring a specific distro and kernel. I could not find any free/cheap caching software to use under Linux that actually worked. I found some open source stuff, it just did not work.

The Hardware/Software Setup

Why the switch from Debian to Windows?

  1. Fusion-IO setup under Windows is much easier
  2. Availability of Intel’s caching software
  3. Being able to easily run any version of mySQL

The server getting the upgrade powers a database – nothing else. The hard drives use the onboard Intel RAID controller, operating in RAID 10, which based on my previous testing gave the best write performance while providing redundancy.

The workload is 90% write, with no users sitting waiting for writes to complete. Write performance still needs to be quick to keep up with the transaction volume. Users do not want slow reads either – of course.

Ideally I would have a Fusion ioDrive big enough to hold all the database files, but that is not financially possible.

Currently the live data on the database is about 110+ gigs. Technically that would fit on the 160gb Fusion ioDrive. As the database grows, there could be issues.

Intel LogoThe Intel Cache Acceleration Software provides read caching only, it operates in write-through mode to populate the cache – write-back is not available in the software but I would probably not use it anyway due to the increased risk of data loss in the event of a crash or power failure.

Lets find out what a PCI based SSD can do with some caching of those spinning disks. All tests were done on the server, with no users connected. The database was restored for each test, automatically populating the cache if caching enabled.

110+ Gig Database Restore

  • 2 hours, 57 minutes – Restore to RAID 10 (no caching)
  • 3 hours, 10 minutes – Restore to RAID 10 (with caching)
  • 2 hours, 7 minutes – Restore to Fusion-IO storage (no caching)

Sample Reads

I conducted a number of read tests, accessing various databases running reports and user queries. The same queries were run for a tests after a fresh database restore. Some queries make full use of indexes, some do not.

  • 22.54 seconds – Read from RAID 10 (no caching)
  • 22.62 seconds – Read from RAID 10 (with caching)
  • 21.57 seconds – Read from Fusion-IO storage (no caching enabled)

Surprised by those numbers?!? I sure am, underwhelming to say the least. Time to do some research into why the performance is so bad when it should be no much better, both when using cache and when direct from the Fusion-IO card.

Testing The Disks

RAID 10

A previous benchmark on this exact same hardware, but with a different version of Windows showed a maximum write speed of 138 MB/sec. Now running the same benchmark software, on the same hardware but a different version of Windows it is maxing out at only 58 MB/sec. Things get more and more strange today.
RAID 10 - Slow

Fusion-IO ioDrive

I have no previous benchmark for this one, but the current graph shows speeds that I would expect from this PCI memory card. An impressive maximum write speed of 811 MB/sec. and a read speed of 847 MB/sec.
fusion-io-benchmark

With a write speed increase of 1360% over the spinning disk, why is the mySQL restore only 39% faster when restoring to the Fusion-IO disk?

MySQL replication woes – error 1236

I have been using MySQL replication for years, specifically multi-master where you can do both read and write to either server.

mySQL_Dock_Icon_by_Presto_XTo make my situation even worse, I do it across the global Internet. The added latency and/or outages between sites can cause ‘issues’ that you might not see if your servers were connected locally with high speed Ethernet.

This weekend one of my servers lost a hard drive. One of the NEW WD red hard drives, I just wrote about a few weeks ago. The drives have only been in production for less than two months and one has failed already.

Using Linux software RAID, the data is OK and the machine is still humming along while we wait for a replacement drive to be installed.

One thing that did not survive the hard drive crash is the MySQL replication. The other server (the one where no hard disk crashed) actually started showing this error after the replication stopped:

Got fatal error 1236 from master when reading data from binary log: ‘binlog truncated in the middle of event; consider out of disk space on master; the first event ‘mysql-bin.006259’ at 2608901, the last event read from ‘/var/log/mysql/mysql-bin.006259’ at 2608901, the last byte read from ‘/var/log/mysql/mysql-bin.006259′ at 2609152.’

They suggest out of disk in the message, but not the case here. The problem was probably from the server being restarted without being properly shut down (a big NO NO).

So if you get this error, or any other replication error that references a binlog and a location how can you find out what the problem is?

If you have been running MySQL replication for any length of time, you have seen times where the replication stops and you need to know why.

To view the transaction that is killing your server, head over to the master server and use the mysqlbinlog utility to view what is going on.

In my case I was greeted with this message.

# Warning: this binlog is either in use or was not closed properly.

Essentially it is saying the file is messed up, with no valid content in that file the replication is stuck. To get it started again you will need to update your slave(s) with a new instruction, telling them to move to the next binglog. I advanced to the next binlog like this:

STOP SLAVE;
CHANGE MASTER TO MASTER_LOG_FILE = 'mysql-bin.006260';
CHANGE MASTER TO MASTER_LOG_POS = 0;
START SLAVE;

With that the slave can start up again, reading from the next binlog which contains valid content.

Crisis diverted, all we need now is a new hard drive and the server should be a happy camper once more.

Testing NO RAID vs RAID 5 vs RAID 10

WD-Red-1TBI am setting up a new server, with four 2.5 inch hot swap 1tb WD Red disks – model WD10JFCX. Write performance is key in this setup as 90% of the workload is putting data to disk.

SSD drives would be nice, but not cost effective to get the space needed. I suspect the best option is going to be RAID 10, but lets run a few tests first.

No RAID

Testing with a single drive gives us a max. write speed of 74 MB/Sec and maximum read speed of 113 MB/Sec which is not very impressive.

Single Drive

RAID 5

Next up, a four disk RAID 5 setup – the performance is BAD! You do gain quite a bit of read performance maxing out at just under 300 MB/Sec but the write performance has taken a major hit vs. a single drive coming in at only 31 MB/Sec.

Raid 5

RAID 10

Now lets check out RAID 10, which should be the fastest configuration out of the bunch. The downside is you need at least four hard drives and you loose 50% of your capacity to redundancy.

As expected the performance is about double what we get from a single drive. Maximum write comes in at 138 MB/sec and maximum read is 178 MB/sec. The read did not increase by double, but the performance did improve for sure over a single drive.

If your workflow is mostly read the RAID 5 configuration actually performs better. I suspect the more drives you have in your RAID 5 configuration the faster it would probably get since you have more spinning heads.

raid-10

For our write intensive workload we will be going with the RAID 10, which is very slow vs. SSD but the cost of SSD would be over $1,000 for the same space. These 1tb WD Red drives we are using currently cost about $75 each. By using RAID 10 we get the fastest write performance and quite a bit of redundancy.

RAR & UNRAR on Linux

Seems linux people don’t like RAR very much, it is not easy to install from most package managers it seems. Some used to have it, then removed it because it was not free.

Then some people come up with limited free versions etc. etc. I ended up just getting it direct from the source @ rarlab.

Move to your tmp folder and download from rarlab site.

cd /tmp
wget https://www.rarlab.com/rar/rarlinux-x64-5.8.b1.tar.gz

Extract the downloaded file (extracts to rar folder).

tar -zxvf rarlinux-x64-5.8.b1.tar.gz

Move the files to your /bin folder so you can execute rar and unrar from anywhere on your system.

cp rar/rar rar/unrar /bin

That’s about it. Now you can actually use the full power of rar on your linux box.

Oh no, Microsoft says Windows 2003 is dead!

windows-server-2003Microsoft has announced July 14, 2015 as the end of support for Windows Server 2003. Since there will be no support, patches, or security updates for your servers running Windows Server 2003 you might find yourself between a rock and a hard place.

I for one still have production servers that are running Server 2003, it has proven to be a very stable operating system for some systems.

If you still have systems running 2003, you might want to start planning your upgrade path. Leaving 2003 systems in production past the end of support could leave you with your pants down.