|
|
|
Forum Newbie
Group: Forum Members
Last Login: Monday, November 17, 2008 11:08 PM
Posts: 5,
Visits: 13
|
|
I'm testing a 2x VSA configuration on the same ESX 3.5 server. Box is a dell 2950 with a perc5. I'm testing speeds in a single VM (the test VM + 2 vsa's are the only things running) - I have the following lun's in the (Linux) VM:
1. Virtual disk on VMFS3 filesystem on local disk (the backing store for the vsa's)
2. Virtual disk on VMFS3 on iscsi lun from vsa's
3. Raw iscsi lun from VSA's
performance testing: sequential read/write of 3GB file, VM has 1GB ram.
sequential write to Local Disk array - 106MB/s
sequential read from Local disk array - 127MB/s
sequential write to VMFS3 over iscsi: 76MB/s
sequential read from VMFS3 over iscsi: 37MB/s
sequential write to raw iscsi: 79MB/s
sequential read from raw iscsi: 38.5MB/s
I notice that vmware says the version of tools on my VSA is out of date, is it possible the poor performance is related to a wonky NIC driver in the VSA images?
During all the testing none of the VSA cpu's get much over 50% utilization, so I'm rather confused as to what the bottleneck might be, other than needing a vmware tools update? the disparately in the read vs. write numbers are particularly confusing.
|
|
|
|
|
Forum Newbie
Group: Forum Members
Last Login: Monday, November 17, 2008 11:08 PM
Posts: 5,
Visits: 13
|
|
| also - I noticed when the vsa's were initializing, each one of them was doing 50-65MB/s sustained writes to the array concurrently, so it seems like they can achieve disk performance fine, it seems the bottleneck is just in the iscsi layer
|
|
|
|
|
Supreme Being
Group: Moderators
Last Login: Yesterday @ 2:22 PM
Posts: 101,
Visits: 354
|
|
From the high view presented it looks to me like you are getting what we would expect although your numbers seem reversed.
Something else might be fishy because we expect read rates to be higher than write rates, and your numbers are the opposite of that.
First, the VMware tools out of date does not matter except to annoy you, the VSA itself was released just before ESX 3.5 was. SAN/iQ 7 SP1 will update them to 3.5 level so the message says "ok" instead, I expect no functional or performance differences though.
The performance of VSA are expected to be lower than direct to the physical drives for 2 reasons.
1-Assuming you selected the default configuration in SAN/iQ the data is replicated twice (mirrored) for redundancy across clustered systems. So twice the writes are occurring.
2-The data has to travel through VMkernel networking and virtual networking which is not exactly amazing at bonding (sorry VMware).
iSCSI itself is not so much a culprit as just the network itself.
In general sequential throughput is not a strong suite of the VSA on typical GIGe virtual networks. It is also not usually an interesting benchmark for ESX servers. Random IO is much more important to VMs. Usually random IO on VSA are very close or the same as the physical disks.
As general guidance, if you are getting 60-70 MB/s write and 90-110 MB/s read out of a VSA cluster then you are exactly where you should be.
IOPS are a function of how many disks are in the configuration, but as very very gross estimation about 100 IOPS per disk would be right.
Adam C
Product Manager
LeftHand Networks
|
|
|
|
|
Forum Newbie
Group: Forum Members
Last Login: Monday, November 17, 2008 11:08 PM
Posts: 5,
Visits: 13
|
|
When you say vmware network isn't good at 'bonding', what bonding are you referring to? With the vmkernel aspect removed, I can get 90MB/s network read and writes from the VM, so are you just saying vmkernel is a slow network?
I too am baffled by the read performance, I fully expected the numbers to be reversed as well, but they're not.
The reason I/m benchmarking sequential reads and writes, is that my application that would be on it does sequential reads and writes. I've tested a non-replicated LUN, and the performance is right about the same , 40MB/s reads and 85MB/s writes. still fairly horrible compared to the vmdk access direct to the array.
I'm happy to run something more iop intensive and less throughput oriented to see if there's a problem going on, or if saniq is really just horrible at sequential performance. iozone? bonnie++? What would you recommend/consider valid result so as not to be dismissive about it?
Also, I played with the VSA vmx files, and it looks like the vmware tools included works fine when vmware gives it the enhanced vmx driver, jumbo frames don't work (while I understand it wont improve vmkernel performance, it would be great to have if we can get the performance issues resolved and we use it with a jumbo-enabled initiator).
|
|
|
|
|
Forum Newbie
Group: Forum Members
Last Login: Monday, November 17, 2008 11:08 PM
Posts: 5,
Visits: 13
|
|
Actually, the more I ponder this, the more it makes a little sense, there's write back enabled on the raid controller, so that might help give the write performance a boost - but since these are sharing a drive (and depending how internally saniq orders things on the drives), these really are neither sequential reads nor writes, as each vsa is reading/writing to a different part of the array, it's just multiplying the amount of seeks. I'm going to try this with a second esx/saniq module and see what happens.
|
|
|
|
|
Supreme Being
Group: Moderators
Last Login: Yesterday @ 2:22 PM
Posts: 101,
Visits: 354
|
|
Do you mean you are running two VSA on the same ESX server up to this point?
That should only be done if you are just playing around or are a SAN/iQ ninja and know how to order VSA correctly in campus or multi site architectures.
Adam C
Product Manager
LeftHand Networks
|
|
|
|
|
Forum Newbie
Group: Forum Members
Last Login: Monday, November 17, 2008 11:08 PM
Posts: 5,
Visits: 13
|
|
I was just playing around..I've now got this going on 2x VSA's, and I'm still seeing a read performance bottleneck at 50MB/s.
This is both through esx/vmkernel, iscsi initiator inside the vm and third party non-virtualized iscsi initiators.
each VSA is now on on a 2950 with 6x15k sas raid10. Testing direct access from a vm to either array gets ~ 130MB/s read speeds.
I've also tried parallel reads from multiple sources at once, the performance ends up being 50MB/s divided by the # of initiators.
Also, the vsa's are the only things running on each esx server (other than the test vm).
What performance would you expect to see from saniq on a pair of 2950 with 12x 15k sas drives? Why is the VSA so much worse?
I'd really like to move forward this, conceptually I really like it, but being capped at 50MB/s across all vm's using it makes it a non-starter
|
|
|
|
|
Supreme Being
Group: Moderators
Last Login: Yesterday @ 2:22 PM
Posts: 101,
Visits: 354
|
|
50 is a strange number to be capped at, and that is adds up to that even if you do more than one thread / volume too sounds like some networking bottle neck.
I'd expect 90 - 100 MB/s read doing what you describe.
Are you running IOmeter? What queue depth are you using?
Adam C
Product Manager
LeftHand Networks
|
|
|
|