|
|
|
Supreme Being
Group: Moderators
Last Login: Yesterday @ 3:37 PM
Posts: 108,
Visits: 375
|
|
| Due to the many possible virtualization platforms and configurations the actual performance of Virtual SAN Appliances (VSA) will vary. The expectation of VSAs performance is that IOPS will be defined mostly by the underlying disk architecture and bandwidth will be defined mostly by network configuration. This forum should be used for posting performance results achieved by users of VSAs to help others size and configure their systems. Its also fun to show off how well you built your particular VSAs of course. Please fill out all fields in the template. Report on any results in addition to the three in the template but post at least those listed. Having the same set of 3 very basic numbers will allow for some level of comparison. Below is an IOmeter configuration to specify those 3 specifications in detail. Just save the text as a .icf file. IOmeter can be found here... http://sourceforge.net/projects/iometer/. #######Please follow this template when posting performance results.########### Physical Server-Make: Model: CPU: Mem: Notes: Physical Storage Architecture-Disks: #: RAID: Notes: Physical Network-Adapter: #: Team Settings: Notes: VSA Cluster- SAN/iQ Version: # of nodes: Replication Level: Notes: Host running IOmeter-OS: CPU: Mem: Initiator used: Network: Notes: IOmeter 50-50 read-write 512b random IOPS, queue 32= IOmeter write 64K sequential MB/s, queue 32= IOmeter read 64K sequential MB/s, queue 32= Notes: ############################################################# Version 2006.07.27
'ACCESS SPECIFICATIONS =========================================================
'Access specification name,default assignment
512B; 50% Read; 100% random,NONE
'size,% of size,% reads,% random,delay,burst,align,reply
512,100,50,100,0,1,0,0
'Access specification name,default assignment
64K; 100% Read; 0% random,NONE
'size,% of size,% reads,% random,delay,burst,align,reply
65536,100,100,0,0,1,0,0
'Access specification name,default assignment
64K; 0% Read; 0% random,NONE
'size,% of size,% reads,% random,delay,burst,align,reply
65536,100,0,0,0,1,0,0
'END access specifications
Version 2006.07.27
Adam C
Product Manager
LeftHand Networks
|
|
|
|
|
Forum Newbie
Group: Forum Members
Last Login: Wednesday, June 04, 2008 10:14 AM
Posts: 2,
Visits: 5
|
|
Physical Server-Make: HP Proiant DL380 G5 CPU: 4x2.8 GHz Mem: 14GB
Physical Storage Architecture: 146GB SAS 10k #: 4 RAID: 5
Physical Network-Adapter: NetXtreme II 5706 #: 4 Team Settings: 2 x 2 teams
Physical Network-Adapter: NetXtreme II 5708 #: 2 Team Settings: 1 x 2 teams
VSA Cluster- SAN/iQ Version: 7.0.01.6046.0 # of nodes: 3 Replication Level: 2-Way
Host running IOmeter-OS: Win2k3 R2 SP2 CPU: Xeon E5440 2.83GHz Mem: 4GB
Initiator used: Microsoft
VMware ESX: 3.5.0, 82663
IOmeter 512B; 50% Read; 100% random = 405.50 IO/s, 00.20 MB/s, 2.47ms Ave IO, 345.05ms Max IO, 3.37% CPU
IOmeter 64K; 100% Read; 0% random = 225.54 IO/s, 14.00 MB/s, 4.45ms Ave IO, 478.55ms Max IO, 2.26% CPU
IOmeter 64K 0% Read; 0% random = 187.32 IO/s, 11.70 MB/s, 5.35ms Ave IO, 423.84ms Max IO, 1.10% CPU[/font][/font]
|
|
|
|
|
Forum Newbie
Group: Forum Members
Last Login: Wednesday, June 04, 2008 10:14 AM
Posts: 2,
Visits: 5
|
|
| These results seem very slow compared to the same hardware running Windows natively using local disks. The transfer rate is well over 500MB/s with the same Iometer settings!
|
|
|
|
|
Supreme Being
Group: Moderators
Last Login: Yesterday @ 3:37 PM
Posts: 108,
Visits: 375
|
|
That is really slow. But don't compare it to the same server just running windows accessing local disk. Totally different beast.
Given the configuration you describe (and assuming you have battery backed cache enabled on there) you should be getting about 600 IOPS, ~70 MB/s write, ~100 MB/s read.
Judging by your ms response times though you are using a queue depth of 1? Is that right?
Change your outstanding IOs to more like 16 and let me know what you get.
If you then get close to the 600 IOPS, ~70MB/s, ~100MB/s then you are where you should be for VSAs in a cluster.
Guessing you'll ask why...
600 IOPS is based on how many disks you have, in what raid, and what replication level. You'll not get that higher unless you increase the number of disks, their rotation speed, or RAID configuration.
~ 100MB/s read, about the network limit, the disks, and the VSA itself, might be capable of going faster but the network will stand in your way.
~ 70MB/s write, network limit too with some overhead for the replication of data to other nodes.
The network bonds on ESX servers do a poor job, if any, of load balancing. Unless you are a ESX network ninja (you might be), you basically have active passive bonds on those. Doing "IP hash" load balancing will help the read workload a bit, ~140 MB/s in my lab.
Frankly... if you are really looking for that 500MB/s the VSA is not the right solution.
I'll caveat that by saying some are playing with 10GB and looking to make VSA do more than was originally intended. I'd call that a science project at the moment though.
Hope that helps
Adam C
Product Manager
LeftHand Networks
|
|
|
|
|
Forum Newbie
Group: Forum Members
Last Login: Friday, August 01, 2008 2:34 AM
Posts: 6,
Visits: 28
|
|
Hi further to this I reconfigured the disk array in the nodes to a pair of mirrors. ESX is install on one mirror, the VMFS is on the second.
Physical Server-Make: HP Proiant DL380 G5 CPU: 4x2.8 GHz Mem: 14GB
Physical Storage Architecture: 146GB SAS 10k #: 4 RAID: 1 x 2
Physical Network-Adapter: NetXtreme II 5706 #: 4 Team Settings: 2 x 2 teams
Physical Network-Adapter: NetXtreme II 5708 #: 2 Team Settings: 1 x 2 teams
VSA Cluster- SAN/iQ Version: 7.0.01.6046.0 # of nodes: 3 Replication Level: 2-Way
Host running IOmeter-OS: Win2k3 R2 SP2 CPU: Xeon E5440 2.83GHz Mem: 4GB
Initiator used: Microsoft
VMware ESX: 3.5.0, 82663
IOmeter 512B; 50% Read; 100% random = 6220 IO/s, 3 MB/s, 1.62ms Ave IO, 149.89.05ms Max IO, 18.93% CPU
IOmeter 64K; 100% Read; 0% random = 1434 IO/s, 90.00 MB/s, 6.9ms Ave IO, 317ms Max IO, 11.52% CPU
IOmeter 64K 0% Read; 0% random = 1179.32 IO/s, 73.66 MB/s, 8.20ms Ave IO, 252.46ms Max IO, 8.17% CPU
Although these results are better I'm still have issue with these speeds :
a) Test Rig
We have a demo ESX system we use for presentations. It's based on desktop Intel PC's. Each ESX has 4Gb RAM, a quad core Core 2 Duo CPU with a 160Gb SATA ( ESX ) and a 250Gb SATA ( VMFS ) -both 7.2k's on the onboard controller. Currently in the same config as the proper hardware above ( except 2 ESX nodes ) the results on the 64K test is as follows :
IOmeter 64K; 100% Read; 0% random = 1191 IO/s, 74.00 MB/s, 8.9ms Ave IO, 252ms Max IO, 8.78% CPU
This isn't massively less than the above tests on much more inferior hardware.
b) Machine responsiveness
Machines installed on the above demo rig boot quicker and generally feel more responsive. The hosts installed on the production hardware act as if you don't have the VM Tools installed and have the video acceleration set to off ( they have full acceleration and the tools on )
c) Disks
What are your recommended configurations for the disks ? We put 4 147Gb in each server and set it up with RAID5 giving 441Gb available to ESX + VSA. However sharing the disk between ESX and the VSA seems to result in the appalling performance reported by Mark Hodge - why would this be the case ?
Cheers,
Anthony
|
|
|
|
|
Forum Newbie
Group: Forum Members
Last Login: Friday, August 01, 2008 2:34 AM
Posts: 6,
Visits: 28
|
|
Additional note : I upped the block size to 8M on the VMFS partition and the results are now what I was expecting ~133MB/sec @ 2100IOPS.
However I would have expected to get results close to this with four disks in a RAID5. It is looking like I'll have to have a seperate system drive / VMFS which means more disks, and alot of wasted space on the ESX drives. Any recommendations ?
|
|
|
|
|
Supreme Being
Group: Moderators
Last Login: Yesterday @ 3:37 PM
Posts: 108,
Visits: 375
|
|
When using ESX I usually do one big RAID 5 set because the usable capacity is more important than performance for most VSA cases.
One option for you might be to use ESXi running on some flash device so you can devote all HDD to VSA.
Adam C
Product Manager
LeftHand Networks
|
|
|
|
|
Forum Newbie
Group: Forum Members
Last Login: Tuesday, December 02, 2008 10:46 AM
Posts: 2,
Visits: 18
|
|
Physical Server-Make: Sun Model: SUNFIRE4250 CPU: 2 x Xeon E5450 @3Ghz Mem: 32Gb Notes: running ESX 3.5 U2
Physical Storage Architecture-Disks: SAS 146Gb @10k 2.5" #: 16 RAID: 5 Notes:
Physical Network-Adapter: Intel PRO/1000PT #:2 Team Settings: IP Hash Notes: connected to HP GB switch
VSA Cluster- SAN/iQ Version: 7.0.01.6046.0 # of nodes: 2 Replication Level: 2 Notes:
Host running IOmeter-OS: W2k3 x64 CPU:1vcpu Mem: 8 Initiator used: ESX Software Network: 1 vnic Notes: host running on same ESX server as VSA i/o meter pointing at VM disk hosted on VSA
IOmeter 50-50 read-write 512b random IOPS, queue 1 = 602 IOPS,300 Read IOPS, 302 WriteIOPS, 0.294Mbps, 1.66ms Ave IO, 394.8ms Max IO, 5.23 % CPU
IOmeter 50-50 read-write 512b random IOPS, queue 16= 1060 IOPS,530 Read IOPS, 530 WriteIOPS, 0.52Mbps, 15.09ms Ave IO, 438.8ms Max IO, 5.99 % CPU
IOmeter 50-50 read-write 512b random IOPS, queue 32= 1003 IOPS,500 Read IOPS, 503 WriteIOPS, 0.24Mbps, 31.88ms Ave IO, 455.38ms Max IO, 6.48 % CPU
IOmeter write 64K sequential MB/s, queue 1 = 332 IOPS, 20.78Mbps, 3.007ms Ave IO, 327.8ms Max IO, 4.384% CPU
IOmeter write 64K sequential MB/s, queue 16= 495 IOPS, 30.94Mbps, 32ms Ave IO, 226ms Max IO, 3.27% CPU
IOmeter write 64K sequential MB/s, queue 32= 218 IOPS, 13.6Mbps, 143ms Ave IO, 698ms Max IO, 9.09% CPU
IOmeter read 64K sequential MB/s, queue 1 = 501.7 IOPS, 31.36Mbps, 1.993ms Ave IO, 274.28ms Max IO, 5.263% CPU
IOmeter read 64K sequential MB/s, queue 16= 1644 IOPS, 102.73Mbps, 9.66ms Ave IO, 397.46ms Max IO, 20.53% CPU
IOmeter read 64K sequential MB/s, queue 32= 1602 IOPS, 100.15Mbps, 19.51ms Ave IO, 528.37ms Max IO, 9.66% CPU
Notes: Host running IOmeter is only VM on ESX host apart from the VSA node, Other VSA node is in use but not busy, VSA CPU utilisation peaked at 2.8Ghz according to ESX console,
max disk set to 1000k, IOmeter run for 2 minutes
Adam, these look slow | | | |