I spent weeks going over the forums and the net for any publications and advice on how to optimize iSCSI connections and i'm about to give up. I really need some help in determining if its something i'm not configuring right or maybe its an equipment issue.
Hardware:
2x Windows 2012 Hosts with 10 Nics (same NIC configuration) in a Failover Cluster sharing a CSV LUN.
- 3x NICs Teamed for Host/Live Migration (192.168.0.x)
- 2x NICS teamed for Hyper-V Switch 1 (192.168.0.x)
- 1x NIC teamed for Hyper-V Switch 2 (192.168.10.x)
- 4x NICs for iSCSI traffic (192.168.0.x, 192.168.10.x, 192.168.20.x 192.168.30.x)
- Jumbo frames and flow control turned on all the NICs on the host. IpV6 disabled. Client for Microsoft Network, File/Printing Sharing Disabled on iSCSI NICs.
- MPIO Least Queue selected. Round Robin gives me an error message saying "The parameter is incorrect. The round robin policy attempts to evenly distribute incoming requests to all processing paths. "
Netgear ReadyNas 3200
- 4x NICs for iSCSI traffic ((192.168.0.x, 192.168.10.x, 192.168.20.x 192.168.30.x)
Network Hardware:
- Cisco 2960S managed switch - Flow control on, Spanning Tree on, Jumbo Frames at 9k - this is for the .0 subnet
- Netgear unmanaged switch - Flow control on, Jumbo Frames at 9k - this is for .10 subnet
- Netgear unmanaged switch - Flow control on, Jumbo Frames at 9k - this is for .20 subnet
- Netgear unmanaged switch - Flow control on, Jumbo Frames at 9k - this is for .30 subnet
Host Configuration (things I tried turning on and off):
- Autotuning
- RSS
- Chimney Offload
I have 8 VMs stored in the CSV. When try to load all 8 up at the same time, they bog down. Each VM loads very slowly and when they eventually come up, most of the important services did not start. I have to load them up 1 or 2 at a time. Even then the performance is nothing like if they were loading up on the Host itself (VHD stored on the host's hdd). This is what prompted me to add in more iSCSI connections to see if I can improve the VM's performance. Even with 4 iSCSI connections, I feel nothing has changed. The VMs still start up slowly and services do not load right. If I distribute the load with 4 VMs on Host 1 and 4 VMs on Host 2, the load up times do not change.
As a manual test for file copy speed, I moved the cluster resources to Host 1 and copied a VM from the CSV and onto the Host. The speed would start out around 250megs/sec and then eventually drop down to about 50/60 megs/sec. If I turn off all iSCSI connections except one, it get the same speed. I can verify from the Windows Performance Tab under Task Manager that all the NICS are distributing traffic evenly, but something is just limiting the flow. Like what I stated on top, I played around with autotuning, RSS and chimney offload and none of it makes a difference.
The VMs have been converted to VHDx and to fixed size. That did not help.
Is there something I'm not doing right? I am working with Netgear support and they are puzzled as well. The ReadyNas device should easily be able to handle it.
Please help! I pulled my hair out over this for the past two months and I'm about to give up and just ditch clustering all together and just run the VMs off the hosts themselves.
George