I’m working on a project to virtualise all of the business-critical servers for a client of mine (and then again, who isn’t at the moment). There is already an incumbent SAN provided by Datacore SANMelody running on HP Storage Server boxes. The three main servers have Fibre Channel connections to this SAN. They all boot from local C: drives, and also have one or two drive letters each provided by LUNs on the SAN, This configuration protects the data but not the C: drives, and does not provide any failover support in the event of a hardware fault.
The new configuration is going to be two new Dell R710 servers with 48GB RAM and Dual Xeon 5550s running Windows 2008 R2 with the Hyper-V role and Failover Clustering feature. We’ve got this running in a lab at the moment with a 30-day trail version of SAN Melody running on a 64-bit capable workstation and the two R710s connected to the SAN by iSCSI.. We have learned some important lessons from this which I share below.
I have to say, the live migration is very cool. However, it’s a real test of whether you’ve got everything configured correctly. I strongly recommend ensuring that any VM you create is tested with live migration several times over a couple of days before allowing it to be used for live workloads.
Network adapter binding order
I’m going to cover the build of the cluster hosts elsewhere. This post is mainly concerned with provisioning a new guest VM using CSV for storage of configuration and pass-through disks for the OS and data.
Cluster Shared Volumes(CSV) for VM Configuration Data
Well, apart from a clash of TLAs (and with 26^3=17576 available TLSs that does seem a bit unfortunate), CSV is a clever idea. There are some preconditions, for example all nodes in the cluster should share the same system driver (usually C:), and you’re only allowed to use CSV for Hyper-V storage, but apart from that, it seems a good idea, especially in VDI rollouts. I’ve found a slightly different use for them though… I’ve been warned off using CSV by my SAN vendor, who says there are SCSI command queuing issues or some such techie nonsense with putting lots of IO on a single LUN. More to the point though, we’re planning on using pass-through disks for performance and manageability, and that leads us to a little problem. In a Virtual Desktop Infrastructure (VDI) rollout you generally need storage for the VM configuration, the VHD for the C: drive and somewhere for the snapshot data, and a CSV is the perfect place for it all. But I want my C: drives to be on pass-through disks, and that means I can’t put the VM configuration on the same LUN as the C drive as the host can’t see the filesystem on the pass-through disk. So I’m then stuck with needing somewhere to store my VM configuration data. I can’t use a single ordinary LUN as it will need to be owned by a single node and all the VMs will depend on it. And I really don’t want to have to provision a whole separate LUN for each individual VM’s configuration & snapshot data. The answer is to use a single CSV to store the configuration and snapshot data for all my VMs. This helps speed up Live Migration, and significantly reduces the number of LUNs I need to provision, without putting much IO onto the CSV.
Provisioning Storage to my Guests
In server virtualisation there are often (usually?) several volumes that need to be served up to each guest. In an iSCSI world, you can mount additional disks in the guest using the iSCSI initiator inside the guest. When the guest moves from one host to another, it can keep it’s own connections to the SAN, and they can have multi-path IO and so on.. The same is theoretically possible in the Fibre Channel world using N-port ID virtualisation (NPIV – there are 456976 FLAs for no clash there as far as I’m aware). However, I’m again specifically and categorically warned off using NPIV with my SAN – it’s not supported (yet). This means I have to present all storage to the VM host, and then pass it on to the guests. The “Thin Provisioning” feature of my SAN means I can serve up dozens of huge LUNs from a relatively small pool of storage. Only the blocks actually used get allocated. I want all my guests to get direct access to this feature, I don’t like variable-size VHDs and so it’s pass-through disks for me. The important thing about this is that the pass-through disk has to be equally visible to all cluster nodes, and that means that the usual naming convention won’t be portable.
The fun starts with the SAN – you’ve got to provision the right sort of disk. Firstly, all disks which will be involved in clustering need to have SCSI3 Persistent Reservation support. Secondly, a VM can only boot from Virtual IDE, not from virtual SCSI disk, and there’s a maximum size for the disk presented to the Virtual IDE controller which is slightly less hat the default 2047.5GB (2096640 MB) presented by SANMelody by default. Here I am fixing the size and persistent reserve:
I’ve also selected Asymmetric Logical Unit Access as this give flexibility in assigning a preference in a multi-path IO (MPIO) mirrored active-active SAN configuration. I then mapped the volume to both servers in the cluster.
You need to select Rescan Disks to see the newly provisioned LUN showing up as disk 4 in Disk Administrator. Furthermore, you need to do this on all cluster hosts
So, it’s a Basic disk, 2TB (thin provisioned from the SAN), but it won’t work as a pass-through disk in Hyper-V unless we take it offline. Once it’s offline, I’ add the disk to the cluster:
It’s a good idea to rename the clustered disk resource to something more memorable than “Cluster Disk 1”:
So, now I’m going to create a new clustered guest VM. The first thing I do is create the VM from the Services and Applications node of the cluster manager:
I put the configuration on my Clustered Shared Volume:
The other thing I need to do is not connect a VHD, because I’m not going to be using VHDs:
When I click Finish, the cluster software does its thing to make my VM highly available:
But remember – at this point it knows nothing about the pass-through disk. This is what it looks like in Failover Cluster Manager:
So next we edit the VM configuration:
We need to add a pass-through disk. Remember that only off-line disks on the VMs current host will show in the list,:
The cluster stuff will read the Hyper-V settings and work out the dependencies now. If you do things in a different order, you’ll most likely get errors in the virtual machine configuration refresh,. It will be very important to have a clean configuration refresh before you try to do anything with the VM.
At this point the Clustered VM should look like this:
Its time to start the VM and load the OS! I Connect to the VM, Capture the D: drive (which contains my Windows 2008 R2 DVD) to the virtual DVD and click start. Here you can see the pass-through disk in the installer:
Once the OS is installed there will be one or two partitions on your new LUN.
Shut down the guest and run a refresh on the storage node in the Cluster Manager. You will see the newly created partition(s) and volumes on the LUN. Note that Windows 7 editions which support BitLoccker and Windows Server 2008R2 create the 100MB partitionas well as the main partition for the C: drive:
Note that it is vitally important that the volume paths start with \\?\GLOBALROOT\. If they don’t, they won’t migrate between cluster nodes and you will become very unhappy.
Refresh the VM in Cluster Manager:
This is how it’s supposed to look:
I’d recommend a final refresh in Disk Administrator on the other cluster nodes.
Check your VM has the correct Live Migration network selected:
And that the dependencies look OK. Note that these are created by the Failover Cluster manager, not by me:
The final thing to do before try a live migration is a refresh of the VM Configuration. If you’re forgetful like me you’ll have left a captured DVD drive or ISO image mounted which isn’t cluster compatible:
The error will look like this:
Disk path ‘IDE\CDROMPLDS_DVD+-RW_DS-8A4S____________________JD51____\5&36A28B5E&0&0.0.0’ is not a path to storage in the cluster or to storage that can be added to the cluster. You must ensure this storage is available to every node in the cluster to make this virtual machine highly available.
Don’t panic, just remove the captured DVD from the VM. It’ll automatically do a refresh, but you can do another one to convince yourself it’s a clean configuration then we’re good to go with our 1st live migration attempt.
And it works.
Provision volumes from the SAN with SCSI3 Persistent Reservation and (if available) ALUA support.
- Bootable LUNs must be smaller that 2TB. Second and subsequent volumes can be as big as you like as they’ll be connected to the virtual SCSI controller on the VM, not the virtual IDE controller.
- After provisioning, rescan the disks in Disk Administrator on all cluster nodes.
- Bring the new disk into the cluster as a storage resource and give it a sensible name.
- Take the disk offline (from the storage node of the Cluster Manager)
- Create the new guest VM with its configuration on the CSV, but with no VHD attached
- Edit the settings of the new VM (or an existing one) and add the LUN as a physical disk
- Fire up the VM and install the OS (or partition & format the disk for non-boot disks) – remember always to use Quick Format or you’ll break the thin provisioning.
- Shut down the VM.
- Refresh disk administrator on all cluster nodes
- Refresh the Storage node and expand the disk resource to see the volumes. They must start with \\?\GLOBALROOT\.
- Check Clustered VM properties for correct live migration network and dependencies. Note that dependencies should be built for you.
- Remove any non-clustered resources, such as the DVD or ISO you used to install the OS
- Do a final refresh of the VM configuration in cluster manager and check it’s clean
- Start your VM and try a live migration!
I hope this helps somebody.