Книга: Mastering VMware® Infrastructure3
Configuring HA
Разделы на этой странице:
Configuring HA
Before we detail how to set up and configure the HA feature, let's review the requirements of HA. To implement HA, all of the following requirements should be met:
? All hosts in an HA cluster must have access to the same shared storage locations used by all virtual machines on the cluster. This includes any fibre channel, iSCSI, and NFS datas tores used by virtual machines.
? All hosts in an HA cluster should have an identical virtual networking configuration. If a new switch is added to one host, the same new switch should be added to all hosts in the cluster.
? All hosts in an HA cluster must resolve the other hosts using DNS names.
A Test for HA
An easy and simple test for identifying HA capability for a virtual machine is to perform a VMotion. The requirements of VMotion are actually more stringent than those for performing an HA failover, though some of the requirements are identical. In short, if a virtual machine can successfully perform a VMotion across the hosts in a cluster, then it is safe to assume that HA will be able to power on that virtual machine from any of the hosts. To perform a full test of a virtual machine on a cluster with four nodes, perform a VMotion from node 1 to node 2, node 2 to node 3, node 3 to node 4, and finally node 4 back to node 1. If it works, then you have passed the test!
First and foremost, to configure HA a cluster must be created. Once the cluster is created, you can enable and configure HA. Figure 10.28 shows a cluster enabled for HA.
Figure 10.28 A cluster of ESX Server hosts can be configured with HA and DRS. The features are not mutually exclusive and can work together to provide availability and performance optimization.
Configuring an HA cluster revolves around three different settings:
? Host failures allowed
? Admission control
? Virtual machine options
The configuration option for the number of host failures to allow, shown in Figure 10.29, is a critical setting. It directly influences the number of virtual machines that can run in the cluster before the cluster is in jeopardy of being unable to support an unexpected host failure.
Figure 10.29 The number of host failures allowed dictates the amount of spare capacity that must be retained for use in recovering virtual machines after failure.
Real World Scenario
HA Configuration Failure
It is not uncommon for a host in a cluster to fail during the configuration of HA. Remember the stress we put on DNS earlier in the chapter. Well, if DNS is not set correctly, you will find that the host cannot be configured for HA. Take, for example, a cluster with three nodes being configured as an HA cluster to support two-node failure. Enabling HA forces a configuration of each node in the cluster. The image here shows an HA cluster where one of the nodes, Silo104, has thrown an error related to the HA agent and is unable to complete the HA configuration:
In this example, because the cluster was attempting to allow for two-node failure and there are only two nodes successfully configured, this would be impossible. The cluster in this case is now warning that there are insufficient resources to satisfy the HA failover level. Naturally, with only two nodes we cannot cover two-node failure. The image here shows an error on the cluster due to the failure in Silo104:
In the tasks pane of the graphic, you may have noticed that Silo105 and Silo106 both completed the HA configuration successfully. This provides evidence that the problem is probably isolated to Silo104. Reviewing the Tasks & Events tab to get more detail on the error reveals exactly that. The next image shows that the error was caused by an inability to resolve a name. This confirms the suspicion that the error is with DNS.
Perform the following steps to review or edit the DNS server for an ESX Server:
1. Use the Virtual Infrastructure (VI) Client to connect to a VirtualCenter server.
2. Click the name of the host in the inventory tree on the left.
3. Click the Configuration tab in the details pane on the right.
4. Click DNS and Routing from the Advanced menu.
5. If needed, edit the DNS server, as shown in the image here, to a server that can resolve the other nodes in the HA cluster:
Although they should not be edited on a regular basis, you can also check the /etc/hosts and /etc/resolv.conf files, which should contain static lists of hostnames to IP addresses or the DNS search domain and name servers, respectively. This image offers a quick look at the information inside the /etc/hosts and /etc/resolv.conf files. These tools can be valuable for troubleshooting name resolution.
Once the DNS server, /etc/hosts, or /etc/resolv.conf has been corrected, the host with the failure can be reconfigured for HA. It's not necessary to remove the HA configuration from the cluster and then re-enable it. The image here shows the right-click context menu of Silo104, where it can be reconfigured for HA now that the name resolution problem has been fixed.
Upon completion of the configuration of the final node, the errors at the host and cluster levels will be removed, the cluster will be configured as desired, and the error regarding the inability to satisfy the failover level will disappear.
To explain the workings of HA and the differences in the configuration settings, let's look at implementation scenarios. For example, consider five ESX Server hosts named Silo101 through Silo105. All five hosts belong to an HA cluster configured to support single-host failure. Each node in the cluster is equally configured with 12GB of RAM. If each node runs eight virtual machines with 1GB of memory allocated to each virtual machine, then 8GB of unused memory across four hosts is needed to support a single-host failure. The 12GB of memory on each host minus 8GB for virtual machines leaves 4GB of memory per host. Figure 10.30 shows our five-node cluster in normal operating mode.
Figure 10.30 A five-node cluster configured to allow single-host failure.
Let's assume that service console and virtual machine overhead consume 1GB of memory, leaving 3GB of memory per host. If Silo101 fails, the remaining four hosts will each have 3GB of memory to contribute to running the virtual machines orphaned by the failure. The 8GB of virtual machines will then be powered on across the remaining four hosts that collectively have 12GB of memory to spare. In this case, the configuration supported the failover. Figure 10.31 shows our five-node cluster down to four after the failure of Silo101. Assume in this same scenario that Silo101 and Silo102 both experience failure. That leaves 16GB of virtual machines to cover across only three hosts with 3GB of memory to spare? In this case, the cluster is deficient and not all of the orphaned virtual machines will be restarted.
Figure 10.31 A five-node cluster configured to allow single-host failure is deficient in resources to support a second failed node.
Primary Host Limit
In the previous section introducing the HA feature, we mentioned that the Automated Availability Manager (AAM) caps the number of primary hosts at five. This limitation translates into a maximum of four host failures allowed in a cluster.
The admission control setting goes hand in hand with the Number of host failures allowed setting. There are two possible settings for admission control:
? Do not power on virtual machines if they violate availability constraints (known as strict admission control).
? Allow virtual machines to be powered on even if they violate availability constraints (known as guaranteed admission control).
In the previous example, virtual machines would not power on when Silo102 experienced failure because by default an HA cluster is configured to use strict admission control. Figure 10.32 shows an HA cluster configured to use the default setting of strict admission control.
Figure 10.32 Strict admission control for an HA cluster prioritizes resource balance and fairness over resource availability.
With strict admission control, the cluster will reach a point at which it will no longer start virtual machines. Figure 10.33 shows a cluster configured for two-node failover. A virtual machine with more than 3GB of memory reserved is powering on, and the resulting error is posted stating that insufficient resources are available to satisfy the configured HA level.
Figure 10.33 Strict admission control imposes a limit at which no more virtual machines can be powered on because the HA level would be jeopardized.
If the admission control setting of the cluster is changed from strict admission control to guaranteed admission control, then virtual machines will power on even in the event that the HA failover level is jeopardized. Figure 10.34 shows a cluster reconfigured to use guaranteed admission control.
Figure 10.34 Guaranteed admission control reflects the idea that when failure occurs, availability is more important than resource fairness and balance.
With that same cluster now configured with guaranteed admission control, the virtual machine with more than 3GB of memory can now successfully power on. In Figure 10.35, the virtual machine has successfully powered on despite the large memory use and lack of available unused resources to achieve the proper HA failover.
Figure 10.35 Guaranteed admission control allows resource consumption beyond the levels required to maintain spare resources for use in the event of a server failure.
Overcommitment in an HA cluster
When the admission control setting is set to allow virtual machines to be powered on even if they violate availability constraints, you could find yourself in a position where there is more physical memory allocated to virtual machines than actually exists. This situation, called overcommitment, can lead to poor performance on virtual machines that become forced to page information from fast RAM out to the slower disk based swap file.
HA Restart Priority
Not all virtual machines are equal. There are some that are more important or more critical and that require higher priority when ensuring availability. When an ESX Server host experiences failure and the remaining cluster nodes are tasked with bringing virtual machines back on line, they have a finite amount of resources to fill before there are no more resources to allocate to virtual machines that need to be powered on. Rather than leave the important virtual machines to chance, an HA cluster allows for the prioritization of virtual machines. The restart priority options for virtual machines in an HA cluster include Low, Medium, High, and Disabled. For those virtual machines that should be brought up first, the restart priority should be set to High. For those virtual machines that should be brought up if resources are available, the restart priority can be set to Medium and/or Low. For those virtual machines that will not be missed for a period of time and should not be brought on line during the period of reduced resource availability, the restart priority should be set to Disabled. Figure 10.36 shows an example of virtual machines with various restart priorities configured to reflect their importance. The diagram details a configuration where virtual machines like domain controllers, database servers, and cluster nodes are assigned higher restart priority.
Figure 10.36 Restart priorities help minimize the downtime for more important virtual machines.
The restart priority is only put into place for the virtual machines running on the ESX Server host that experienced an unexpected failure. Virtual machines running on hosts that have not failed are not affected by the restart priority. It is possible then that virtual machines configured with a restart priority of High may not be powered on by the HA feature due to limited resources, which are in part due to lower-priority virtual machines that continue to run. For example, as shown in Figure 10.37, Silo101 hosts five virtual machines with a priority of High and five other virtual machines with priority values of Medium and Low. Meanwhile, Silo102 and Silo103 each hold ten virtual machines, but of the 20 virtual machines between them, only four are considered of high priority. When Silo101 fails, Silo102 and Silo103 will begin powering the virtual machines with a high priority. However, assume there were only enough resources to power on four of the five virtual machines with high priority. That leaves a high-priority virtual machine powered off while all other virtual machines of medium and low priority continue to run on Silo102 and Silo103.
Figure 10.37 High-priority virtual machines from a failed ESX Server host may not be powered on because of a lack of resources — resources consumed by virtual machines with a lower priority that are running on the other hosts in an HA cluster.
At this point in the VI3 product suite, you can still manually remedy this imbalance. Any disaster-recovery plan in a virtual environment built on VI3 should include a contingency plan that identifies virtual machines to be powered off to make resources available for those virtual machines with higher priority as a result of the network services they provide. If the budget allows, construct the HA cluster to ensure that there are ample resources to cover the needs of the critical virtual machines, even in times of reduced computing capacity.
HA Isolation Response
Previously, we introduced the Automated Availability Manager (AAM) and its role in conducting the heartbeat that occurs among all the nodes in the HA cluster. The heartbeat among the nodes in the cluster identifies the presence of each node to the other nodes in the cluster. When a heartbeat is no longer presented from a node in the HA cluster, the other cluster nodes spring into action to power on all the virtual machines that the missing node was running.
But what if the node with the missing heartbeat was not really missing? What if the heartbeat was missing but the node was still running? And what if the node with the missing heartbeat is still locking the virtual machine files on SAN LUN, thereby preventing the other nodes from powering on the virtual machines?
Let's look at two particular examples of a situation VMware refers to as a split-brained HA cluster. Let's assume there are three nodes in an HA cluster: Silo101, Silo102, and Silo103. Each node is configured with a single virtual switch for VMotion, and a second virtual switch consisting of a Service Console port and a virtual machines port group, as shown in Figure 10.38.
Figure 10.38 ESX Server hosts in an HA cluster using a single virtual switch for Service Console and virtual machine communication.
To continue with the example, suppose that an administrator mistakenly unplugs the Silo101 Service Console network cable. When each of the nodes identifies a missing heartbeat from another node, the discovery process begins. After 15 seconds of missing heartbeats, each node then pings an address called the isolation response address. By default this address is the default gateway IP address configured for the Service Console. If the ping attempt receives a reply, the node considers itself valid and continues as normal. If a host does not receive a response, as Silo101 wouldn't, it considers itself in isolation mode. At this point, the node will identify the cluster's Isolation Response configuration, which will guide the host to either power off the existing virtual machines it is hosting or leave them powered on. This isolation response value, shown in Figure 10.39, is set on a per-virtual machine basis. So what should you do? Power off the existing virtual machine? Or leave it powered on?
The answer to this question is highly dependent on the virtual and physical network infrastructures in place. In our example, the Service Console and virtual machines are connected to the same virtual switch bound to a single network adapter. In this case, when the cable for the Service Console was unplugged, communication to the Service Console and every virtual machine on that computer was lost. The solution, then, should be to power off the virtual machines. By doing so, the other nodes in the cluster will identify the releases on the locks and begin to power on the virtual machines that were not otherwise included.
Figure 10.39 The Isolation Response identifies the action to occur when an ESX Server host determines it is offline but powered on.
In the next example, we have the same scenario but a different infrastructure, so we don't need to worry about powering off virtual machines in a split-brain situation. Figure 10.40 diagrams a virtual networking architecture in which the Service Console, VMotion, and virtual machines all communicate through individual virtual switches bound to different physical network adapters. In this case, if the network cable connecting the Service Console is removed, the heartbeat will once again be missing; however, the virtual machines will be unaffected since they reside on a different network that is still passing communications between the virtual and physical networks.
Figure 10.40 Redundancy in the physical infrastructure with isolation of virtual machines from the Service Console in the virtual infrastructure provides greater flexibility for isolation response.
Figure 10.41 shows the isolation response setting of Leave powered on, which would accompany an infrastructure built with redundancy at the virtual and physical network levels.
Figure 10.41 The option to leave virtual machines running when a host is isolated should only be set when the virtual and the physical networking infrastructures support high availability.
Real World Scenario
Configuring the Isolation Response Address
In some highly secure virtual environments, Service Console access is limited to a single, nonrouted management network. In some cases, the security plan calls for the elimination of a default gateway on the Service Console port configuration. The idea is to lock the Service Console onto the local subnet, thus preventing any type of remote network access. The disadvantage, as you might have guessed, is that without a default gateway IP address configured for the Service Console, there is no isolation address to ping as a determination of isolation status.
It is possible, however, to customize the isolation response address for scenarios just like this. The IP address can be any IP address, but should be an IP address that is not going to be unavailable or taken from the network at any time.
Perform the following steps to define a custom isolation response address:
1. Use the VI Client to connect to a VirtualCenter server.
2. Right-click on an existing cluster and select the Edit Settings option.
3. Click the VMware HA node.
4. Click the Advanced Options button.
5. Type das.isolationaddress in the Option column in the Advanced Options (HA) dialog box.
6. Type the IP address to be used as the isolation response address for ESX Server hosts that miss the AAM heartbeat. The following image shows a sample configuration in which the servers will ping the IP address 172.30.0.2:
7. Click the OK button twice.
This interface can also be configured with the following options:
? das.isolationaddress1: To specify the first address to try.
? das.isolationaddress2: To specify the second address to try.
? das.defaultfailoverhost: To identify the preferred host to fail over to.
? das.failuredetectiontime: Used to change the amount of time required for failover detection.
To support a redundant HA architecture, it is best to ensure that the Service Console port is sitting atop a NIC team where each physical NIC bound to the virtual switch is connected to a different physical switch.
- Chapter 5. Preparations
- Chapter 8. Saving and restoring large rule-sets
- Chapter 10. Iptables matches
- Chapter 11. Iptables targets and jumps
- Chapter 12. Debugging your scripts
- Chapter 5 Installing and Configuring VirtualCenter 2.0
- Chapter 14. Example scripts
- Choosing, Configuring, and Installing the Boot Loader
- Configuring Wireless Networks
- Configuring X
- Installing and Configuring OpenOffice.org
- Configuring and Managing Print Services