Applies To:

Show Versions Show Versions

Manual Chapter: FirePass® Controller version 5.5 Administrator Guide: Using FirePass Controllers for Failover
Manual Chapter
Table of Contents   |   << Previous Chapter   |   Next Chapter >>


10

Using FirePass Controllers for Failover


Understanding FirePass controller high availability

A failover configuration is ideal for providing high availability for one site at a single location. High availability is the process of ensuring access to resources despite any failures or loss of service in the setup. For hardware, high availability is ensured by the presence of a redundant system, a configuration that transfers service to another piece of hardware in the event of failure on the first piece of hardware.

Two FirePass controllers have the capacity to act as a redundant system, or failover pair: two identically configured controllers working together to provide a higher degree of availability than a standalone controller for remote users.

A failover pair of FirePass controllers is composed of one controller in an active state, and one in a standby state, at any given moment. The active controller serves all requests from users. If the active controller fails, the standby controller takes over the active role. This process of transferring control from one device to another is called failover.

You can configure a failover pair with newly acquired controllers, or you can expand your current setup into a failover configuration.

Important

If you plan to introduce a failover configuration into your environment, and your configuration is already in production, you should review Introducing a failover member into a production environment before continuing.

For organizations with larger sites and multiple locations with FirePass controllers, a cluster of FirePass controllers can provide additional scalability of a high-availability configuration. For more information about clustering, see Understanding FirePass controller clusters, on page 12-1.

Introducing failover configuration

This chapter assumes you already have installed the FirePass controllers and have completed their initial network configuration. For installation information, see the FirePass Controller Getting Started Guide, available as a separate document. For initial network configuration information, see Configuring web services, on page 8-17 of this guide.

The procedures in this chapter guide you through the process of setting up FirePass controller failover pairs. Once the controllers are properly configured for failover, you only need to make subsequent configuration changes on the active controller. The primary controller synchronizes information on the standby controller, except network configuration and SNMP configuration.

These are the requirements for creating a failover pair:

  • You must have two FirePass 1000, 4000, or 4100 systems available.
  • Each pair of systems must be running the same software version.
  • Either both systems have identical features licensed, or one of the two units is licensed as a failover-only FirePass controller.
Important

If you have a failover-only controller, you must configure it second. For more information, see Configuring the standby FirePass controller.

Reviewing the configuration process

To configure the failover settings on the FirePass controllers, you need to complete several tasks, in order.

This section presents an overview of the configuration process and links to procedures containing specific steps.

The process for setting up failover has three main tasks.

  • First you configure the active FirePass controller.
  • Then you complete similar tasks to configure the standby FirePass controller.
  • Once both controllers are configured, you will want to verify the configuration.
Important

If you plan to introduce a failover configuration into your environment, and your configuration is already in production, you should review Introducing a failover member into a production environment before continuing.

Reviewing the configuration of the active failover member

The first part of the failover configuration process involves setting up the active member of the failover pair.

  • Enable failover
    Enabling failover is the first task in configuring the active controller. There are two parts to this task, each of which is covered in Enabling failover on the active controller.
    • Activating failover.
      You must activate the failover option and restart the controller to enable additional failover screens.
    • Configuring a fully qualified domain name (FQDN)
      You must make sure that the controllers in a failover pair share a name.
  • Configure a device-specific, self IP address
    This is part of the initial installation and configuration tasks, but if you have not already done so, configure at least one device-specific (that is, not virtual), self IP address for each interface and VLAN interface you plan to use for failover. For more information, see Configuring the active controller with a self IP address.
  • Note: If you change an IP address on a VLAN interface, verify that the configuration is using the new IP address on the synchronization agent and for the heartbeat.

  • Configure a shared, virtual IP address
    Configure the active controller with a shared or virtual IP address. A shared or virtual IP address is a shared identifier of a computer. The active controller and the standby controller share this IP address, so that either controller can assume the shared IP address when it is the active controller. For more information, see Configuring the active controller with a shared IP address.
  • Then you must finalize the changes and restart the controller before continuing. For more information, see To finalize the setup, and To restart the controller or service.

  • Configure entries on your Domain Name Service (DNS) server
  • You complete this part of the process on your network's DNS server, not on the FirePass controllers.
    On your DNS server, create an entry that maps the FQDN of the pair to the shared IP address, and create entries for each device using the self IP address.

    The FirePass controller creates host names for each physical device by appending numbers. You must create a DNS entry for each of these names using each device's self IP address. If the FQDN you are using is Failover, the FirePass controller assigns Failover-1 to the first failover pair member and Failover-2 to the second one. You will need DNS entries for each.

  • Add and configure web services, and specify a synchronization service
    When you have configured a self IP and a shared IP address on the active controller, you can configure web services associated with each IP address. You also need to make some configuration changes and specify a synchronization agent for web services on the self IP address. For more information, see Configuring web services for the shared IP address of the active controller, and Configuring a web service as a synchronization agent for the active controller's self IP address.
  • Configure the heartbeat
    The heartbeat is a activity indicator signal that the active controller broadcasts to the subnet, where the standby controller receives it. For information about configuring the heartbeat, see Configuring the active controller's heartbeat, synchronization, and miscellaneous settings.
  • Finalize and restart the active controller
    After configuring web services, you must finalize and restart the controller.
    Restarting the controller puts the failover configuration changes into effect and reveals additional failover screens and settings. For more information, see To finalize the setup, and To restart the controller or service.

Reviewing the configuration of the standby failover member

The second part of the failover configuration process involves setting up the standby member of the failover pair.

Reviewing the verification process

When you have configured both the active and standby controllers, verify that the configuration is working correctly. For more information, see Verifying the failover configuration.

Warning

If you are configuring failover in a production environment, the order in which the pair of controllers restart is very important, and can result in data loss if the two controllers do not restart in the correct order. For more information, see Introducing a failover member into a production environment, following.

Introducing a failover member into a production environment

If you are creating a failover pair from a new controller and an existing controller, make sure to carefully watch when the units restart so that you never let the new, potentially partially configured controller take over accidentally. Restarting the active controller causes the standby to take over, which erases all configuration on the previously active controller.

Note

Always back up any production controller before configuring failover. You can read more about backing up the FirePass controller in Backing up and restoring the FirePass controller, on page 8-37.

Once you enable failover and configure the IP addresses for the active and standby controllers, the process of restarting one controller fails-over automatically to the other one. Failover configuration requires a system restart.

A good strategy to follow when deploying a FirePass high availability configuration has three parts:

  • Configure one FirePass controller for failover, typically, the existing one you have in production, and then shut it down.
  • Configure the second FirePass controller, and then shut it down.
  • Restart the controller that you want to serve as the active failover member, and then start the standby controller.

You can use the backup and restore feature to set up failover pairs. Backup and restore transfers settings to a one or more FirePass controllers. Even though you transfer settings, you must still complete other configuration tasks. For more information, see Reviewing the configuration process.

Important

When you restore the backup, do not restore the network settings to the standby controller.
Note

For more information on using the backup and restore feature, see Backing up and restoring the FirePass controller, on page 8-37. For more information about using the backup and restore feature to transfer identical settings to a number of FirePass controllers, see the Configuring the BIG-IP System with FirePass Controllers for Load Balancing and SSL Offload document on the Solution Center at the F5 corporate web site, http://www.f5.com/solutions/.

Configuring the active FirePass controller

When setting up a failover configuration, your first set of tasks are performed on the active controller.

Before configuring failover settings on the active controller, make sure both the active and standby controllers are configured with the same FQDN. After confirming this, enable failover on the active controller, create a virtual IP address, and configure web services for that IP address.

You must configure the following IP addresses for each failover controller:

  • A dedicated self IP address and port on each controller. The IP address and port setting required is the self IP address and port of the interface or VLAN interface on the controller. This address must be unique for each controller in the failover pair. The self IP addresses for the two failover controllers must be on the same IP subnet. Typically, you configure synchronization for the same port, but using a different IP address.
  • At least one shared IP address for the failover pair. This shared IP address is what establishes the association between the members of a failover pair.
Important

If you are configuring failover in a production environment, or on an existing FirePass controller, make a full backup of the controller before making any configuration changes. For information about backing up a FirePass controller, see Backing up and restoring the FirePass controller, on page 8-37.

Enabling failover on the active controller

You need to enable failover on the active FirePass controller before continuing with the configuration tasks. When you enable failover the system prompts you to restart the controller. After you restart the controller, the navigation pane and the Web Services configuration screen present additional failover configuration screens and options.

Note

If the screen does not show the Failover tab or other failover-related menu items after you enable failover, refresh the view in your web browser.

To enable failover on the active FirePass controller

  1. In the navigation pane, click Device Management, expand Configuration, and click Network Configuration.
    The Network Configuration screen opens.
  2. Click the Hosts tab at the top of the screen.
    The Hosts screen opens.
  3. Confirm that the name of the controller in the FQDN of the controller box is the name you want to use for the failover pair. These names must match on both the active and standby controllers.
  4. In the navigation pane, click Clustering and Failover.
    The Clustering and Failover screen opens.
  5. Scroll down to the Failover (High-Availability) Configuration area, and make these changes:
    1. Check the Enable Failover Configuration check box.
    2. From the Failover Pair Member list, select First.
    3. Copy the value from the Failover ID box.
      Paste this value into a text file or write it down. You will need this value for configuring the standby FirePass controller.
  6. In the Clustering/Failover Global ID area, copy the value from Cluster/Failover Global ID box.
    Paste this value into a text file or write it down. You will need this value to configure the standby FirePass controller.
  7. To commit the settings, click Apply Clustering/Failover Settings.
  8. Finalize the setup, and restart the controller.
    For more information, see Finalizing the configuration, following, and Restarting the controller or services after configuration, following.

Finalizing the configuration

Many web services configuration changes require a finalize step. You can use steps in this procedure for all finalize operations described in these procedures.

To finalize the setup

  1. Click the Finalize tab at the top of the screen.
  2. Review the changes.
  3. Click the Finalize Changes button.

Restarting the controller or services after configuration

Some configuration changes also require a controller restart or services restart. You can use steps in this procedure for all restart operations described in these procedures.

To restart the controller or service

  1. Click the indicated text.
  2. Confirm the restart.

Configuring the active controller with a self IP address

For the failover process to work, each active and standby FirePass controller must have an IP address that it uses to communicate with the other. This IP address, the self IP address, uniquely identifies each FirePass controller interface or VLAN interface for the purpose of synchronization.

You created a self IP address during the initial configuration process. If you want to use that self IP address for failover, you can skip this procedure. If you want to use an interface other than the recommended one, you can create a new self IP address on another interface. If you do so, you must also connect the two FirePass controller interfaces using a separate straight or crossover cable.

Warning

Be extremely careful when changing the FirePass controller's IP configuration settings. If you enter incorrect settings, the FirePass controller might become inaccessible from the network. If that happens, you must have physical access to the FirePass controller device to start up the controller again. You cannot use the browser interface to start up the FirePass controller, you must use the Maintenance Console connected to the device.

To configure the active controller's self IP address

  1. In the navigation pane, click Device Management, expand Configuration, click Network Configuration, and click the IP Config tab.
    The IP Configuration screen opens.
  2. Under Add New IP in the IP Address /Netmask box, type the IP address in dotted-decimal notation, and the subnet mask in bits notation.
    (In the online help for this screen, you can find a table that shows the mapping between bits, dotted-decimal, and hexadecimal netmask.)
  3. In Broadcast IP, type the IP for the controller to use to send messages to the subnet. If you do not specify a broadcast IP address, the FirePass controller calculates a default broadcast address from the IP address and mask.
  4. From the Interface list, select the device-specific interface or VLAN interface associated with the IP address.
    You can configure web services so that all the traffic goes through a single interface, or you can use one interface for synchronization and another for other traffic.
  5. For FirePass 1000 controllers, we recommend that the public subnet be associated with the eth0 interface, and the private subnet be associated with the eth1 interface. For FirePass 4100 controllers we recommend the public subnet be associated with the eth1.1 interface.

  6. Click Add New to add the self IP address.
  7. Click the Finalize tab at the top of the screen.
    The Finalize Settings screen opens.
  8. Review the changes, and click the Finalize Changes button.

Configuring the active controller with a shared IP address

The failover pair of controllers shares a virtual IP address. Sharing this IP address makes it possible for the standby controller to take over the network traffic in the event of a failure.

To configure a shared, virtual IP address on the active FirePass controller

  1. In the navigation pane, click Device Management, expand Configuration, and click Network Configuration.
    The IP Configuration screen opens.
  2. In the Add New IP area, in the IP Address/Netmask box, type a new IP address and subnet mask for the shared IP address.
  3. Check the Virtual check box.
    The Virtual check box is present only after you have enabled the unit for failover.
  4. Leave the Broadcast IP box empty for the shared IP address.
  5. Select the appropriate network interface from the Interface list.
  6. Click Add New to add the shared IP address.
  7. Finalize the changes, and restart if necessary.
    For specific steps, see Finalizing the configuration, and Restarting the controller or services after configuration.

Configuring web services for the shared IP address of the active controller

After adding a shared IP address to the active controller, you need to configure its web services. Which services you configure, and the ports you use, depend on how your local network and firewall are set up, and on what FirePass controller features you use.

Configuring secure web services on port 443 for the active controller

The secure web services on port 443 is the one users log on to.

To configure web services on port 443 of the self IP address for the active controller

  1. In the navigation pane, click Device Management, expand Configuration, and click Network Configuration.
    The IP Configuration screen opens.
  2. Click the Web Services tab at the top of the screen.
    The Web Server Configuration screen opens.
  3. In the Add new service area, from the IP list, select the shared IP address.
  4. In the Port box, type 443.
  5. In the Name box, type the FQDN of the FirePass controller.
  6. From the For Mode list, select Always.
  7. Check the SSL check box.
  8. To add the new service, click Add New.
    The Web Service Configuration for <hostname> screen opens for the new service.
  9. From the Certificate list, select the certificate.
  10. Check the User Login check box.
  11. If you want, check the Admin Login check box.
    If you check this option, you can log on to the controller's Administrative Console. Otherwise, you are redirected to the active failover member.
  12. Leave all other options unchecked.
  13. To commit the settings, click Update.
  14. Finalize the changes, and restart if necessary.
    For specific steps, see Finalizing the configuration, and Restarting the controller or services after configuration.

Configuring web services on port 80 for the active controller

Port 80 is not required, though you may need to configure it based on your network configuration.

To configure web services on port 80 of the physical IP address for the active controller

  1. In the navigation pane, click Device Management, expand Configuration, and click Network Configuration.
    The IP Configuration screen opens.
  2. Click the Web Services tab at the top of the screen.
    The Web Server Configuration screen opens.
  3. In the Add new service area, from the IP list, select the shared IP address.
  4. In the Port box, type 80.
    You can configure web services for any port, not just port 80.
  5. In the Name box, type the FQDN of the FirePass controller.
  6. From the For Mode list, select ActiveOnly.
    This setting causes the controller to load web services only when it is the active controller in a failover pair.
  7. To add the new service, click Add New.
    The Web Service Configuration for <hostname> screen opens for the new service.
  8. In the HTTPS URL to redirect to box, type the URL of the HTTPS Web service (port 443) on the shared IP address that you configured in Configuring secure web services on port 443 for the active controller, in the preceding procedure.
  9. Check the User Login check box.
  10. Leave all other options unchecked.
  11. To commit the settings, click Update.
  12. Finalize the changes, and restart if necessary.
    For specific steps, see Finalizing the configuration, and Restarting the controller or services after configuration.

Configuring a web service as a synchronization agent for the active controller's self IP address

After configuring web services for the active controller's virtual IP address you need to also configure a synchronization service for the controller's physical IP address.

Note

You can configure synchronization for any port, not just port 81.

To create a web service as a synchronization agent on port 81 of the self IP address for the active controller

  1. In the navigation pane, click Device Management, expand Configuration, and click Network Configuration.
    The IP Configuration screen opens.
  2. Click the Web Services tab at the top of the screen.
    The Web Server Configuration screen opens.
  3. From the IP list, in the Add new service area, select the self IP address of the active controller.
  4. In the Port box, type 81.
    You can configure synchronization for any port, not just port 81.
  5. In the Name box, type the FQDN of the FirePass controller.
    You can leave Name blank if the self IP address does not have a domain name specified in DNS, or if you want to use the self IP address as the name.
  6. From the For Mode list, select ActiveOnly.
    This setting causes the controller to load web services only when it is the active controller in a failover pair.
  7. To add the new service, click Add New.
    The Web Service Configuration for <hostname> screen opens for the new service.
  8. Check the Do not redirect to HTTPS check box.
  9. Check the Synchronization Agent check box.
  10. Leave all other options unchecked.
  11. To commit the settings, click Update.
  12. Finalize the changes, and restart if necessary.
    For specific steps, see Finalizing the configuration, and Restarting the controller or services after configuration.

Configuring the active controller's heartbeat, synchronization, and miscellaneous settings

The active and standby controllers communicate with each other using a heartbeat. The heartbeat is a signal sent at 100-millisecond intervals that notifies the standby node that the active node is running. If the standby node does not receive a heartbeat within 200 milliseconds of the expected arrival time, the standby node considers its peer inactive, assumes its virtual IP address, and becomes the active member of the pair.

Heartbeat settings specify the interface and port a controller uses while it is the active member of the failover pair.

Synchronization settings consist of the self IP address of the active controller and the self IP address of the standby controller. To use a port, it must be configured as a synchronization service. A synchronization service is a web service that is enabled for HTTP and configured as a synchronization agent. For a procedure to follow, see Configuring a web service as a synchronization agent for the active controller's self IP address.

To configure the active controller's heartbeat, synchronization settings, and miscellaneous settings

  1. In the navigation pane, click Device Management, expand Configuration, and click Network Configuration.
    The IP Configuration screen opens.
  2. Click the Failover tab at the top of the screen.
    The Failover Configuration screen opens.
  3. From Network Interface to use for the heartbeat, select the interface you want to use for transmitting the heartbeat.
  4. In UDP port to use for heartbeat, specify the port number you want to use for transmitting the heartbeat. The default is 694.
    If you want to use a different port for UDP, specify that value instead. You must specify the same value on the standby controller.
  5. In IP address and port on this machine to use for synchronization, select the self IP address and port you configured for the active controller in Configuring the active controller with a self IP address .
  6. In IP address and port on the other member of this failover pair to use for synchronization, specify the self IP address and port of the other member of the failover pair. You configure this setting in Configuring the active FirePass controller.
    The standby controller uses this IP address and port for synchronization with the active controller.
  7. Click the Misc tab at the top of the screen.
    The Misc screen opens.
  8. From the Local X11 server source address list, select the shared IP address.
  9. From the NetBIOS broadcast source address list, select the shared IP address.
  10. From the Network Access source address list, select the shared IP address.
  11. From the NAS IP Address for RADIUS Requests list, select the shared IP address.
  12. To commit the settings, click Update.
  13. Finalize the changes, and restart if necessary.
    For specific steps, see Finalizing the configuration, and Restarting the controller or services after configuration.

Configuring the standby FirePass controller

After you have configured the active FirePass controller, you must configure the standby controller so that it can take over in the event of a failure of the active controller.

The basic process consists of the following tasks

The configuration process is similar to the one you followed when you configured the active controller, with two exceptions.

Enabling failover on the standby controller

Enabling failover on the standby controller is almost the same as enabling failover on the active controller, so you may find it useful to review the procedure for configuring the active controller, Enabling failover on the active controller, following.

To configure standby settings on the Clustering and Failover Settings screen

  1. In the navigation pane, click Device Management, expand Configuration, and click Clustering and Failover.
    The Clustering and Failover screen opens.
  2. From the Failover Pair Member list, select Second.
    You select Second when you are configuring a failover member that is licensed as failover only, and when you are configuring the second fully licensed member of a failover pair.
  3. In the Failover ID box, type or paste the failover ID you recorded in step 5 of To enable failover on the active FirePass controller.
  4. In the Clustering/Failover Global ID area in the Cluster/Failover Global ID box, type or paste the Cluster/Failover Global ID you recorded in step 6 of To enable failover on the active FirePass controller.

Configuring the standby controller with a self IP address

Before continuing with this task, make sure to review the associated procedure for configuring the active controller, Configuring the active controller with a self IP address.

To specify the self IP address for the standby controller

  1. In the navigation pane, click Device Management, expand Configuration, and click Network Configuration, and click the Failover tab.
    The Failover Settings screen opens.
  2. In the IP address and port on this machine to use for synchronization box on the Failover Settings screen, select the self IP address and port you configured for the standby controller in Configuring the active controller with a self IP address.
  3. In IP address and port on the other member of this failover pair to use for synchronization, specify the self IP address and port of the other member of the failover pair. You configure this setting in Configuring the active FirePass controller.
    The standby controller uses this IP address and port for synchronization with the active controller.

Configuring a shared IP address

Follow the procedure steps shown in Configuring the active controller with a shared IP address.

Checking the FQDN

For more information, see Enabling failover on the active controller.

Configuring DNS server entries

For more information, see Configure entries on your Domain Name Service (DNS) server.

Adding and configuring web services, and specify a synchronization service

Follow the procedure steps shown in Configuring web services for the shared IP address of the active controller, and Configuring a web service as a synchronization agent for the active controller's self IP address.

Configuring the heartbeat

Follow the procedure steps shown in Configuring the active controller's heartbeat, synchronization, and miscellaneous settings.

Finalizing and restarting the active controller

For more information, see To finalize the setup, and To restart the controller or service.

Accessing a standby controller that is already in production

If you want to log on to a standby controller that is already a member of a failover pair, you must log on using https://standby-self-IP/admin/.

The presence of the trailing /admin/ designation in the URL enables access to the standby controller directly. If you do not specify the trailing /admin/, you are redirected to the active failover member.

For example, to access a physical device named fail2 that has an IP address of 10.4.1.2.198, you could specify one of the following in the browser address bar, and then log on as usual.

https://fail2.siterequest.com/admin/

https://10.4.12.198/admin/

These examples assume that you created a DNS entry for the standby controller using its self IP address.

Post-configuration tasks

After you have configured both FirePass controllers for failover, confirm that the failover configuration is working.

Starting failover controllers

If both failover controllers are turned off, the first controller that you start automatically assumes the role of active controller, and the second controller you start becomes the standby controller. The two controllers remain in this state until either the active controller fails and the standby controller takes over, or you restart the active controller, and the standby controller becomes the active controller.

If a pair of failover controllers is started simultaneously, the controller configured as First on the Failover settings screen becomes the active controller, and the controller configured as Second on the Failover settings screen becomes the standby controller. You can determine which controller is first and which is second by checking the value in the Failover Pair Member box on the Clustering and Failover screen. To access the screen, click Device Management, expand Configuration, and click Clustering and Failover. The first one is designated First, and the second one is designated Second.

Verifying the failover configuration

After configuring the active and standby FirePass controllers, verify that the configuration is properly working.

To verify that your failover configuration is working

  1. In the navigation pane, click Failover.
    The Failover : Settings screen opens.
  2. Verify that the failover controllers are properly configured:
    1. Confirm that the current controller is active by looking at the value of This node.
      If the controller is active, it contains the designation (active).
    2. Confirm that the two controllers are communicating by looking at the status line that indicates how many seconds have passed since the active controller synchronized data with the standby controller.
      If the interval has been too long, the screen displays a warning. Synchronization might take more time if a lot of data has to be transferred, for example, if you make significant changes on the primary controller.
  3. Restart the current, active controller so that the standby controller fails over.
    1. Click Restart This Node, Make <standby controller> Active to restart the current controller.
      The current controller restarts and becomes the standby controller, and the standby controller takes over as the active controller.
    2. After restart, check the identity of each controller. For more information, see Verifying controller identity, following.

Verifying controller identity

You can determine the identity of the controllers by logging on to each one.

To verify the identity of the controller

  1. Log on to the active controller directly.
  2. Verify that the Welcome screen indicates that the device is the standby controller.
    The screen presents the following message:
    This node is in failover active mode
  3. Log on to the standby controller directly.
    For more information, see Configuring the standby FirePass controller.
  4. Verify that the Welcome screen indicates that the device is the standby controller.
    The screen presents the following message:
    This node is in failover standby mode

Triggering manual failover

You can manually trigger a failover to verify that the configuration of the failover pair is correct. You might also need to manually trigger a failover if you need to make changes to your active controller.

To manually trigger a failover to a standby controller

  1. In the navigation pane, click Failover, and then click Settings.
    The Failover Settings screen opens.
  2. Click Restart This Node, Make <standby controller> Active.
    The current controller restarts and becomes the standby controller, while the standby controller takes over as the active controller.

You can also trigger failover manually using the Restart Controller link on the Restart Services screen. To access the screen, in the navigation pane, click Device Management, expand Maintenance, and then click Restart Services.




Table of Contents   |   << Previous Chapter   |   Next Chapter >>

Was this resource helpful in solving your issue?




NOTE: Please do not provide personal information.



Incorrect answer. Please try again: Please enter the words to the right: Please enter the numbers you hear:

Additional Comments (optional)