AskF5 Knowledge Base

Applies To:

Show Versions Show Versions
sol14106: Troubleshooting virtual server and link auto-discovery (11.x)
TroubleshootingTroubleshooting

Original Publication Date: 01/04/2013

Purpose

You should consider using this procedure under the following conditions:

  • You configured virtual server or link auto-discovery
  • The BIG-IP GTM system failed to automatically discover virtual servers or link objects that are associated with defined BIG-IP systems

Description

The virtual server and link auto-discovery feature allows the BIG-IP GTM and Link Controller systems to automatically discover virtual servers and links that are associated with defined BIG-IP systems.

If auto-discovery is enabled for a server or link object, the system automatically discovers virtual servers and link objects as follows:

  • The gtmd process running on the local BIG-IP GTM system reads the auto-discovery settings from the /config/bigip_gtm.conf file
  • At the defined auto-discovery request interval, gtmd sends the <getconfig> request element over iQuery (SSL, port 4353) to the big3d process on the remote BIG-IP system
  • The receiving big3d process returns any defined virtual server and link information in the <getconfig_elements> iQuery block data
  • The local gtmd process then adds the discovered virtual servers and links to the BIG-IP GTM configuration

Symptoms

As a result of auto-discovery issues, you may encounter the following symptom:

  • The BIG-IP GTM system fails to automatically discover the virtual servers and links associated with defined BIG-IP systems.

Procedures

When experiencing auto-discovery issues, you can use the following troubleshooting steps to determine the root cause:

Confirm that the BIG-IP virtual servers do not use address translation

Auto-discovery is unavailable for virtual servers using translated IP addresses. Before troubleshooting auto-discovery issues, confirm that the BIG-IP virtual servers do not use address translation. For example, if the target BIG-IP virtual server IP addresses reside in a private network space, as defined by RFC1918, and are mapped to public IP addresses that are defined on a network device such as a firewall, the BIG-IP GTM system will silently disable the auto-discovery feature for the BIG-IP system.

Note: For more information, refer to SOL9138: The BIG-IP GTM disables virtual server auto-discovery for BIG-IP systems which utilize translated virtual server addresses.

Verify time synchronization

The BIG-IP GTM system time must be synchronized with GTM sync group members (30 seconds by default), and should be synchronized with the remote BIG-IP systems in which virtual server / link auto-discovery is enabled. To verify time synchronization and NTP configuration, perform the following procedure:

Impact of procedure: None.

  1. Log in to the command line of the F5 device.
  2. Type the following command:

    date; tmsh list /sys ntp

  3. Perform steps 1 through 2 on all devices.
  4. Verify that each device is configured to use an NTP server and the times are in sync.

Note: To configure NTP a BIG-IP system, refer to SOL3122: Configuring the BIG-IP system to use an NTP server using the Configuration utility.

Confirm iQuery communication

When the network topology consists of BIG-IP GTM systems load-balancing BIG-IP virtual servers, the systems must be able to communicate using the iQuery protocol. Upon starting, the gtmd process attempts to establish an SSL connection on port 4353 with the big3d process running on BIG-IP systems. The BIG-IP GTM system performs this action for all defined BIG-IP server objects, creating an iQuery mesh between gtmd and big3d processes. If there is a failure in the iQuery communication channel, gtmd will fail to automatically discover objects on the target BIG-IP system. One way to confirm that iQuery communication is functioning is to check the BIG-IP server status on all BIG-IP GTM devices. The BIG-IP server objects should appear green. If the BIG-IP server objects are red or blue, you should verify the following:

  • The self IP addresses and iQuery port are reachable between F5 devices

    To verify that the self IP addresses and iQuery port are accessible between F5 devices, use the telnet command to open a connection on port 4353 from one F5 device to another. For example, to Telnet on port 4353 from the BIG-IP GTM system to the remote BIG-IP device, type the following command:

    telnet <remote_bigip_selfip> 4353

  • The SSL device certificates are valid

    F5 devices use SSL certificates for iQuery communication. If device certificates are missing or expired on an F5 device, iQuery communication will fail, and error messages that appear similar to the following example are logged to the /var/log/gtm file of the BIG-IP GTM system initiating the iQuery session:

    gtmd[8472]: 011ae020:5: Connection in progress to <iquery_peer>
    gtmd[8472]: 011ae01c:5: Connection complete to <iquery_peer>. Starting SSL handshake
    iqmgmt_ssl_connect: SSL error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

    Note: If iQuery communication fails due to missing or expired SSL device certificates, refer to SOL8187: Troubleshooting BIG-IP LTM and GTM device certificates.

Verify that the big3d process is broadcasting the virtual server and link data

Once the iQuery mesh is established, the gtmd process sends probes to the big3d process in the form of zlib compressed XML data blocks. Multiple block types are used. The big3d process compiles a list and broadcasts the response by sending the XML block to BIG-IP GTM systems that have an established iQuery connection with the big3d process.

If the remote BIG-IP devices are marked green by the BIG-IP GTM systems, but the BIG-IP GTM systems fail to automatically discover the virtual server and link objects, you can use the iqdump utility to verify that the BIG-IP system is properly broadcasting the virtual server and link objects.

For example, when you create a new virtual server configuration on the BIG-IP system, the mcpd process notifies big3d of the new configuration, and the big3d agent then broadcasts the virtual server information along with the complete list of virtual servers to all gtmd processes in the same sync group in a <getconfig> element block. You can verify that the big3d process is broadcasting the virtual server information by running the iqdump command from the BIG-IP GTM system. For example:

iqdump <remote_bigip_selfip>

The command output appears similar to the following example:

<getconfig_element>
   <type>2</type>
   <ip>::ffff:172.16.65.73</ip>
   <netmask>::</netmask>
   <port>8085</port>
   <flags>0</flags>
   <pool>vip6</pool>
</getconfig_element>
<getconfig_element>
   <type>2</type>
   <ip>::ffff:172.16.65.73</ip>
   <netmask>::</netmask>
   <port>8090</port>
   <flags>0</flags>
   <pool>vip10</pool>
</getconfig_element>

The <getconfig> element blocks display the virtual server IP address, service port, and name. If you verify that the big3d process is properly broadcasting the virtual server information, but the virtual servers do not appear in the BIG-IP GTM configuration, you may need to restart the big3d process. To do so, type the following command on the BIG-IP system:

bigstart restart big3d

Verify GTM sync group requirements

For BIG-IP GTM systems that are joined to a synchronization group, you should confirm that sync group members are communicating and properly synchronizing configuration data. You must meet several minimum requirements for BIG-IP GTM synchronization group members to communicate and synchronize properly. For information, refer to SOL13734: BIG-IP GTM synchronization group requirements.

Supplemental Information

Was this resource helpful in solving your issue?




NOTE: Please do not provide personal information.



Incorrect answer. Please try again: Please enter the words to the right: Please enter the numbers you hear:

Additional Comments (optional)