Original Publication Date: 01/04/2013
Updated Date: 12/16/2014
You should consider using this procedure under the following conditions:
The virtual server and link auto-discovery feature allows the BIG-IP GTM and BIG-IP Link Controller systems to automatically discover virtual servers and links that are associated with defined BIG-IP systems.
If auto-discovery is enabled for a server or link object, the system automatically discovers virtual servers and link objects as follows:
As a result of auto-discovery issues, you may encounter the following symptom:
When experiencing auto-discovery issues, you can use the following troubleshooting steps to determine the root cause:
Auto-discovery is unavailable for virtual servers using translated IP addresses. Before troubleshooting auto-discovery issues, confirm that the BIG-IP virtual servers do not use address translation. For example, if the target BIG-IP virtual server IP addresses reside in a private network space, as defined by RFC1918, and are mapped to public IP addresses that are defined on a network device such as a firewall, the BIG-IP GTM system will silently disable the auto-discovery feature for the BIG-IP system.
Note: For more information, refer to SOL9138: The BIG-IP GTM system disables virtual server auto-discovery for BIG-IP systems that use translated virtual server addresses.
The BIG-IP GTM system time must be synchronized with GTM sync group members (30 seconds by default), and should be synchronized with the remote BIG-IP systems in which virtual server / link auto-discovery is enabled. To verify time synchronization and NTP configuration, perform the following procedure:
Impact of procedure: Performing the following procedure should not have a negative impact on your system.
date; tmsh list /sys ntp
Note: To configure NTP on a BIG-IP system, refer to SOL3122: Using the BIG-IP Configuration utility to add an NTP server.
When the network topology consists of BIG-IP GTM systems load-balancing BIG-IP virtual servers, the systems must be able to communicate using the iQuery protocol. Upon starting, the gtmd process attempts to establish an SSL connection on port 4353 with the big3d process running on BIG-IP systems. The BIG-IP GTM system performs this action for all defined BIG-IP server objects, creating an iQuery mesh between gtmd and big3d processes. If there is a failure in the iQuery communication channel, gtmd will fail to automatically discover objects on the target BIG-IP system. One way to confirm that iQuery communication is functioning is to check the BIG-IP server status on all BIG-IP GTM devices. The BIG-IP server objects should appear green. If the BIG-IP server objects are red or blue, you should verify the following:
To verify that the self IP addresses and iQuery port are accessible between F5 devices, use the telnet command to open a connection on port 4353 from one F5 device to another. For example, to Telnet on port 4353 from the BIG-IP GTM system to the remote BIG-IP device, type the following command:
telnet <remote_bigip_selfip> 4353
F5 devices use SSL certificates for iQuery communication. If device certificates are missing or expired on an F5 device, iQuery communication will fail, and error messages that appear similar to the following example are logged to the /var/log/gtm file of the BIG-IP GTM system initiating the iQuery session:
gtmd: 011ae020:5: Connection in progress to <iquery_peer>
gtmd: 011ae01c:5: Connection complete to <iquery_peer>. Starting SSL handshake
iqmgmt_ssl_connect: SSL error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Note: If iQuery communication fails due to missing or expired SSL device certificates, refer to SOL8187: Overview of BIG-IP device certificates (9.x - 10.x).
Once the iQuery mesh is established, the gtmd process sends probes to the big3d process in the form of zlib compressed XML data blocks. Multiple block types are used. The big3d process compiles a list and broadcasts the response by sending the XML block to BIG-IP GTM systems that have an established iQuery connection with the big3d process.
If the remote BIG-IP devices are marked green by the BIG-IP GTM systems, but the BIG-IP GTM systems fail to automatically discover the virtual server and link objects, you can use the iqdump utility to verify that the BIG-IP system is properly broadcasting the virtual server and link objects.
For example, when you create a new virtual server configuration on the BIG-IP system, the mcpd process notifies big3d of the new configuration, and the big3d agent then broadcasts the virtual server information along with the complete list of virtual servers to all gtmd processes in the same sync group in a <getconfig> element block. You can verify that the big3d process is broadcasting the virtual server information by running the iqdump command from the BIG-IP GTM system.
The command output appears similar to the following example:
The <getconfig> element blocks display the virtual server IP address, service port, and name. If you verify that the big3d process is properly broadcasting the virtual server information, but the virtual servers do not appear in the BIG-IP GTM configuration, you may need to restart the big3d process. To do so, type the following command on the BIG-IP system:
bigstart restart big3d
For BIG-IP GTM systems that are joined to a synchronization group, you should confirm that sync group members are communicating and properly synchronizing configuration data. You must meet several minimum requirements for BIG-IP GTM synchronization group members to communicate and synchronize properly. For information, refer to SOL13734: BIG-IP GTM synchronization group requirements.