Applies To:

Show Versions Show Versions

sol14106: Troubleshooting virtual server and link auto-discovery (11.x - 12.x)

Original Publication Date: 01/04/2013
Updated Date: 09/27/2015


You should consider using this procedure under the following conditions:

  • You have configured virtual server or link auto-discovery.
  • The BIG-IP DNS system failed to automatically discover virtual servers or link objects that are associated with defined BIG-IP systems.


The virtual server and link auto-discovery feature allows the BIG-IP DNS and BIG-IP Link Controller systems to automatically discover virtual servers and links that are associated with defined BIG-IP systems.

If auto-discovery is enabled for a server or link object, the system automatically discovers virtual servers and link objects as follows:

  • The gtmd process running on the local BIG-IP DNS system reads the auto-discovery settings from the /config/bigip_gtm.conf file.
  • At the defined auto-discovery request interval, gtmd sends the <getconfig> request element over iQuery (SSL, port 4353) to the big3d process on the remote BIG-IP system.
  • The receiving big3d process returns any defined virtual server and link information in the <getconfig_elements> iQuery block data.
  • The local gtmd process then adds the discovered virtual servers and links to the BIG-IP DNS configuration.


As a result of auto-discovery issues, you may encounter the following symptom:

  • The BIG-IP DNS system fails to automatically discover the virtual servers and links associated with defined BIG-IP systems.


When experiencing auto-discovery issues, you can use the following troubleshooting steps to determine the root cause:

Confirm that the BIG-IP virtual servers do not use address translation

Auto-discovery is unavailable for virtual servers using translated IP addresses. Before troubleshooting auto-discovery issues, confirm that the BIG-IP virtual servers do not use address translation. For example, if the target BIG-IP virtual server IP addresses reside in a private network space, as defined by RFC1918, and are mapped to public IP addresses that are defined on a network device such as a firewall, the BIG-IP DNS system will silently disable the auto-discovery feature for the BIG-IP system.

Note: For more information, refer to SOL9138: The BIG-IP GTM system disables virtual server auto-discovery for BIG-IP systems that use translated virtual server addresses.

Verify time synchronization

The BIG-IP DNS system time must be synchronized across all sync group members (30 seconds by default), and should be synchronized with the remote BIG-IP systems in which virtual server / link auto-discovery is enabled. To verify time synchronization and NTP configuration, perform the following procedure:

Impact of procedure: Performing the following procedure should not have a negative impact on your system.

  1. Log in to the command line of the F5 device.
  2. Type the following command:

    date; tmsh list /sys ntp

  3. Perform steps 1 through 2 on all devices.
  4. Verify that each device is configured to use an NTP server and that the times are in sync.

Note: To configure NTP on a BIG-IP system, refer to SOL3122: Using the BIG-IP Configuration utility to add an NTP server.

Confirm iQuery communication

When the network topology consists of BIG-IP DNS systems load-balancing BIG-IP virtual servers, the systems must be able to communicate using the iQuery protocol. Upon starting, the gtmd process attempts to establish an SSL connection on port 4353 with the big3d process running on BIG-IP systems. The BIG-IP DNS system performs this action for all defined BIG-IP server objects, creating an iQuery mesh between gtmd and big3d processes. If there is a failure in the iQuery communication channel, gtmd will fail to automatically discover objects on the target BIG-IP system. One way to confirm that iQuery communication is functioning is to check the BIG-IP server status on all BIG-IP DNS devices. The BIG-IP server objects should appear green. If the BIG-IP server objects are red or blue, you should verify the following:

  • The self IP addresses and iQuery port are reachable between F5 devices

    To verify that the self IP addresses and iQuery port are accessible between F5 devices, use the telnet command to open a connection on port 4353 from one F5 device to another. For example, to Telnet on port 4353 from the BIG-IP DNS system to the remote BIG-IP device, type the following command:

    telnet <remote_bigip_selfip> 4353

  • The SSL device certificates are valid

    F5 devices use SSL certificates for iQuery communication. If device certificates are missing or expired on an F5 device, iQuery communication will fail, and error messages that appear similar to the following example are logged to the /var/log/gtm file of the BIG-IP DNS system initiating the iQuery session:

    gtmd[8472]: 011ae020:5: Connection in progress to <iquery_peer>
    gtmd[8472]: 011ae01c:5: Connection complete to <iquery_peer>. Starting SSL handshake
    iqmgmt_ssl_connect: SSL error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

    Note: If iQuery communication fails due to missing or expired SSL device certificates, refer to SOL8187: Overview of BIG-IP device certificates (9.x - 10.x).

Verify that the big3d process is broadcasting the virtual server and link data

Once the iQuery mesh is established, the gtmd process sends probes to the big3d process in the form of zlib compressed XML data blocks. Multiple block types are used. The big3d process compiles a list and broadcasts the response by sending the XML block to BIG-IP DNS systems that have an established iQuery connection with the big3d process.

If the remote BIG-IP devices are marked green by the BIG-IP DNS systems, but the BIG-IP DNS systems fail to automatically discover the virtual server and link objects, you can use the iqdump utility to verify that the BIG-IP system is properly broadcasting the virtual server and link objects.

For example, when you create a new virtual server configuration on the BIG-IP system, the mcpd process notifies big3d of the new configuration, and the big3d agent then broadcasts the virtual server information along with the complete list of virtual servers to all gtmd processes in the same sync group in a <getconfig> element block. You can verify that the big3d process is broadcasting the virtual server information by running the iqdump command from the BIG-IP DNS system.

For example:

iqdump <remote_bigip_selfip>

The command output appears similar to the following example:


The <getconfig> element blocks display the virtual server IP address, service port, and name. If you verify that the big3d process is properly broadcasting the virtual server information, but the virtual servers do not appear in the BIG-IP DNS configuration, you may need to restart the big3d process. To do so, type the following command on the BIG-IP system:

bigstart restart big3d

Verify DNS sync group requirements

For BIG-IP DNS systems that are joined to a synchronization group, you should confirm that sync group members are communicating and properly synchronizing configuration data. You must meet several minimum requirements for BIG-IP DNS synchronization group members to communicate and synchronize properly. For information, refer to SOL13734: BIG-IP GTM synchronization group requirements.

Supplemental Information

Was this resource helpful in solving your issue?

NOTE: Please do not provide personal information.

Incorrect answer. Please try again: Please enter the words to the right: Please enter the numbers you hear:

Additional Comments (optional)