Original Publication Date: 02/28/2010
Updated Date: 03/01/2010
emerg logger: Re-starting big3d
Jun 3 13:23:28 gtmtest emerg logger: Re-starting big3d
You may encounter this message in the following locations:
This message occurs when any of the following conditions are met:
The big3d process runs on all BIG-IP systems, and provides metrics collection data on behalf of the BIG-IP GTM system using the iQuery protocol. BIG-IP GTM systems use the gathered metrics data to determine the availability state of that system and configured objects. In the event that the big3d process stops functioning on a particular F5 system, the system is unable to provide metrics information to other F5 devices, and the BIG-IP GTM system will not be able to determine that state of the device and configured objects. When this issue occurs, the GTM Statistics screen displays an Unknown (blue), or Offline (red) status for the device. If you notice that an F5 system is unable to provide metrics information to other F5 devices, log in to the command line for the device and check the log files for messages related to the big3d process.
If the big3d process restarts or is continually restarting unexpectedly, refer to the following actions to mitigate the issue:
If the big3d process is attempting to restart unexpectedly, the error message may continually output to the console. To prevent the big3d process from attempting to restart, run the following command from the BIG-IP command line:
When troubleshooting big3d restart issues for 10.x and later, F5 recommends uploading a recent Qkview file from the affected BIG-IP GTM system to the F5 diagnostics site https://ihealth.f5.com. Access to the iHealth site requires a previous registration for the F5 Support site.
If the iHealth analysis of the BIG-IP GTM Qkview file does not indicate known issues for big3d, you can proceed with the remaining procedures.
Verify SSL certificate placement and operability
When the big3d process does not start, or continually restarts on one of your F5 devices, it is commonly an issue related to the SSL device certificates. In order for authentication and communication between F5 devices to work properly, you must install the SSL certificates in the proper locations and the device certificate and key must be a matching pair.
When you run bigip_add or big3d_install from the local BIG-IP GTM system to a remote F5 system, the BIG-IP GTM system copies its device certificate (by default, /config/httpd/conf/ssl.crt/server.crt) to the remote F5 system's trusted device certificate file (/config/big3d/client.crt). The BIG-IP GTM system then copies the remote F5 system's device certificate to its trusted server certificate file (/config/gtm/server.crt).
All BIG-IP systems use a device certificate and key pair for inter-device authentication and communication. The F5 system uses the device certificate to authenticate login requests to the Configuration utility, and to identify itself to a requesting F5 client system. From the command line, the device certificate and key are contained in the following files:
Note: When the big3d process starts up, it searches the for key and certificate pair in the /config/httpd/conf/ssl.key/ and /config/httpd/conf/ssl.crt/ directories used for iQuery communication. If there are certificates in the /configs/httpd/conf/ssl.crt directory other than server.crt or keys in the /configs/httpd/conf/ssl.key directory other than server.key, big3d could select a mismatched pairing and the result will cause big3d communication to fail. There should only be one matching certificate and only one matching key stored within these directories.
Verifying that the device certificate and key are a matching pair
After you have verified the proper placement for the device certificate and key files on the F5 system, you should next verify that the certificate and key are a matching pair. To verify that an SSL certificate/key pair matches, refer to SOL6746: Verifying SSL certificates and keys from the command line.
If you determine that the device certificate and key are not a matching pair, you can generate a new certificate and key pair. To generate a new device certificate and key pair, refer to SOL9114: Creating an SSL device certificate and key pair using OpenSSL.
Trusted device certificate file
The big3d process of the local BIG-IP device uses the trusted device certificate file to authenticate a connection from a remote F5 device. From the command line, the location of the trusted device certificate file is as follows:
Trusted server certificate file
The local BIG-IP GTM system uses the trusted server certificate file to authenticate to a remote F5 device. From the command line, the location of the trusted server certificate file is as follows:
For more information about troubleshooting SSL device certificates, refer to SOL8187: Troubleshooting BIG-IP LTM and GTM device certificates.
If you identified and corrected a device certificate issue, you must now restart the big3d and httpd processes. To do so, enter the following command:
If you have verified the SSL certificate placement on the F5 device, and the big3d process continues to restart, you can place the big3d process in debug mode for a short period of time, such as 10 to 15 minutes. After running the big3d process in debug mode, disable debug mode, then gather the system log files to send in to F5 Technical Support. To place the big3d process in debug mode, enter the following command:
After you place the big3d process in debug mode for 10 to 15 minutes, you can set the big3d process back to the default log level. To disable debug mode and place the big3d process back in the default log level of notice, enter the following command:
You can now gather the log files from the system. To gather the system log files, enter the following command:
tar -czpf /var/tmp/logfiles.tar.gz /var/log/*
For information about requirements for opening a BIG-IP support case, refer to SOL135: Information required when opening a support case for BIG-IP systems.
Check for big3d core files
In addition to gathering the system log files, you should check for possible big3d core files. To verify whether the big3d process has produced a core file, enter the following command:
ls -las /shared/core/
For information about submitting core files for analysis to F5 Technical Support, refer to SOL10062: Working with BIG-IP core files.