Error Message

emerg logger: Re-starting big3d

For example:

Jun  3 13:23:28 gtmtest emerg logger: Re-starting big3d

Message Location

You may encounter this message in the following locations:

• The console
• The /var/log/daemon.log file
  

Description

This message occurs when any of the following conditions are met:

• The BIG-IP LTM or GTM system has determined that the big3d process is not running.
• The big3d process encounters an issue that requires a restart.

The big3d process runs on all BIG-IP systems, and provides metrics collection data on behalf of the BIG-IP GTM system using the iQuery protocol. BIG-IP GTM systems use the gathered metrics data to determine the availability state of that system and configured objects. In the event that the big3d process stops functioning on a particular F5 system, the system is unable to provide metrics information to other F5 devices, and the BIG-IP GTM system will not be able to determine that state of the device and configured objects. When this issue occurs, the GTM Statistics screen displays an Unknown (blue), or Offline (red) status for the device. If you notice that an F5 system is unable to provide metrics information to other F5 devices, log in to the command line for the device and check the log files for messages related to the big3d process.

Recommended Actions

If the big3d process restarts or is continually restarting unexpectedly, refer to the following actions to mitigate the issue:

Stop the big3d process

Upload a qkview to BIG-IP iHealth

Verify SSL certificate placement and operability

Place the big3d process in debug mode

Check for big3d core files

Stop the big3d process

If the big3d process is attempting to restart unexpectedly, the error message may continually output to the console. To prevent the big3d process from attempting to restart, run the following command from the BIG-IP command line:

• 10.x and later:
  
  tmsh stop /sys service big3d
  

• Versions prior to 10.x:
  
  bigstart stop big3d

Upload a qkview to BIG-IP iHealth

When troubleshooting big3d restart issues for 10.x and later, F5 recommends uploading a recent Qkview file from the affected BIG-IP GTM system to the F5 diagnostics site https://ihealth.f5.com. Access to the iHealth site requires a previous registration for the F5 Support site.

BIG-IP iHealth User Guide

BIG-IP iHealth Diagnostic Tool

If the iHealth analysis of the BIG-IP GTM Qkview file does not indicate known issues for big3d, you can proceed with the remaining procedures.

Verify SSL certificate placement and operability

When the big3d process does not start, or continually restarts on one of your F5 devices, it is commonly an issue related to the SSL device certificates. In order for authentication and communication between F5 devices to work properly, you must install the SSL certificates in the proper locations and the device certificate and key must be a matching pair.

When you run bigip_add or big3d_install from the local BIG-IP GTM system to a remote F5 system, the BIG-IP GTM system copies its device certificate (by default, /config/httpd/conf/ssl.crt/server.crt) to the remote F5 system's trusted device certificate file (/config/big3d/client.crt). The BIG-IP GTM system then copies the remote F5 system's device certificate to its trusted server certificate file (/config/gtm/server.crt).

Device certificates

All BIG-IP systems use a device certificate and key pair for inter-device authentication and communication. The F5 system uses the device certificate to authenticate login requests to the Configuration utility, and to identify itself to a requesting F5 client system. From the command line, the device certificate and key are contained in the following files:

• Device certificate:
  
  /config/httpd/conf/ssl.crt/server.crt
• Device key:
  
  /config/httpd/conf/ssl.key/server.key

Note: When the big3d process starts up, it searches the for key and certificate pair in the /config/httpd/conf/ssl.key/ and /config/httpd/conf/ssl.crt/ directories used for iQuery communication. If there are certificates in the /configs/httpd/conf/ssl.crt directory other than server.crt or keys in the /configs/httpd/conf/ssl.key directory other than server.key, big3d could select a mismatched pairing and the result will cause big3d communication to fail. There should only be one matching certificate and only one matching key stored within these directories.

Verifying that the device certificate and key are a matching pair

After you have verified the proper placement for the device certificate and key files on the F5 system, you should next verify that the certificate and key are a matching pair. To verify that an SSL certificate/key pair matches, refer to SOL6746: Verifying SSL certificates and keys from the command line.

If you determine that the device certificate and key are not a matching pair, you can generate a new certificate and key pair. To generate a new device certificate and key pair, refer to SOL9114: Creating an SSL device certificate and key pair using OpenSSL.

Trusted device certificate file

The big3d process of the local BIG-IP device uses the trusted device certificate file to authenticate a connection from a remote F5 device. From the command line, the location of the trusted device certificate file is as follows:

/config/big3d/client.crt

Trusted server certificate file

The local BIG-IP GTM system uses the trusted server certificate file to authenticate to a remote F5 device. From the command line, the location of the trusted server certificate file is as follows:

/config/gtm/server.crt

For more information about troubleshooting SSL device certificates, refer to SOL8187: Troubleshooting BIG-IP LTM and GTM device certificates.

If you identified and corrected a device certificate issue, you must now restart the big3d and httpd processes. To do so, enter the following command:

• 10.x and later:
  
  tmsh restart /sys service big3d httpd
• Versions prior to 10.x:
  
  bigstart restart big3d httpd

Place big3d in debug mode

If you have verified the SSL certificate placement on the F5 device, and the big3d process continues to restart, you can place the big3d process in debug mode for a short period of time, such as 10 to 15 minutes. After running the big3d process in debug mode, disable debug mode, then gather the system log files to send in to F5 Technical Support. To place the big3d process in debug mode, enter the following command:

• 10.x and later:
  
  tmsh modify  /sys db log.big3d.level value debug
• Versions prior to 10.x:
  
  bigpipe db Log.Big3d.Level debug

After you place the big3d process in debug mode for 10 to 15 minutes, you can set the big3d process back to the default log level. To disable debug mode and place the big3d process back in the default log level of notice, enter the following command:

• 10.x and later:
  
  tmsh modify  /sys db log.big3d.level value notice
• Versions prior to 10.x:
  
  bigpipe db Log.Big3d.Level notice

You can now gather the log files from the system. To gather the system log files, enter the following command:

tar -czpf /var/tmp/logfiles.tar.gz /var/log/*

For information about requirements for opening a BIG-IP support case, refer to SOL135: Information required when opening a support case for BIG-IP systems.

Check for big3d core files

In addition to gathering the system log files, you should check for possible big3d core files. To verify whether the big3d process has produced a core file, enter the following command:

ls -las /shared/core/

For information about submitting core files for analysis to F5 Technical Support, refer to SOL10062: Working with BIG-IP core files.

Supplemental Information

• For more information about the big3d process, refer to the Configuration Guide for BIG-IP Global Traffic Manager
• SOL12878: Generating BIG-IP diagnostic data using the qkview utility (10.x - 11.x)
• SOL2486: Providing files to F5 Technical Support