Description

The syslog-ng utility may cause high CPU utilization, resulting in a watchdog timeout and system reboot.

This issue occurs when the following conditions are met:

• The syslog-ng utility is configured to log to a remote syslog server using a TCP connection.
• The BIG-IP system is unable to communicate with the remote syslog server.

If there is a disruption in the communication with the remote syslog server, the syslog-ng utility will not attempt to re-establish the connection until the syslog-ng service is restarted.

Impact

The system may crash and traffic through the BIG-IP system may be disrupted.

Symptoms

As a result of this issue, you may encounter the following symptoms:

• The syslog-ng process shows CPU utilization greater than 90 percent using the following command:
  
  ps aux | grep syslog-ng
  
  For example:
  
  USER PID   %CPU %MEM  VSZ   RSS  TTY STAT  START  TIME COMMAND
  root 6394  0.93  0.0  3308  1508 ?   Ss    2012   0:02 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
  
• The system may experience a watchdog timeout and reboot.
• After a crash, the system displays an error message in the /var/log/ltm file that appears similar to the following example:
  
  warning chmand[5255]: 012a0004:4: Host CPU subsystem reset caused by *** Super I/O watchdog timeout ***

Status

F5 Product Development has assigned ID 387640 to this issue, and has confirmed that this issue exists in the products listed in the Applies To box. For information about releases or hotfixes that resolve this issue, refer to the following table:

Workaround

You can mitigate this issue by ensuring the remote syslog server is online and restarting the syslog-ng process by entering the following command:

bigstart restart syslog-ng

To work around this issue, you can remove the TCP remote server and use a UDP connection to transmit syslog data. To do so, perform the following procedure:

Impact of workaround: The UDP protocol does not guarantee the delivery of messages to the remote syslog server. UDP may not be a viable protocol for your environment.

Resetting the syslog remote server configuration to UDP (11.x)

1. Log in to the BIG-IP command line.
2. Type tmsh [enter].
3. List the configured syslog parameters with by entering the following command:
   
   list /sys syslog  
4. Note the include configuration.
   
5. Remove the include for the TCP syslog server by entering the following command:
   
   modify /sys syslog include none
   
   Note: This step removes all custom parameters defined in the include. You may need to re-add any additional parameters defined in your syslog include statement.
   
6. Re-add the remote server as a UDP server by entering the following command:
   
   modify /sys syslog remote-servers add {<server name> {host <server IP address> remote-port <port number>}}
   
   For example:
   
   modify /sys syslog remote-servers add {server{host 10.1.1.1 remote-port 514}}
7. Save the configuration by entering the following command:
   
   save /sys config
   

Resetting the syslog remote server configuration to UDP (10.x)

1. Log in to the BIG-IP command line.
2. List the configured syslog parameters by entering the following command:
   
   bigpipe syslog list all
   
3. Note the include configuration.
   
4. Remove the include for the TCP syslog server by entering the following command:
   
   Note: This procedure assumes that the bigpipe syslog include command was used to create the TCP remote server.
   
   bigpipe syslog include none
   
   Note: This step will remove all custom parameters defined in the include. You may need to re-add any additional parameters defined in your syslog include statement.
   
5. Re-add the remote server as a UDP server by entering the following command:
   
   bigpipe syslog remote-servers add {<server name> {host <server IP address> remote-port <port number>}}
   
   For example:
   
   bigpipe syslog remote-servers add {server{host 10.1.1.1 remote-port 514}}
6. Save the configuration by entering the following command:
   
   bigpipe save all

You can also mitigate this issue by ensuring the remote syslog server is online and restarting the syslog-ng process by entering the following command:

bigstart restart syslog-ng

Supplemental Information

• SOL4918: Overview of the F5 critical issue hotfix policy
• SOL167: Downloading software and firmware from F5
• SOL8259: The bigpipe syslog remote server command may only be used to add a single remote UDP syslog server
• SOL13317: Configuring the level of information that syslog-ng sends to log files (11.x)
• SOL5531: Configuring the level of information that syslog-ng sends to log files (9.x - 10.x)
• SOL13083: Configuring syslog settings from the command line (11.x)
• SOL8260: Configuring syslog settings from the command line (9.x - 10.x)