Applies To:

Show Versions Show Versions

sol14096: The BIG-IP system may unexpectedly reboot when the syslog-ng utility uses TCP to a remote host
Known IssueKnown Issue

Original Publication Date: 01/07/2013
Updated Date: 06/05/2013

Description

The syslog-ng utility may cause high CPU utilization, resulting in a watchdog timeout and system reboot.

This issue occurs when the following conditions are met:

  • The syslog-ng utility is configured to log to a remote syslog server using a TCP connection.
  • The BIG-IP system is unable to communicate with the remote syslog server.

If there is a disruption in the communication with the remote syslog server, the syslog-ng utility will not attempt to re-establish the connection until the syslog-ng service is restarted.

Impact

The system may crash and traffic through the BIG-IP system may be disrupted.

Symptoms

As a result of this issue, you may encounter the following symptoms:

  • The syslog-ng process shows CPU utilization greater than 90 percent using the following command:

    ps aux | grep syslog-ng

    For example:

    USER PID   %CPU %MEM  VSZ   RSS  TTY STAT  START  TIME COMMAND
    root 6394  0.93  0.0  3308  1508 ?   Ss    2012   0:02 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid

  • The system may experience a watchdog timeout and reboot.
  • After a crash, the system displays an error message in the /var/log/ltm file that appears similar to the following example:

    warning chmand[5255]: 012a0004:4: Host CPU subsystem reset caused by *** Super I/O watchdog timeout ***

Status

F5 Product Development has assigned ID 387640 to this issue, and has confirmed that this issue exists in the products listed in the Applies To box. For information about releases or hotfixes that resolve this issue, refer to the following table:

Type of Fix
Versions Fixed
Related Articles
Release 11.3.0
SOL2200: Most recent versions of F5 software
Hotfix 11.2.1 HF6
11.2.0 HF6
10.2.4 HF7
SOL9502: BIG-IP hotfix matrix

 

Workaround

You can mitigate this issue by ensuring the remote syslog server is online and restarting the syslog-ng process by entering the following command:

bigstart restart syslog-ng

To work around this issue, you can remove the TCP remote server and use a UDP connection to transmit syslog data. To do so, perform the following procedure:

Impact of workaround: The UDP protocol does not guarantee the delivery of messages to the remote syslog server. UDP may not be a viable protocol for your environment.

Resetting the syslog remote server configuration to UDP (11.x)

  1. Log in to the BIG-IP command line.
  2. Type tmsh [enter].
  3. List the configured syslog parameters with by entering the following command:

    list /sys syslog  
  4. Note the include configuration.
  5. Remove the include for the TCP syslog server by entering the following command:

    modify /sys syslog include none

    Note: This step removes all custom parameters defined in the include. You may need to re-add any additional parameters defined in your syslog include statement.
  6. Re-add the remote server as a UDP server by entering the following command:

    modify /sys syslog remote-servers add {<server name> {host <server IP address> remote-port <port number>}}

    For example:

    modify /sys syslog remote-servers add {server{host 10.1.1.1 remote-port 514}}
  7. Save the configuration by entering the following command:

    save /sys config

Resetting the syslog remote server configuration to UDP (10.x)

  1. Log in to the BIG-IP command line.
  2. List the configured syslog parameters by entering the following command:

    bigpipe syslog list all
  3. Note the include configuration.
  4. Remove the include for the TCP syslog server by entering the following command:

    Note: This procedure assumes that the bigpipe syslog include command was used to create the TCP remote server.


    bigpipe syslog include none

    Note: This step will remove all custom parameters defined in the include. You may need to re-add any additional parameters defined in your syslog include statement.
  5. Re-add the remote server as a UDP server by entering the following command:

    bigpipe syslog remote-servers add {<server name> {host <server IP address> remote-port <port number>}}

    For example:

    bigpipe syslog remote-servers add {server{host 10.1.1.1 remote-port 514}}
  6. Save the configuration by entering the following command:

    bigpipe save all

You can also mitigate this issue by ensuring the remote syslog server is online and restarting the syslog-ng process by entering the following command:

bigstart restart syslog-ng

Supplemental Information

Was this resource helpful in solving your issue?




NOTE: Please do not provide personal information.



Incorrect answer. Please try again: Please enter the words to the right: Please enter the numbers you hear:

Additional Comments (optional)