Loading

How to deal with high file descriptor usage due to orphaned sockets (stale) on linux by tunning TCP keep-alive configuration

Publish Date: Mar 2, 2024
Resolution

SYMPTOM

You're experiencing high file descriptor usage on a linux machine which is running a mule instance.  Alternatively, the following error can appear on the mule logs, regarding file descriptor table becoming full for the current user, like the one listed below: 
2016-11-27 04:30:12,272 [amqpReceiver.1018] ERROR org.mule.exception.CatchMessagingExceptionStrategy - 
Message               : Error sending HTTP request. Message payload is of type: String
Code                  : MULE_ERROR--2

Exception stack is:
1. Too many open files (java.net.SocketException) sun.nio.ch.Net:-2 (null)
2. java.net.SocketException: Too many open files (java.util.concurrent.ExecutionException)
  org.glassfish.grizzly.impl.SafeFutureImpl$Sync:349 (null)
3. java.util.concurrent.ExecutionException: java.net.SocketException: Too many open files (java.io.IOException)
  org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:223 (null)
4. Error sending HTTP request. Message payload is of type: String (org.mule.api.MessagingException)
  org.mule.module.http.internal.request.DefaultHttpRequester:287 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html)

Root Exception stack trace:
java.net.SocketException: Too many open files
at sun.nio.ch.Net.socket0(Native Method)
at sun.nio.ch.Net.socket(Net.java:411)

CAUSE

As mule applications turns to be bigger, they tend to incorporate more receiver listener endpoints associated with the different protocols being involved in messaging:  HTTP, JMS, AMQP, and so on, along the different applications and flows involved on the orchestrations. Any of these endpoints will allocate a number of threads to listen to the corresponding connector's target.
 
Communication over TCP in mule is done purely in java, through the use of java sockets, which are basically reusable communication endpoints that consists of a logic representation at the session / presentation  level of a physical connection at a lower level ( physical, link, network, transport in terms of the OSI/ISO model).

Sockets can be connected/unconnected or stale (orphaned) sockets. Connected sockets is such a state where they have issued a SYN packet to connect to another remote endpoint, which again, is a socket also. Unconnected sockets are client sockets which can be actually not bound to any specific network interface and so, are available for further binding. Stale (orphaned) sockets are sockets that were connected in a past time, but for some reason, the connection was broken in a rough manner from one or another side.
 
Sockets can also turn into different states as long as they are transitioning from unconnected to connected, or viceversa. If for some reason, the transition cannot be completed, a socket can remain in an intermediate status for a long time.

A common example of such a condition are sockets in the CLOSE_WAIT state. Such state indicates that the remote endpoint has dropped the connection abruptly, without previous indication, and that the socket is waiting for the application that has open it, to close it accordingly.

The problem with such intermediate-state sockets is that they actually consume space to allocate a file descriptor on the operative system file-table space. The file-table has a capacity to accommodate file descriptors;  when such a capacity is exhausted, no more file descriptors can be allocated and the java process complains with the typical "too many open files" exception.  

Once such a condition is reached, no further processing is allowed; some file descriptors are still available, but the operative systems locks them for internal usage to avoid a panic condition. Application throughput decreases and there's a high chance of JVM to become frozen since the ClassLoader itself can be affected, which will not be able to load classes if needed.

SOLUTION

The recommended solution for the orphaned sockets, is to analyze at the deepest level possible, which condition is causing sockets to be abruptly closed and to fix. Such conditions can be one of the following:

- Connection being closed by the invoking client on inbound TCP connections, before the actual server response comes back.
- Firewall that is sitting in the middle between the client and the mule instance, which is dropping connections
- Load balancer which is forwarding traffic to the origin servers ( mule instances), whose TCP idle timeout is configured with a value which is lower than the maximum response time from the mule instance itself,  causing connection to be dropped before the actual response comes in.

Usually, a broken pipe indicating the connection was broken, will be thrown under this scenario,  although actually there's high chances that  the listening socket can become stale and not being closed by the application, or even, closed by the app but not finally freed-up at the OS layer, remaining in the CLOSE_WAIT state for a long time.
 
Any of these conditions should be analyzed to determine at which point the connection is being dropped. This apply both to incoming TCP connections ( client -> mule), and also to outgoing TCP connections ( mule -> legacy system / external provider).  

ALTERNATIVE SOLUTION
 
To avoid this situation -  and to have the file table under control, until the root cause for the dropped connections can be found -some tuning is necessary to basically "purge" unused sockets from the file table to free up space to accommodate new file descriptors. This is done through the  TCP keep-alive tuning, which is available on every operating system, including Linux. 

KeepAlive is a mechanism intended for detect  "Dead" connections due to peer gone conditions. From time to time, keep-alive sends a probe packet to each socket remote endpoint to see if some ACK response is sent back. If not, keep-alive marks the connection as "dead" and the operative system then is able to successfully "clean" the dead socket, freeing up space on the file table.
 
The keep-alive configuration is controlled through three main parameters in Linux:

- The TCP keep-alive time: which dictates how many time - in seconds - a socket would remain in an "idle" condition before the keep-alive connection check procedure kicks-in. Default value is 7200 seconds on the majority of the linux distributions.

- The TCP keep-alive interval: which dictates the interval between each keep-alive probe to detect if the peer is still there. Default value is 75 seconds between probes.
    
- The TCP keep-alive probes: how many probes should be tried before declaring the peer as gone and closing the socket.  Default number of probes is 9.

By default, linux comes with an OOB TCP keep-alive configuration as indicated. This configuration implies that a connection will be tested only after 7200 seconds (2 hours) has been elapsed since the socket was detected in an "idle" condition. An idle condition is such that no bytes are actually sent or received over the socket.

This is a very high value that can compromise the file table since this particular configuration takes too much time to get realized that one, many or thousand ( on a highly-loaded system) of connections have become "stale" (orphaned), before they can be disposed as expected. For such a reason, it is highly recommended to change this default configuration to a more sensible one, which can act as soon as possible.  

If keep-alive can be kicked-in soon, socket count can be reduced soon, and then, there will be little chances to get to a file table full condition , as explained earlier.

Following are the recommended initial values for the keep-alive probes. These values are just initial values and would be adjusted to fit adequately on each installation, having in mind that keep-alive probes also consume network bandwidth and so, the keep-alive interval should not fall below certain values that may be compromising the network overall performance. It is important to remark also that TCP keep-alive configuration affects to every single socket connection for every single process that can actually be running inside a system, so it is for sure a wide system network parameter to tune and keep an eye on:
 
ParameterRecommended Value
net.ipv4.tcp_keepalive_time
300 ( 5 minutes )
net.ipv4.tcp_keepalive_intvl
20
net.ipv4.tcp_keepalive_probes
3
 
If you're running mule over RHEL, it is recommended as a best practice, to add these values at the very end of the /etc/sysctl.conf file to make them persist across system reboots, just as indicated below: 
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_intvl = 20
net.ipv4.tcp_keepalive_probes = 3

Once done, issue the following command as the superuser (root), for the changes to take effect:
  
sysctl -p

This command will load and push the parameters from the persistent-parameter configuration file ( /etc/sysctl.conf ) into the kernel space.
After that, a whole server reboot is recommended since this parameter will affect to the whole system network interfaces and probably to processes that are loaded during the run-level initialization at the system-boot stage.
If a server reboot is not possible, for any reason,  the reboot can be postponed, but then, the mule instance should be reboot for the changes to take effect on mule. ( however, a full reboot will be needed when there is a chance to do it ).

CHECKING IF KEEP-ALIVE IS WORKING

After the recommended changes has been applied, you can check the current FD socket usage by means of the netstat command, as shown below: 
# netstat -tonp | grep CLOSE_WAIT
tcp       89      0 10.0.0.59:45598     10.0.0.12:1998       CLOSE_WAIT  -                   
tcp       15      0 10.0.0.59:60861     10.0.0.12:1938       CLOSE_WAIT  -                   
tcp        5      0 10.0.0.59:56173     10.0.0.12:1700       CLOSE_WAIT  -

This will give a listing of the orphaned connections in the CLOSE_WAIT state.  If CLOSE_WAIT count remains stable, problem is solved. If it continues to go up, then maybe further adjusting on the keep-alive will be needed as well.
 
Knowledge Article Number

001123765

 
Loading
Salesforce Help | Article