MuleSoft How to Identify Leaked File Descriptors Shown Only as “can't identify protocol” in lsof

MuleSoft Mule Runtime

In Linux operating system every file or socket is associated with a file descriptor. Incorrectly closing these could cause a file descriptor leak. File descriptor information can be queried with the lsof (List Open Files) command.

Often this problem is noticed when opening new files or socket and receiving the "Too many open files" error but not necessarily the last file descriptor being opened is the root cause of this issue.

Example:

Sat Aug 8 12:15:49 CDT 2015

java 5556 myuser 661u IPv4 1593396777 0t0 TCP myserver:8080->server:59936 (CLOSE_WAIT)

File descriptors of sockets left in CLOSE_WAIT state eventually are ‘cleaned’ by Linux after a couple of hours, losing the information about the protocol and port. These are listed with the description "can't identify protocol". Once in this state is not possible to recover what was the original connection, making the analysis of the issue more difficult.

Example:

Sat Aug 8 14:14:49 CDT 2015

java 5556 myuser 661u sock 0,6 0t0 1593396777 can't identify protocol

Also take in account that eventually the user could run out of available file handles when trying to open new file/sockets. It would be incorrect to assume that the connector failing is the culprit as it could be running out of resources consumed by the original ‘leaking’ connector.