Mule Runtime not starting with "Invalid cluster configuration: Node failed to start!"

SYMPTOM

In a cluster environment one of the nodes will fail to start with the following log snippet:

ERROR 2018-01-16 09:41:34,435 [WrapperListener_start_runner] org.mule.module.launcher.MuleContainer:
********************************************************************************
Message : Invalid cluster configuration: Node failed to start!
Element : org.mule.module.launcher.coreextension.DefaultMuleCoreExtensionManager@47c62177 @ app
--------------------------------------------------------------------------------
Exception stack is:
Invalid cluster configuration: Node failed to start! (org.mule.api.lifecycle.InitialisationException)
com.mulesoft.mule.cluster.hazelcast.HazelcastManager.createManager(HazelcastManager.java:139)
com.mulesoft.mule.cluster.hazelcast.HazelcastManager.create(HazelcastManager.java:147)
com.mulesoft.mule.cluster.boot.ClusterCoreExtension.createHazelcastManager(ClusterCoreExtension.java:105)

CAUSE

There could be several causes to this problem. We have identified the following:

• A Hazelcast library update in one of the Runtimes would cause the Runtime to fail communicating properly with the rest of the nodes. An example would be an update of libraries for this issue. 
• A network configuration change that prevents the existing node to communicate with the other nodes. An example would be restricting access to port 5701 which by default is used to establish connections to other ports, i.e 

  INFO  2018-02-20 19:06:58,085 [hz.2.cached.thread-7] com.hazelcast.nio.tcp.InitConnectionTask: [10.200.4.5]:5701 [3.8.1] Could not connect to: /10.200.4.7:5703. Reason: SocketException[Connection refused to address /10.200.4.7:5703]

SOLUTION

To resolve this issue, first check mule-cluster.properties for all the nodes in the cluster. After confirming everything looks correct proceed checking the latest changes:

• Hazelcast libraries updated
• Networking configurations. If not able to telnet to IP address and port (5701 by default) then work with infrastructure team to assist you resolve networking issues.