Mule Runtime On-Premises Servers Frequently Entering Connected/Disconnected State

Mule Runtime on-premises servers were frequently cycling between "Connected" and "Disconnected" states in Anypoint Runtime Manager over multiple days.

Environment

Applicable to:

* Mule Runtime Hybrid Deployments
* On-Premises Standalone Cluster Environments
* Runtime Manager Managed Servers

 Symptoms

The following symptoms were observed during the incident:

1. Mule runtime servers oscillated between "Connected" and "Disconnected" states, impacting cluster stability.

2. The Runtime Manager UI did not always update cleanly to "Disconnected" status. In some cases, servers remained stuck in an incorrect "Connected" state, preventing automated platform triggers from detecting the failure properly.

3. When a node experienced a brief network interruption, Hazelcast generated cluster membership exceptions and forcibly removed the node from the cluster. Once evicted, the node did not automatically rejoin the cluster.

 Error Indicators from Logs

The following log entries were commonly observed:

ERROR 2026-05-20 20:05:43,741 [WebSocketInboundExecutor] [processor: ; event: ] com.mulesoft.agent.transport.handlers.GenericWebSocketHandler: Closing Mule Agent WebSocket - Close Code: 1006 - Close Reason: Connection was closed abnormally (that is, with no close frame being received).

INFO 2026-05-20 20:05:43,741 [WebSocketInboundExecutor] [processor: ; event: ] com.mulesoft.agent.transport.handlers.GenericWebSocketHandler: Mule Agent WebSocket closed

INFO 2026-05-20 20:05:44,035 [pool-15-thread-1] [processor: ; event: ] com.mulesoft.agent.transport.connections.AsyncHttpWSConnectionThread: Initializing Mule Agent WebSocket connection

INFO 2026-05-20 20:05:44,035 [pool-15-thread-1] [processor: ; event: ] com.mulesoft.agent.transport.connections.AsyncHttpWSConnectionThread: Mule Agent WebSocket connecting to host: runtime-manager.anypoint.mulesoft.com

INFO 2026-05-20 20:05:57,396 [WebSocketInboundExecutor] [processor: ; event: ] com.mulesoft.agent.transport.handlers.GenericWebSocketHandler: Opening Mule Agent WebSocket

INFO 2026-05-20 20:05:57,396 [WebSocketInboundExecutor] [processor: ; event: ] com.mulesoft.agent.transport.handlers.GenericWebSocketHandler: Mule Agent WebSocket opened

INFO 2026-05-20 20:05:57,396 [pool-15-thread-1] [processor: ; event: ] com.mulesoft.agent.transport.connections.AsyncHttpWSConnectionThread: Mule Agent WebSocket connection was initialized after: 1 attempts

 Root Cause

The repeated reconnection events were triggered by a series of routine infrastructure and service update activities performed on MuleSoft-managed services.

* Existing WebSocket connections between Mule Runtime and MuleSoft control plane services were temporarily reset.
* Runtime agents attempted automatic reconnection as designed.

This behavior was identified as:

* A temporary infrastructure alignment activity
* Not a product defect
* Not caused by customer-side configuration issues

No recurring impact is expected following completion of the maintenance activities.

Expected Behavior

The following behavior is considered expected during standard infrastructure updates or platform service maintenance windows:

* Short-lived reconnection windows lasting a few minutes
* Temporary Connected/Disconnected state transitions
* Automatic runtime reconnection within approximately one minute

 Important Note

If a server remains stuck in a "Connected" state for more than 5 minutes, verify whether a manual restart was initiated before the runtime completed its automatic recovery process.

Manual restarts during transient reconnect windows can interrupt normal recovery behavior.