Loading

Troubleshooting MuleSoft Solace Connector Reconnection Issues

Publish Date: Sep 15, 2025
Description

Certain MuleSoft APIs fail to reconnect to the new broker pods despite using the same Solace GEL connector configurations with "retry forever" settings. Behaviour includes:

  • Some APIs reconnect successfully
  • Some APIs fail to reconnect, showing channel closure warnings/errors
  • Some APIs show no reconnection attempts at all

This inconsistent behavior occurs even though all APIs use the same global configuration and Solace GEL source listener retry strategy.

Common Observations: 

  • API logs show channel closed exceptions or warnings during failed reconnection attempts, but these alone may not pinpoint the root cause.
  • Solace broker logs and syslog forwarding may show only successful client binds from certain workers; failed workers do not appear connected.
  • Reconnection issues vary by worker instance, even with identical configuration.
  • Deadlocks related to Solace JCSMP library code have been observed in diagnostic thread dumps.
  • The Solace connector for MuleSoft is a partner-managed component maintained by Solace.

Errors seen:

SubChannel [...] threw exception, non-recoverable.
com.solacesystems.jcsmp.JCSMPTransportException: Channel is closed by peer

 

"[MuleRuntime].uber.21: [app_name].uber@org.mule.runtime.module.extension.internal.runtime.source.ExtensionMessageSource.lambda$reallyDoStart$17:488 @670ef9fb":
waiting to lock monitor 0x00007f46882fdeb0 (object 0x00000000ea7636d0, a com.solacesystems.jcsmp.impl.ContextImpl),
which is held by "[MuleRuntime].uber.20: [app_name].uber@org.mule.runtime.module.extension.internal.runtime.source.ExtensionMessageSource.lambda$reallyDoStart$17:488 @34eebe99"

Java stack information for the threads listed above:
===================================================
"[MuleRuntime].uber.20: [app_name].uber@org.mule.runtime.module.extension.internal.runtime.source.ExtensionMessageSource.lambda$reallyDoStart$17:488 @34eebe99":
at com.solacesystems.jcsmp.impl.JCSMPBasicSession.closeSession(JCSMPBasicSession.java:614)
- waiting to lock <0x00000000ea7357b8> (a com.solacesystems.jcsmp.impl.JCSMPBasicSession)
at com.solacesystems.jcsmp.impl.ContextImpl.destroy(ContextImpl.java:209)
- locked <0x00000000ea7636d0> (a com.solacesystems.jcsmp.impl.ContextImpl)
at com.solace.connector.mulesoft.internal.connection.SolaceConnection.invalidate(SolaceConnection.java:304)
at com.solace.connector.mulesoft.internal.source.ConsumerSource.cleanupResources(ConsumerSource.java:288)
at com.solace.connector.mulesoft.internal.source.ConsumerSource.onStop(ConsumerSource.java:229)
at org.mule.runtime.module.extension.internal.runtime.source.legacy.SdkSourceAdapter.onStop(org.mule.runtime.extensions.support@4.6.19/SdkSourceAdapter.java:58)
at org.mule.runtime.module.extension.internal.runtime.source.SourceAdapter.stop(org.mule.runtime.extensions.support@4.6.19/SourceAdapter.java:452)
at org.mule.runtime.module.extension.internal.runtime.source.ExtensionMessageSource.stopSource(org.mule.runtime.extensions.support@4.6.19/ExtensionMessageSource.java:329)
at org.mule.runtime.module.extension.internal.runtime.source.ExtensionMessageSource.restart(org.mule.runtime.extensions.support@4.6.19/ExtensionMessageSource.java:464)
- locked <0x00000000ea195618> (a java.util.concurrent.atomic.AtomicBoolean)
at org.mule.runtime.module.extension.internal.runtime.source.ExtensionMessageSource.lambda$onException$13(org.mule.runtime.extensions.support@4.6.19/ExtensionMessageSource.java:417)
at org.mule.runtime.module.extension.internal.runtime.source.ExtensionMessageSource$$Lambda$8179/0x00007f46c6209340.run(org.mule.runtime.extensions.support@4.6.19/Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.14/Executors.java:539)
at java.util.concurrent.FutureTask.run(java.base@17.0.14/FutureTask.java:264)
at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(org.mule.service.scheduler@1.6.19/AbstractRunnableFutureDecorator.java:180)
at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(org.mule.service.scheduler@1.6.19/RunnableFutureDecorator.java:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.14/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.14/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17.0.14/Thread.java:840)
[MuleRuntime].uber.21: [app_name].uber@org.mule.runtime.module.extension.internal.runtime.source.ExtensionMessageSource.lambda$reallyDoStart$17:488 @670ef9fb":
at com.solacesystems.jcsmp.impl.ContextImpl.removeSession(ContextImpl.java:166)
- waiting to lock <0x00000000ea7636d0> (a com.solacesystems.jcsmp.impl.ContextImpl)
at com.solacesystems.jcsmp.impl.JCSMPBasicSession.closeSession(JCSMPBasicSession.java:649)
- locked <0x00000000ea7357b8> (a com.solacesystems.jcsmp.impl.JCSMPBasicSession)
at com.solace.connector.mulesoft.internal.connection.SolaceConnection.invalidate(SolaceConnection.java:300)
at com.solace.connector.mulesoft.internal.source.ConsumerSource.cleanupResources(ConsumerSource.java:288)
at com.solace.connector.mulesoft.internal.source.ConsumerSource.onStop(ConsumerSource.java:229)
at org.mule.runtime.module.extension.internal.runtime.source.legacy.SdkSourceAdapter.onStop(org.mule.runtime.extensions.support@4.6.19/SdkSourceAdapter.java:58)
at org.mule.runtime.module.extension.internal.runtime.source.SourceAdapter.stop(org.mule.runtime.extensions.support@4.6.19/SourceAdapter.java:452)
at org.mule.runtime.module.extension.internal.runtime.source.ExtensionMessageSource.stopSource(org.mule.runtime.extensions.support@4.6.19/ExtensionMessageSource.java:329)
at org.mule.runtime.module.extension.internal.runtime.source.ExtensionMessageSource.restart(org.mule.runtime.extensions.support@4.6.19/ExtensionMessageSource.java:464)
- locked <0x00000000ea189f68> (a java.util.concurrent.atomic.AtomicBoolean)
at org.mule.runtime.module.extension.internal.runtime.source.ExtensionMessageSource.lambda$onException$13(org.mule.runtime.extensions.support@4.6.19/ExtensionMessageSource.java:417)
at org.mule.runtime.module.extension.internal.runtime.source.ExtensionMessageSource$$Lambda$8179/0x00007f46c6209340.run(org.mule.runtime.extensions.support@4.6.19/Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.14/Executors.java:539)
at java.util.concurrent.FutureTask.run(java.base@17.0.14/FutureTask.java:264)
at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(org.mule.service.scheduler@1.6.19/AbstractRunnableFutureDecorator.java:180)
at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(org.mule.service.scheduler@1.6.19/RunnableFutureDecorator.java:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.14/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.14/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17.0.14/Thread.java:840)
 
Resolution

1. Validate Configuration Consistency

  • Confirm that reconnection settings (solace.connect.retries or equivalent) are consistent across all API instances and workers.
  • Verify that retry logic is implemented at the Solace GEL listener level (not just globally for the connector).
  • Check that there are no conflicting retry or failover configurations between publishing and consuming connectors.

2. Review Logs and Enable Detailed Debugging

  • Collect and analyze MuleSoft API logs around the time of the failover for any warnings or errors.
  • Enable detailed debug logs on the Solace connector to capture connection lifecycle events. (com.solace.connector.mulesoft.api.param as DEBUG)
  • Review Solace broker logs and syslog forwarding for client bind/unbind events and errors.

3. Isolate Worker-specific Issues

  • Consider temporarily deploying affected APIs to a single MuleSoft worker to determine if the issue is worker-specific or global.
  • Observe if reconnection succeeds or fails in single-worker deployments.

4. Analyze Diagnostic Dumps for Deadlocks

  • Capture multiple thread dumps and heap dumps during failure periods.
  • Look for thread contention or deadlocks involving Solace JCSMP internal classes.
  • Deadlocks may indicate unresolved issues in the Solace connector or JCSMP library code.

5. Coordinate with Solace Support

  • Share diagnostic logs, thread dumps, and reconnection failure details with Solace Support.
  • Investigate if the Solace connector’s internal reconnect logic or use of JCSMP APIs is causing failures.
  • Discuss the impact of using MuleSoft’s retry framework versus JCSMP-level retry properties.
  • Solace Support may recommend adjustments to connector usage or provide fixes in the Solace library.

Key Insights and Recommendations: 

  • The Solace connector for MuleSoft relies on MuleSoft’s retry framework to manage reconnections at the GEL listener level.
  • MuleSoft’s retry framework may mark certain queue listeners as "fails deployment," potentially preventing automatic queue binding after broker failover.
  • Solace broker expects the client to explicitly start queue listeners after a restart; failure to do so can cause binding failures.
  • Solace connector exceptions like JCSMPTransportException: Channel is closed by peer may be non-recoverable without connector code changes.
  • Deadlocks detected in diagnostic dumps are typically within Solace’s JCSMP implementation, which MuleSoft does not own or control.
  • Creating a minimal, reproducible test case for such intermittent issues is challenging but valuable for engineering investigations.
  • Collaboration between MuleSoft support, Solace support, and customers is critical to identify the root cause and appropriate fixes.
Additional Resources

References:

Solace MuleSoft Connector Exchange Listing:
https://www.mulesoft.com/exchange/com.solace.connector/solace-mulesoft-connector/

Knowledge Article Number

005166929

 
Loading
Salesforce Help | Article