IBM MQ Connector | com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN')

SYMPTOM

The IBM MQ Connector intermittently throws the exception "MQRC_CONNECTION_BROKEN" while using the Mule 4 application. During the processing of a load, it is noticed that the MQ server reported a loss of messages. However, we were unable to find any trace of the message in the driver logs or application logs.

[02-27 20:08:28.120] INFO  IBMJmsCachingConnectionFactory [JMSCCThreadPoolWorker-8]: Handling exception in Mule's CompositeExceptionListener: {}
com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has occurred.
	at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:595) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.msg.client.wmq.internal.WMQMessageConsumer.checkJmqiCallSuccess(WMQMessageConsumer.java:217) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.msg.client.wmq.internal.WMQMessageConsumer.checkJmqiCallSuccess(WMQMessageConsumer.java:273) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.msg.client.wmq.internal.WMQAsyncConsumerShadow.consumer(WMQAsyncConsumerShadow.java:686) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.callConsumer(RemoteProxyQueue.java:3784) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.processEvent(RemoteProxyQueue.java:4930) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.driveConsumer(RemoteProxyQueue.java:4751) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.deliverMsgs(RemoteProxyQueue.java:5346) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.deliverMsgsReconnectable(RemoteDispatchThread.java:557) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.deliverMsgs(RemoteDispatchThread.java:531) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run(RemoteDispatchThread.java:305) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:319) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:99) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:343) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:312) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
	at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1240) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]
Caused by: com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
	at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203) ~[com.ibm.mq.allclient-9.3.0.1.jar:9.3.0.1 - p930-001-220907]

[02-27 20:08:28.139] ERROR DefaultSystemExceptionStrategy [JMSCCThreadPoolWorker-8]: 
********************************************************************************
Message               : JMSWMQ1107: A problem with this connection has occurred.
Element               : (None)
Element DSL           : (None)
Error type            : (None)
FlowStack             : (None)
--------------------------------------------------------------------------------
Root Exception stack trace:
com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
	at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
	at com.ibm.msg.client.wmq.internal.WMQMessageConsumer.checkJmqiCallSuccess(WMQMessageConsumer.java:217)
	at com.ibm.msg.client.wmq.internal.WMQMessageConsumer.checkJmqiCallSuccess(WMQMessageConsumer.java:273)
	at com.ibm.msg.client.wmq.internal.WMQAsyncConsumerShadow.consumer(WMQAsyncConsumerShadow.java:686)
	at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.callConsumer(RemoteProxyQueue.java:3784)
	at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.processEvent(RemoteProxyQueue.java:4930)
	at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.driveConsumer(RemoteProxyQueue.java:4751)
	at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.deliverMsgs(RemoteProxyQueue.java:5346)
	at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.deliverMsgsReconnectable(RemoteDispatchThread.java:557)
	at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.deliverMsgs(RemoteDispatchThread.java:531)
	at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run(RemoteDispatchThread.java:305)
	at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:319)
	at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:99)
	at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:343)
	at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:312)
	at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1240)

CAUSE

There are various factors that may lead to a broken connection. The return code, specifically 2009/MQRC_CONNECTION_BROKEN, suggests that there was a problem establishing a connection to the Queue Manager. The primary reasons for this issue include the firewall terminating the connection, configuration problems with the server Connection channel, etc.

SOLUTION

It seems that updating the Server Connection channel with sharecnv=10 and hbint=120 has resolved the connection failures with the IBM MQ Server. Tune the Heart Beat interval of the channel according to your environment and requirements.

Disclaimer: This article involves products and technologies which do not form part of the MuleSoft product set. Technical assistance for such products is limited to this article.