Loading

CloudHub | Recommended Log Appenders for Third-Party Log Aggregator Services

Publiceringsdatum: Aug 20, 2025
Beskrivning

Mule applications may suffer from performance issues due to logging.

It may not necessarily be an issue with apps having intensive logging. This could also be caused by one of the following reasons: 

  • A performance degradation at the third-party log aggregator service. E.g. API endpoint degradation, network slowness, etc.;
  • A use of an inefficient generic HTTP log appender;
  • A combination of the above.

If an application is not using a log4j2 HTTP appender, or logging is not intensive, the suggested approach would be to contact the support of the log aggregator directly. This might be an issue on the remote side. 

However, when using a HTTP appender, a mule app may suffer from performance issues; even everything else is excluded. 

A typical indication of the degradation caused by logging would be logging threads in a BLOCKED or TIMED_WAITING state: 

 

"[http-log4j2-appender].http.requester.hello_appender_HTTP_Request_configuration.01 SelectorRunner" #114 prio=5 os_prio=0 cpu=3038.15ms elapsed=721.07s tid=0x00007fc9901039b0 nid=0x38f waiting for monitor entry  [0x00007fc8db9f5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor.enqueue(org.apache.logging.log4j.core@2.20.0/AsyncLoggerConfigDisruptor.java:365)
    - waiting to lock <0x0000000702732d88> (a java.lang.Object)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor.enqueueEvent(org.apache.logging.log4j.core@2.20.0/AsyncLoggerConfigDisruptor.java:320)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfig.logInBackgroundThread(org.apache.logging.log4j.core@2.20.0/AsyncLoggerConfig.java:181)
    at org.apache.logging.log4j.core.async.EventRoute$1.logMessage(org.apache.logging.log4j.core@2.20.0/EventRoute.java:46)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfig.handleQueueFull(org.apache.logging.log4j.core@2.20.0/AsyncLoggerConfig.java:171)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfig.logToAsyncDelegate(org.apache.logging.log4j.core@2.20.0/AsyncLoggerConfig.java:158)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfig.log(org.apache.logging.log4j.core@2.20.0/AsyncLoggerConfig.java:138)
    at org.apache.logging.log4j.core.config.LoggerConfig.logParent(org.apache.logging.log4j.core@2.20.0/LoggerConfig.java:674)
    at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(org.apache.logging.log4j.core@2.20.0/LoggerConfig.java:643)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(org.apache.logging.log4j.core@2.20.0/LoggerConfig.java:624)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfig.log(org.apache.logging.log4j.core@2.20.0/AsyncLoggerConfig.java:143)
    at org.apache.logging.log4j.core.config.LoggerConfig.logParent(org.apache.logging.log4j.core@2.20.0/LoggerConfig.java:674)
    at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(org.apache.logging.log4j.core@2.20.0/LoggerConfig.java:643)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(org.apache.logging.log4j.core@2.20.0/LoggerConfig.java:624)
    at org.apache.logging.log4j.core.async.AsyncLoggerConfig.log(org.apache.logging.log4j.core@2.20.0/AsyncLoggerConfig.java:143)

To take a thread thread dump and confirm it, use one of the following documents articles: 

The reason it is happening often when the generic HTTP appender is used lies in its nature. Every log message is sent to a log aggregator service using a separate HTTP POST request. This is quite an expensive operation. If the logging is intensive and/or verbose, the number of these outbound requests may overwhelm the remote log aggregator API endpoint advertised sending logs there. Sooner or later, if the rate of messages generated by the mule app exceeds the rate of messages being sent by the appender to a third-party log aggregator,  per an interval of time, the RingBuffer (as explained in the article above) will be full. The logging will stop being asynchnous, i.e. it will become synchornous. Essentially, any mule app flow will be stopped for every mule logger in it waiting for the log message to be placed in the RungBuffer. As a result, Mule flow threads will be in BLOCKED/TIMED_WAITING state.

Lösning

To improve the application performance and mitigate the issue whenever possible, it it recommended to use proprietary log appenders developed by the respective vendors. In general, may still use HTTP protocol under the hood. However, they allow us to process logs in a more efficient way. For example, by sending several log messages in a single HTTP request, i.e as a batch. 

Below you can find the list of proprietary log appenders per vendor discovered by MuleSoft Support and recommended for use in Mule app.

Vendor

Recommended Log Appender Page

Notes

SplunkSplunk Logging for Java
  • Requires fine-tuning of batching parameters
  • Based on HTTP protocol
SplunkSocket Appender
  • Outstanding performance with minimum impact on the mule app performance
  • Requires Splunk instance to be configured with support for the socket listener
  • TLS authentication is a must to protect the data
NewrelicNew Relic Log4j2 Appender
  • Requires fine-tuning of batching parameters
  • Based on HTTP protocol
Sumo Logicsumologic-log4j2-appender
DatadogContact Datadog SupportJava Log Collection

If the proprietary log appender is not provided by the vendor, MuleSoft Support suggests to raise a support case with a respective vendor.

Ytterligare resurser

To integrate a log aggregator service for apps deployed to CloudHub, refer to the official custom log appender documentation page:

To integrate a log aggregator service for apps deployed on-premise, consider using proprietary log collection agents suggested by a respective vendor. A mule app would log to files and an agent would pick up files and forward the content to the log aggregator service

Note, MuleSoft Support may only provide ‌limited support by helping to troubleshoot connectivity issues when integrating with third-party appenders. For most of the questions, contact respective vendor.

 

Knowledge-artikelnummer

005131511

 
Laddar
Salesforce Help | Article