FAQ on Performance Issues with MuleSoft CloudHub

What is application sizing with Mule applications?

Application sizing is the thorough knowledge of the amount of resources that your application will need when deployed in production for a steady load. There are different ways to right-size your application. These tasks could include knowing the size of the objects you will keep in memory at a given time, amount of concurrent transactions, volume of transactions and also some known tests to validate if these limits are within the expected parameters, load tests, stress test, endurance test, etc.

CloudHub sizing

Sizing in CloudHub follows the same rules as any sizing, only that you need to be aware of the fixed hardware settings available and try to mimic your testing environment to CloudHub worker sizes. Be aware that even though you replicate heap, memory, core count and other CloudHub worker settings, it will likely not result in a 1:1 comparison. For instance, if you dedicate 1GB of memory to run your application, committing 500MB to heap size, in order to replicate a 0.1 vCore environment you may overlook than on the non-heap 500MB the CloudHub worker needs to run the OS, monitoring routines and other supporting software so results might defer from an actual CloudHub deployment if your testing environment reserves this 500MB only for the runtime and your application.
Another consideration here is that CPU on CloudHub workers has limited CPU credits available. As a result, the application working fine locally on a 4 CPU core laptop, starts struggling to process payload after some time deployed to CloudHub.

If you are unsure about the load your application can manage, and you are not getting the expected performance, you may want to consider increasing your worker size. If deploying to a larger worker makes the issue go away, it could be indicating that it was a sizing issue after all. If a larger worker just made the issue appear less frequently, then it might be either that you are facing a performance-impacting bug (see "What is a performance impacting bug?") or that you have not yet sized appropriately your application for the workload it has to handle.

What's application tuning with CloudHub?

It's the process of getting more bang for your buck. Application tuning involves profiling your application to understand where it spends most of the time. Then you should either review your design and/or tweak configuration settings. Reviewing application design bases on the premise that, though the application does what you want, it may not do it optimally, so redoing some parts of it in a different way can increase performance. Tweaking configuration settings implies setting values for things like thread pools and processing strategies so the same application code is more adapted to the actual workload it will be handling.

How can I begin to tune my application with Mule?

Mule 3

Mandatory first read is the performance tuning white paper. Then take a baseline of your application by profiling in Mule. If you feel comfortable with your current application design and think it's about tweaking time, then go to the documentation and read on performance tuning and processing strategies. Keep in mind application tuning is not a one-off but an iterative process. Application design is a very extensive topic, too. Be patient and test everything thoroughly.

Mule 4

On Mule 4 the Non-blocking Execution Engine should provide for self tuning of your application. You should just be mindful of using the Async block where appropriate. Identify first what are the bottlenecks in the application and the level of concurrency that your applications can sustain using load tests. Use the maxConcurrency attribute to limit the concurrency above values that can cause performance degradation. This becomes even more important when developing applications which utilize batch processing. Choosing the batch concurrency too high for a small, say 0.2 vCore worker, can easily lead to thread exhausting.
We don't recommend to change the thread pool configurations, as it requires knowledge of how the non-blocking implementation works and how threads from each pool are used. Also, you have to perform extensive performance tests to understand if a change affects performance in a positive or negative way.

What is a performance-impacting bug with CloudHub?

A performance-impacting bug is one that impacts directly in the performance of the application even though the application has passed through a sizing process and is under normal running conditions (normal load, healthy hardware, etc), but still you notice the application performance is deteriorating. Please note that running out of system resources (like CPU time, memory or disk space), having an out-of-memory error, an unresponsive application, a runtime restart, a too many open files error and the like do not constitute proof of a performance bug. These are usually sizing issues. Performance bugs exist when an error in MuleSoft code causes the system resources to be exhausted, and they always exhaust no matter how much available hardware, or don't show any improvement when more resources are available.

Performance bugs can be caused by different components, but from MuleSoft's Support perspective, only issues caused by MuleSoft's components or their libraries will be considered for analysis and eventually for a fix.

How should I proceed if I think I have a performance-impacting bug with MuleSoft?

The first step is to check our knowledge base. If the bug was widespread, a knowledge article about it should be written, for instance: Performance overhead found using APIKit router. If there's no specific article that applies to you, a second step could be considering updating. All MuleSoft products are constantly being patched as any bugs are discovered, performance related or not, so it makes no sense to stay on an outdated patch version. Update your Mule Runtime, CloudHub Runtime, Runtime Manager Agent, and Connectors as appropriate.

If the update did not resolve your problem, then you have to gather some information to diagnose this and prove it as a performance bug. Depending on the kind of performance problem, you should take and analyze different information.

Application stalled with Mule

You should experience that applications go into a halt state and that it never comes back from it. Taking a thread dump should reflect relevant threads to be locked or dead-locked. Take a series of thread dumps while the issue happens to double check this situation. Assert that the problem never self-heals given enough time.

How to obtain as much information as possible from an unresponsive Mule server

CPU time maxed out with Mule application

The clear symptom should be that your CPU goes to 100% and never comes down. You should be able to remove the workload, have the application do nothing, and yet the CPU is completely busy. If the CPU frees up when load goes down, then this is a sizing issue. If you have confirmed this to be a bug, then provide a set of thread dumps taken while the issue happens as well as the top output. If not running on Linux, refer to your OS vendor for a similar utility.

Memory maxed out with Mule application

When you are running out of memory with Mule applications and you believe this is a memory leak, you should be able to see that memory consumption goes up and never goes down again to initial levels. Particularly, you should see that removing the load on the applications does not cause the memory to ever go down. Take a heap dump by following the article below, analyze it and discover which instances are maxing out heap memory. If these are instances of classes packaged with the Mule Runtime, then send your information over for analysis on a support case.

How to obtain as much information as possible from an unresponsive Mule server

Other kind of issues with Mule applications

The above titles should cover most of the performance-impacting bugs with Mule. If you believe you are facing a different kind of issue or for some reason can't analyze it using the above recommended procedure, then take this approach. Most performance issues are caused by ‌undersized hardware or an application that requires some tuning, maybe a combination of both. There is a bug when there is a circumstance where, no matter how much hardware is provided, ‌performance is degraded. An example is a memory leak, when no matter how much memory is available, it eventually runs out. Another example could be a regular task that consumes a usual amount of resources that do not correspond to the task being performed. In any of these cases, just send your analysis and the data you have based on. Make sure that your sizing is correct before doing so and that you are running the latest versions of the software. We will review and get back to you.

How should I proceed if I think I'm having a performance or sizing issue with Mule applications?

Start by taking a good profiling snapshot of your application: Profiling Mule Applications.
This would be the baseline for where your application time is consumed. Do a load test using production-like load, do it for all the applications that will run concurrently, and this should also give you an idea of the amount of resources your application needs to run properly. Measure any change you make and also always repeat a load test on the final hardware once you think you have completed your sizing exercise.
To work through the tuning and sizing process, refer to our documentation and other publicly available resources. You can find the most common ones listed in the title above, "How can I begin to tune my application?". We also recommend that you engage in conversation through our forums. You also have the option of reaching out to your Customer Success Manager to inquire about a professional services engagement to have one of our specialists look at the problem for you.

See also Advanced Troubleshooting of Mule 4 Application Performance