OutOfMemoryError: unable to create new native thread - How to prevent it

PROBLEM

The operating system is limiting the number of threads your Mule process can spawn, or the memory settings are wrong (setting more memory than the physically available memory).
The usage of memory in Java is determined by the following formula:

Max memory = [-Xmx] + [-XX:MaxPermSize] + number_of_threads * [-Xss] 

Usually the focus is in the Heap size determined by the JVM argument -Xmx and eventually, in Oracle JVMs, the Permanent Generation Space, specified by the argument -XX:MaxPermSize.

But you must also consider how many threads will your Mule instance handle, as it also adds to the max memory usage.
Suppose you have the following parameters:

• Max Heap size: 3072 MB
• Max PermGen size: 512 MB

If your system has in total 4GB RAM, then you would think that the 512 MB left to the operating system will cause no problem.
But the fact is that, for instance in a Linux 64 bits, every thread will take 1 MB of RAM from your system memory.
So if your Mule instance would have 512 threads (512 x 1MB = 512 MB), then you wouldn't have memory left to the operating system.
But usually that's never the case, as you will first see an OutOfMemoryError: Unable to create new native thread.

SYMPTOM

Exception in thread "throttling-task.13" java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:691)
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
    at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Exception in thread "[case8244].http.request.dispatch.8081.170" java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:691)
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
    at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Exception in thread "[case8244].http.request.dispatch.8081.25" java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:691)
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
    at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

SOLUTION

1. Take a look at our Performance Tuning Guide
2. Determine and adjust your memory settings accordingly, taking into consideration the formula provided above.
3. Review general Operating System limits and for the user running Mule, like file descriptors and max user processes and adjust them accordingly to the expected usage.
4. Load test, fine tune and load test again.

Note

These recommendation might not help if the application is configured to create too large a number of threads for this specific system, or it is actually leaking threads. In the former case further configuration tuning is recommended. In the latest you could use a thread dump and Linux ps -eLf command to analyze which threads are active.
The lsof command (lsof -p PID) could be used to verify if too many file descriptors are being created.
Further References
Plumbr - Unable to create new native thread
Plumbr - Why does my Java process consume more memory than Xmx?