java.lang.OutOfMemoryError: Metaspace

SYMPTOM

When deploying or redeploying one or many APIs, you run out of Java´s metaspace memory.

CAUSE

Every time you deploy an API, its metadata is loaded into a specific Java memory called Metaspace. 
(The metaspace memory was introduced in Java 8. Before that, it was called Java´s PermGen memory)

There are two kinds of causes:
1) Your metaspace memory sizing configuration is just too small
2) At least one of your APIs has a metaspace leak and fills your metaspace memory every time you redeploy it

SOLUTION

Cause # 1) To solve a wrong metaspace memory sizing, go to your $MULE_HOME/conf/wrapper.conf file and edit the following lines to adjust your needs: 

wrapper.java.additional.7=-XX:MaxMetaspaceSize=256m
wrapper.java.additional.8=-XX:MetaspaceSize=128m

(These are the default values for Mule 4.3.0 runtime but you should change them based on your sizing. If you need help with sizing, please contact your Customer Success Manager)

Ensure that the MetaspaceSize is always lower than the MaxMetaspaceSize. For more explanation on the reason, refer to https://help.salesforce.com/articleView?id=001118204&type=1&mode=1&language=en_US.

Cause # 2) Even though the metaspace memory is not readable, we can infer it by reading a heap dump. So there are two alternatives here:

2.A) To wait until it fails:

To do this, just configure your runtime to take an automatic heap dump upon an out-of-memory event, so even though it will fail, it will also create a file with the "hprof" extension that MuleSoft Support can help you analyze to locate the leaking APIs.

More info: How to gather heap dump on JVM out of memory error

2.B) To force the failure:

• We recommend doing this process in a lower environment and restarting the runtime before proceeding, to have a fresh metaspace memory
• For each API you want to analyze, redeploy it three times, and then wait five minutes before taking a manual heap dump (you need to wait that time for the Java´s garbage collector to run and remove non-leaked APIs)
• Once you have the heap dump, upload it to a MuleSoft Support case and let us know the list of APIs you have redeployed.
• During this procedure, you might accidentally fill the metaspace memory and never reach the manual heap dump step. Please reduce the amount of APIs you are analyzing and try again. Also, we recommend you configure the runtime (as explained in the other solution), for the automatic heap dump creation since that is also useful for the analysis. 

More info: How to Obtain as Much Information as Possible from an Unresponsive Mule Server

Important: We recommend redeploying at least three times since with fewer redeployments we can confuse regular class loading and garbage collection with leaks. So please, redeploy at least three times each API

TIP: It´s not necessary to redeploy using Runtime Manager since we know it could be time-consuming. Our recommendation is to go into the API folder at $MULE_HOME/apps/<api_name> and add an empty XML file or perform a small modification like changing a logger text. That will force the redeployment and it will be faster than using the Runtime Manager method.
 

ALTERNATIVE SOLUTION

An alternative to address a metaspace leak is to monitor your Java instance and restart the runtime before reaching the maximum metaspace size. This is not fixing the issue but you may avoid several outages until you get the root cause and solve it.

More info: How to monitor JVM native memory usage by enabling native memory tracking