How to deal with OutOfMemory error on the Compressed Class metadata space on HotSpot JVM 1.8+

SYMPTOM

You're trying to deploy a new application on a mule standalone instance - or a mule cluster via MMC or ARM on a hybrid environment -.  While the applications are coming up, mule does not come completely up.  While looking at the mule logs,  you are able to identify the following error message:

20170326.034735.893 Su I - [onboarding].httpConnector.receiver.1360 CustomDefaultExceptionStrategyWithoutLogging SimpleMessage[message=Caught exception in Exception Strategy: Compressed class space (java.lang.OutOfMemoryError). Message payload is of type: String]
org.mule.api.MessagingException: Compressed class space (java.lang.OutOfMemoryError). Message payload is of type: String
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:32)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:108)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mu

When checking at the heap and O/S free memory, you can see that there is plenty of memory.

CAUSE

This is caused by the new class loading approach introduced with HotSpot 1.8.  Starting with 1.8, loaded classes are confined to a native space called the "Compressed Class Metadata Space" or CCMS. 
 
The default size for the CCMS is 1Gb of memory, except you specify a different value by passing an argument on the JVM command line.  When classes are being loaded by the classloader, it tries to allocate the metadata on CCMS.  If JVM cannot make room for the CCMS arena, it throws out the aforementioned error.

SOLUTION

To solve this issue, you must increase the CCMS arena size.  This can be done by adding the following argument on your $MULE_HOME/conf/wrapper.conf file, as shown below:

wrapper.java.additional.<n>=-XX:CompressedClassSpaceSize=2g

 
This requires a mule reboot for the change to take effect.

If even after having this change done, you still experience the same issue, then probably you're trying to deploy:

1) Big applications, which requires a lot of classes to be instantiated
2) Too many applications on the same mule instance
 
If the first case, then probably an application re-design will be needed.  Maybe the application could be divided into small pieces which could be deployed in a separate way.

In the second case, application deployments re-factor from the architectural PoV is required, where partitioning is the recommended approach.  Partitioning implies to divide your actual deployments into groups and have each group deployed to a specific Mule domain to alleviate the memory pressure on individual Mule instances.

This is usually implemented by scaling horizontally, creating new Mule domains, then having the applications grouped by nature, and then having them deployed to the newly created domains.