How to diagnose com.esotericsoftware.kryo related exceptions in batch

Batch component will create a batch job, serialize it, store it to a persistent queue, then retrieve it later to process the content.

The serialization implementation used is a 3rd party library Kryo: https://github.com/EsotericSoftware/kryo. Kryo doesn't need object to implement java.io.Serializable interface and it is more efficient than the JDK implementation.

However, Kryo can NOT serialize/deserialize everything. There are cases the serialization works but deserialization fails with com.esotericsoftware.kryo related exceptions.

For example the following case:

com.mulesoft.module.batch.serialization.exception.SerializationException: Could not deserialize
at com.mulesoft.module.batch.serialization.kryo.KryoObjectSerializer.deserialize(KryoObjectSerializer.java:154)
at com.mulesoft.module.batch.engine.DefaultBatchJobInstanceStore.doRetrieve(DefaultBatchJobInstanceStore.java:373)
at com.mulesoft.module.batch.engine.DefaultBatchJobInstanceStore.getJobInstance(DefaultBatchJobInstanceStore.java:248)
at com.mulesoft.module.batch.engine.DefaultBatchEngine.refresh(DefaultBatchEngine.java:796)
at com.mulesoft.module.batch.engine.DefaultBatchEngine.updateStatistics(DefaultBatchEngine.java:766)
at com.mulesoft.module.batch.engine.DefaultBatchEngine.updateStatisticsAndRoute(DefaultBatchEngine.java:805)
at com.mulesoft.module.batch.engine.threading.BatchRecordWork.run(BatchRecordWork.java:93)
at org.mule.work.WorkerContext.run(WorkerContext.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: com.esotericsoftware.kryo.KryoException: java.lang.ClassCastException
Serialization trace:
authorities (org.springframework.security.core.userdetails.User)
principal (org.springframework.security.authentication.UsernamePasswordAuthenticationToken)
delegate (org.mule.module.spring.security.SpringAuthenticationAdapter)
authentication (org.mule.module.spring.security.SpringSecurityContext)
securityContext (org.mule.session.DefaultMuleSession)
session (org.mule.DefaultMuleEvent)
muleEvent (com.mulesoft.module.batch.DefaultBatchJobInstance)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at com.mulesoft.module.batch.serialization.kryo.MuleEventKryoSerializer.read(MuleEventKryoSerializer.java:38)
at com.mulesoft.module.batch.serialization.kryo.MuleEventKryoSerializer.read(MuleEventKryoSerializer.java:1)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at com.mulesoft.module.batch.serialization.kryo.KryoObjectSerializer.deserialize(KryoObjectSerializer.java:136)
... 10 more

For this one, we can see the stack trace has actually listed the serialization chain in Serialization trace:

Serialization trace:
authorities (org.springframework.security.core.userdetails.User)
principal (org.springframework.security.authentication.UsernamePasswordAuthenticationToken)
delegate (org.mule.module.spring.security.SpringAuthenticationAdapter)
authentication (org.mule.module.spring.security.SpringSecurityContext)
securityContext (org.mule.session.DefaultMuleSession)
session (org.mule.DefaultMuleEvent)
muleEvent (com.mulesoft.module.batch.DefaultBatchJobInstance)

This means the DefaultBatchJobInstance contains the mule session, which contains a securityContext, within the authentication object, the principal contains authorities, org.springframework.security.core.userdetails.User, which cannot be serialized/deserialized. As long as we identified the cause, we can easily remove the security context from session to workaround this issue.

However, there are also cases a Serialization trace is not available, for example:

com.mulesoft.module.batch.engine.threading.BatchRecordWork: Exception found while processing block 'be476876-31ff-11e5-9014-7ae357ecdd88' for instance 'bba21010-31ff-11e5-9014-7ae357ecdd88' of batch job 'myBatch'. Records will be queued back

java.lang.IllegalAccessError: tried to access class sun.nio.cs.UTF_8 from class sun.nio.cs.UTF_8ConstructorAccess

at sun.nio.cs.UTF_8ConstructorAccess.newInstance(Unknown Source) ~[kryo-shaded-3.0.0.jar:1.7.0_60]

at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy$1.newInstance(Kryo.java:1234) ~[kryo-shaded-3.0.0.jar:?]

at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1086) ~[kryo-shaded-3.0.0.jar:?]

at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) ~[kryo-shaded-3.0.0.jar:?]

at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) ~[kryo-shaded-3.0.0.jar:?]

at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:704) ~[kryo-shaded-3.0.0.jar:?]

at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) ~[kryo-shaded-3.0.0.jar:?]