Limitations of using Object Store v2 (OSv2) in CloudHub applications

About This Knowledge Base Article

Object Store v2 lets CloudHub applications store data and states across multiple Mule components and different workers on the same application. However, based on the way the Object Store v2 is used by the different Mule components and connectors, you might run into some product limitations. This knowledge article explains such use cases and limitations. '

NOTE: This limitation applies to CloudHub applications, including both CloudHub 1.0 and CloudHub 2.0 (clustered or non-clustered).

SYMPTOMS

The symptoms and the scenarios for this issue can be broadly categorized as below,

Single Worker Scenario | Single Replica Scenario

You have a CloudHub application that uses Object Store v2, to share the data and states between Mule components. During high load on the app, you see that some of the direct (ObjectStore Connector/ObjectStore REST API) or indirect (Connectors like HTTP with OAuth, Salesforce connector, B2B Connectors, Aggregator Module, etc., that internally use Object Store v2) Object Store v2 calls are failing. 

Multi Worker Scenario | Multi Replica Scenario

You have a CloudHub application that uses Object Store v2, to share the data and states between Mule components and deployed into multiple workers. During high load on the app, you see that some of the direct (ObjectStore Connector/ObjectStore REST API) or indirect (Connectors like HTTP with OAuth, Salesforce connector, B2B Connectors, Aggregator Module, etc., that internally use Object Store v2) Object Store v2 calls are failing. 
 

Application Restart/Redeploy Scenario

You have a CloudHub application that uses Object Store v2, to share the data and states between Mule components and deployed into single or multiple workers/replicas. You are trying to restart/redeploy the application, you see some of the direct (ObjectStore Connector/ObjectStore REST API) or indirect (Connectors like HTTP with OAuth, Salesforce connector, B2B Connectors, Aggregator Module, etc.,  that internally use Object Store v2) Object Store v2 calls are failing. You see that the application is unable to use the state information from the Object Store v2 and continue from where it left before restart/redeployment.

CAUSE

The above-mentioned behavior or issues are due to the “Distributed Locking” feature not being available in CloudHub while working with Object Store v2. As a result, during certain conditions as mentioned in the SYMPTOMS section, when multiple Mule components try to update the Object Store v2 tables at the same time, such calls will fail.

Example:

Below are some of the Knowledge Articles related to some specific use-cases and examples where the issue is caused by the same limitation explained in this article.

HTTP Request Configuration using client credential authentication[OAuth Managed Token services] running with more than one worker on CloudHub
Aggregator Module using Object Store v2 running with more than one worker on CloudHub
"Object already exists for the key" Error with Salesforce Connector Events Listener in Multiple CloudHub Workers

SOLUTION

There is no reliable solution for this today. Please refer to the “recommendation” section below to avoid or mitigate the issue.

RECOMMENDATION

Single Worker Scenario | Single Replica Scenario

• Surround the call with “Until Successful Scope”. Please NOTE this will retry the operations that are inside the scope, regardless of whether the error is caused by Object Store v2 call or any other technical issue. Hence, it is also important to limit the “Max Retries” while using Until Successful Scope to avoid any performance impacts.
• Depending on your use-case and requirement, you can consider using an in-memory object store instead of Object Store v2. Refer: How to use ObjectStore V2 and Mule ObjectStore (In-Memory) objectstores in the same Mule 4 CloudHub Application This will avoid the limitation and also give better performance. Please NOTE, the data and states that are stored in the in-memory object store are lost when the application gets restarted/redeployed.

Multi Worker Scenario | Multi Replica Scenario 

• Consider vertical scaling. i.e., using a single but bigger worker. Please NOTE, before using a bigger worker, you want to make sure you are effectively using the vCore you have and not too much over sizing the application.
• Consider deploying the same application jar file into multiple applications with different application names. For example, deploy abc.jar into abc-application1 and abc-application2… abc-application-n, etc. This way the Object Store v2 data and states will survive application redeployment and restarts.
• Consider the recommendations from “Single Worker Scenario” section as well. 

Application Restart/Redeploy

• Consider stopping the application completely and then starting it. This way, the old worker instance will be able process all the pending events, release any lock on Object Store v2 tables and successfully shutdown before the new worker is started.
• Consider the recommendations from “Single Worker Scenario” and “Multi Worker Scenario” sections as well.

We encourage customers to upvote the enhancement request in Salesforce IdeaExchange:
https://ideas.salesforce.com/s/idea/a0B8W00000OTzPAUA1/objectstore-should-have-an-update-and-locking-mechanism-for-parallel-processing