HTTP Request Configuration using client credential authentication[OAuth Managed Token services] running with more than one worker on CloudHub

SYMPTOMS

• The application would initially face "NullPointerException" during a token refresh and then would not process any requests received for that Request configuration eventually leaving the clients to close a connection causing 499 response code in the load balancer logs.

  java.util.concurrent.ExecutionException: java.lang.NullPointerException
  	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_265]
  	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_265]
  	at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(AbstractRunnableFutureDecorator.java:118) [mule-service-scheduler-1.3.2/:?]
  	at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(RunnableFutureDecorator.java:54) [mule-service-scheduler-1.3.2/:?]
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265]
  	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
  Caused by: java.lang.NullPointerException
  	at org.mule.service.oauth.internal.AbstractOAuthDancer.lambda$pollForRefreshComplete$0(AbstractOAuthDancer.java:304) ~[?:?]
  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_265]
  	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_265]
  	at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(AbstractRunnableFutureDecorator.java:111) ~[?:?]
  	... 4 more

• The thread dump could have a thread-id on which the events would be queued to be processed. The stack of the thread would be like below.

  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  	at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
  	at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
  	at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
  	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
  	at org.mule.extension.oauth2.api.clientcredentials.ClientCredentialsGrantType.authenticate(ClientCredentialsGrantType.java:68)
  	at org.mule.extension.http.internal.request.HttpRequestFactory.create(HttpRequestFactory.java:162)
  	at org.mule.extension.http.internal.request.HttpRequester.doRequest(HttpRequester.java:122)
  	at org.mule.extension.http.internal.request.HttpRequestOperations.request(HttpRequestOperations.java:139)
  	at org.mule.extension.http.internal.request.HttpRequestOperations$request$MethodComponentExecutor.execute(Unknown Source)
  	at org.mule.runtime.module.extension.internal.runtime.execution.GeneratedMethodComponentExecutor.execute(GeneratedMethodComponentExecutor.java:98)
  	at org.mule.runtime.module.extension.internal.runtime.execution.NonBlockingCompletableMethodOperationExecutor.doExecute(NonBlockingCompletableMethodOperationExecutor.java:38)
  	at org.mule.runtime.module.extension.internal.runtime.execution.AbstractCompletableMethodOperationExecutor.execute(AbstractCompletableMethodOperationExecutor.java:63)
  	at org.mule.runtime.module.extension.internal.runtime.operation.DefaultExecutionMediator.executeCommand(DefaultExecutionMediator.java:212)
  	at org.mule.runtime.module.extension.internal.runtime.operation.DefaultExecutionMediator.executeWithInterceptors(DefaultExecutionMediator.java:195)
  	at org.mule.runtime.module.extension.internal.runtime.operation.DefaultExecutionMediator.lambda$execute$1(DefaultExecutionMediator.java:114)
  	at org.mule.runtime.module.extension.internal.runtime.operation.DefaultExecutionMediator$$Lambda$4983/707734186.process(Unknown Source)
  	at org.mule.runtime.module.extension.internal.runtime.operation.DefaultExecutionMediator.lambda$new$0(DefaultExecutionMediator.java:62)
  	at org.mule.runtime.module.extension.internal.runtime.operation.DefaultExecutionMediator$$Lambda$2874/887469770.execute(Unknown Source)
  	at org.mule.runtime.module.extension.internal.runtime.operation.DefaultExecutionMediator.execute(DefaultExecutionMediator.java:113)
  	at org.mule.runtime.module.extension.internal.runtime.operation.ComponentMessageProcessor.executeOperation(ComponentMessageProcessor.java:570)
  	at org.mule.runtime.module.extension.internal.runtime.operation.ComponentMessageProcessor.prepareAndExecuteOperation(ComponentMessageProcessor.java:778)
  	at org.mule.runtime.module.extension.internal.runtime.operation.ComponentMessageProcessor.access$100(ComponentMessageProcessor.java:189)
  	at org.mule.runtime.module.extension.internal.runtime.operation.ComponentMessageProcessor$4.lambda$innerEventDispatcher$9(ComponentMessageProcessor.java:644)
  	at org.mule.runtime.module.extension.internal.runtime.operation.ComponentMessageProcessor$4$$Lambda$4556/1166282145.accept(Unknown Source)
  	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onNext(FluxPeekFuseable.java:482)
  	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onNext(FluxPeekFuseable.java:496)
  	at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:103)
  	at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:793)
  	at reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:718)
  	at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:153)
  	at org.mule.runtime.module.extension.internal.runtime.operation.ComponentMessageProcessor.lambda$onEvent$16(ComponentMessageProcessor.java:438)
  	at org.mule.runtime.module.extension.internal.runtime.operation.ComponentMessageProcessor$$Lambda$4980/438356769.execute(Unknown Source)
  	at org.mule.runtime.core.internal.policy.DefaultPolicyManager.lambda$static$0(DefaultPolicyManager.java:84)
  	at org.mule.runtime.core.internal.policy.DefaultPolicyManager$$Lambda$1543/702373680.process(Unknown Source)
  	at org.mule.runtime.module.extension.internal.runtime.operation.ComponentMessageProcessor.onEvent(ComponentMessageProcessor.java:449)
  	at org.mule.runtime.module.extension.internal.runtime.operation.ComponentMessageProcessor.lambda$null$9(ComponentMessageProcessor.java:389)
  	at org.mule.runtime.module.extension.internal.runtime.operation.ComponentMessageProcessor$$Lambda$4639/860288318.accept(Unknown Source)
  	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onNext(FluxPeekFuseable.java:482)
  	at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287)
  	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onNext(FluxPeekFuseable.java:496)
  	at org.mule.runtime.core.privileged.processor.chain.AbstractMessageProcessorChain$2.onNext(AbstractMessageProcessorChain.java:430)
  	at org.mule.runtime.core.privileged.processor.chain.AbstractMessageProcessorChain$2.onNext(AbstractMessageProcessorChain.java:425)
  	at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:127)
  	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:204)
  	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:345)
  	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:204)
  	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:204)
  	at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:398)
  	at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:484)
  	at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
  	at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  	at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(AbstractRunnableFutureDecorator.java:111)
  	at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(RunnableFutureDecorator.java:54)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)

• The struck thread can also be with the below structure

  Root Exception stack trace:
  java.lang.NullPointerException
  at org.mule.service.oauth.internal.AbstractOAuthDancer.doRefreshToken(AbstractOAuthDancer.java:227)
  at org.mule.service.oauth.internal.DefaultClientCredentialsOAuthDancer.doRefreshTokenRequest(DefaultClientCredentialsOAuthDancer.java:120)
  at org.mule.service.oauth.internal.DefaultClientCredentialsOAuthDancer.refreshToken(DefaultClientCredentialsOAuthDancer.java:116)
  at org.mule.extension.oauth2.api.clientcredentials.ClientCredentialsGrantType.retryIfShould(ClientCredentialsGrantType.java:98)
  at org.mule.extension.http.internal.request.HttpRequester.resendRequest(HttpRequester.java:251)
  at org.mule.extension.http.internal.request.HttpRequester.lambda$doRequestWithRetry$5(HttpRequester.java:154)
  at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
  at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
  at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
  at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
  at com.mulesoft.service.http.impl.service.client.EEGrizzlyHttpClient.lambda$sendAsync$1(EEGrizzlyHttpClient.java:145)
  at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
  at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
  at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
  at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
  at org.mule.service.http.impl.service.client.async.ResponseAsyncHandler.onCompleted(ResponseAsyncHandler.java:47)
  at org.mule.service.http.impl.service.client.async.ResponseAsyncHandler.onCompleted(ResponseAsyncHandler.java:29)
  at com.ning.http.client.AsyncCompletionHandler.onCompleted(AsyncCompletionHandler.java:55)
  at com.ning.http.client.providers.grizzly.AhcEventFilter.onHttpPacketParsed(AhcEventFilter.java:433)
  at org.glassfish.grizzly.http.HttpCodecFilter.decodeWithTransferEncoding(HttpCodecFilter.java:1420)
  at org.glassfish.grizzly.http.HttpCodecFilter.handleRead(HttpCodecFilter.java:677)
  at org.glassfish.grizzly.http.HttpClientFilter.handleRead(HttpClientFilter.java:182)
  at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
  at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
  at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
  at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
  at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
  at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
  at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:540)
  at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
  at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103)
  at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
  at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415)
  at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384)
  at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348)
  at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(AbstractRunnableFutureDecorator.java:111)
  at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(RunnableFutureDecorator.java:54)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

• Below errors can be seen for CH2 apps deployed with cluster.

2025-03-21T03:06:12.391Z ERROR [fb46x] LifecycleAwareConfigurationInstance event:6b55874c-938b-48f0-aa46-7d5689714f5e [MuleRuntime].uber.09: [oauth-http-concurrent].ouathhttpFlow.CPU_LITE @7a6ef80b - Error initializing LifecycleAwareConfigurationInstance 'HTTP_Request_configuration2'
java.util.ConcurrentModificationException: null
    at java.util.HashMap.computeIfAbsent(HashMap.java:1221) ~[?:?]
    at com.mulesoft.objectstore.cloud.api.PartitioningObjectStoreManager.createObjectStore(PartitioningObjectStoreManager.java:108) ~[?:?]
    at com.mulesoft.objectstore.cloud.api.PartitioningObjectStoreManager.lambda$getOrCreateObjectStore$1(PartitioningObjectStoreManager.java:82) ~[?:?]
    at java.util.Optional.orElseGet(Optional.java:364) ~[?:?]
    at com.mulesoft.objectstore.cloud.api.PartitioningObjectStoreManager.getOrCreateObjectStore(PartitioningObjectStoreManager.java:82) ~[?:?]
    at org.mule.extension.oauth2.api.tokenmanager.TokenManagerConfig.initialise(TokenManagerConfig.java:127) ~[mule-oauth-module-1.1.21-mule-plugin.jar:?]

CAUSE

In OAuth token management, the lock provider must remain consistent across workers. However, this feature is not supported when using Persistent Object Store V2 (OSv2).
For example, when a CloudHub 2 (CH2) application is deployed with cluster mode enabled, replicas share certain resources. This can lead to scenarios where the same shared resources are accessed or modified by multiple requests simultaneously, potentially causing errors like “java.util.ConcurrentModificationException: null

Due to this limitation, applications leveraging OSv2 may be impacted, specifically:
-Applications deployed in CloudHub 1 that utilize multiple workers.
-Applications deployed in CloudHub 2 with clustering enabled.

ALTERNATE SOLUTION

1. Create a non-persistent Object Store and configure the HTTP request configuration module to use that to bypass the limitation in scenarios of multiple workers in the CloudHub environment. Check the following articles to get the details on how to configure an in-memory object store and important considerations when using In-Memory Object Store for Managing tokens
   How to configure In-Memory Object Store for Managing tokens in an HTTP Request configuration
   How to use ObjectStore V2 and Mule ObjectStore (In-Memory) objectstores in the same Mule 4 CloudHub Application Note: Using an In-Memory object store would mean that the information would be lost upon restart of the applications. In addition, the data is stored in the memory. Make sure the application is tested for the mentioned considerations as well
2. For Ch2 app, deploy it as non-clustered application