RTF Agent Upgrade status in Runtime Manager is stuck at "Upgrade in progress" and Cluster is degraded.

SYMPTOM

When clicking "Upgrade RTF Agent" in Runtime Manger, RTF Cluster, RTF cluster in Runtime Manager stuck at "Upgrade In progress" , but cluster degraded with errors/messages
- control-plane-agent-version-consistency
- Runtime Fabric agent version mismatch detected from control plane, please reach out to Mulesoft support if this persists. (a few seconds ago)
However, when logging (SSH) into RTF cluster, and check "rtfctl version", the agent upgraded successfully.
 

CAUSE

This inconsistency is due to sync issue between RTF Agent in RTF cluster to the control plane.

SOLUTION

This can be solved easily by running this below API call which force-sync the Agent upgrade between RTF to Control plane. 

curl -X PATCH -H 'Authorization: Bearer {bearer-token}' -H 'Content-Type: application/json;charset=UTF-8' -H 'Accept: application/json, text/plain, */*' \
https://anypoint.mulesoft.com/runtimefabric/api/organizations/{org-id}/fabrics/{rtf-id} --data-binary '{"version":"x.y.z"}'

Please replace

- {bearer-token} - with bearer token of your Anypoint Platform, which you can refer - https://help.mulesoft.com/s/article/How-to-Get-the-Anypoint-Access-Token-from-Chrome-Browser-Session
- {org-id} - your org id
- {rtf-id} - rtf id which i think you can get from your browser (anypoint platform > runtime manager > RTF > click on your RTF cluster) which shows the rtf id in the URL path.
- Also replace "x.y.z" with the desired RTF Agent version.  

Note: The message might not disappear on the first attempt. Try running the command again once or twice.