Failed to obtain information for API xxx and Error activating API xxx Error/Warning messages

SYMPTOM

User gets intermittent 403, 502, 503, or 504 retrieving API information or activating contracts from API in application log similar to below:

Example 1)
 

[2021-11-21 00:00:55.795] WARN com.mulesoft.mule.runtime.gw.deployment.platform.interaction.clients.PlatformClientsRetriever [agw-contract-polling.01]: Failed to obtain contracts for API 'API my-sapi-v1-v1:12345678 (12345678)'. This request will be retried after some backoff time. Reason: com.mulesoft.mule.runtime.gw.client.exception.HttpResponseException: Server returned wrong status code: 503. Details: Service Unavailable: Back-end server is at capacity
[2021-11-22 11:21:26.180] WARN com.mulesoft.mule.runtime.gw.deployment.platform.interaction.clients.PlatformClientsRetriever [agw-contract-polling.01]: Failed to obtain contracts for API 'API my-sapi-v1-v1:12345678 (12345678)'. This request will be retried after some backoff time. Reason: com.mulesoft.mule.runtime.gw.client.exception.HttpResponseException: Server returned wrong status code: 503. Details: Service Unavailable

Example 2)

[2020-07-23 02:00:07.936] ERROR com.mulesoft.mule.runtime.gw.deployment.runnable.ClientsRunnable [agw-contract-polling.01]: There was an error retrieving contracts for API API xyz-v1:15819185 (15819185). Reason: com.mulesoft.mule.runtime.gw.client.exception.HttpResponseException: Server returned wrong status code: 504. Details: Gateway Timeout Server status: 504
[2020-07-23 07:04:56.898] ERROR com.mulesoft.mule.runtime.gw.deployment.runnable.ClientsRunnable [agw-contract-polling.01]: There was an error retrieving contracts for API API xyz-v1:15819185 (15819185). Reason: com.mulesoft.mule.runtime.gw.client.exception.HttpResponseException: Server returned wrong status code: 504. Details: Gateway Timeout Server status: 504
[2020-07-23 15:26:58.837] ERROR com.mulesoft.mule.runtime.gw.deployment.runnable.ApiKeepAliveRunnable [agw-api-keep-alive.01]: Error activating API xyz-v1:15819185 (15819185). Reason: com.mulesoft.mule.runtime.gw.client.exception.HttpResponseException: Server returned wrong status code: 403. Details: Forbidden. Server status: 403
WARN  2020-07-23 12:13:14,150 [agw-policy-polling.01] [processor: ; event: ] com.mulesoft.mule.runtime.gw.deployment.runnable.ApisRunnable: Failed to retrieve API information for API 'API name-v1-v1:123456789 (123456789)'. This request will be retried after some backoff time. Reason: com.mulesoft.mule.runtime.gw.client.exception.HttpResponseException: Server returned wrong status code: 403. Details: Forbidden

Example 3)

[07-29 06:16:07.168] WARN  ApisRunnable [agw-policy-polling.01] [event: ]: Failed to retrieve API information for API 'API test-exp-api-v1-v1:20212345 (20212345)'. This request will be retried after some backoff time. Reason: com.mulesoft.mule.runtime.gw.client.exception.HttpResponseException: Server returned wrong status code: 503. Details: Service Unavailable

CAUSE

They are caused by an internal call in the platform to authorize the contract activation requests. 
This means that the Gateway could not download the contracts, please remember that the contracts model is an eventual consistency model and the gateway tries to download the contracts every 15 seconds, so if one of those requests fails, there isn’t any problem because it will try again in a few seconds.
The patterns are usually sporadic, not constant.

SOLUTION

These Error/Warning messages have no impact on the customers and it's safe to dismiss the occasional messages like this. The Runtime will keep polling the platform, so it will fetch the contracts again in the next polling cycle (which is set to 15 seconds).

For example, in the above snapshot, only 3 calls were unsuccessful in the time period between 2020-07-23 02:00:07.936 and 2020-07-23 15:26:58.837 which represents about 800 minutes.
That means there were about 3200 calls in the above time period but only 3 of them were unsuccessful.

Hence, these error messages can be safely disregarded.

If you are seeing the message repeatedly and it's happening for all or most of the polling cycles, please contact MuleSoft Support and we will investigate it further.

Below is a diagram of the components involving contract polling between Anypoint Platform's API Manager and Mule Runtime.

Note: You can also see the successful contract pools enabling the following package in DEBUG mode in the log4j2.xml or logging properties from runtime manager:

com.mulesoft.mule.runtime.gw.client