Loading

Diagnosing SSL or Connectivity errors for Runtime Fabric Installs and Application Traffic

Date de publication: Jul 30, 2026
Description

ISSUE:
Running Runtime Fabric outbound connectivity validation returns TLS-shaped errors against specific MuleSoft endpoints, commonly seen as:
- transport-layer.prod.cloudhub.io: remote error: tls: certificate required
- us1.ingest.mulesoft.com: connection reset by peer
A running application on RTF may also show image pull failures, policy download failures (application stuck NOT READY), or PKIX/SSL certificate validation errors when calling other MuleSoft platform endpoints.

CAUSE:
These two error signatures look similar but mean very different things, and conflating them wastes troubleshooting time:
- transport-layer.prod.cloudhub.io returning "tls: certificate required" is an EXPECTED FALSE POSITIVE. This endpoint requires Mutual TLS (mTLS) authentication, so the server correctly rejects a basic validation probe that does not present a client certificate. Seeing this specific error against this specific endpoint during a validate run is not evidence of a network or firewall problem.
- us1.ingest.mulesoft.com returning "connection reset by peer" is a GENUINE CONCERN. A connection reset by peer indicates a proxy or firewall on the customer's own network is actively intercepting or terminating the connection, most often caused by Deep Packet Inspection (DPI) or TLS certificate replacement somewhere on the network path. This endpoint is used exclusively for transmitting Anypoint Monitoring logs, so this specific failure does not affect the availability or performance of deployed applications or the cluster itself, but it does mean monitoring data is not reaching the platform, and it is a strong signal that other endpoints may be similarly intercepted.
More broadly, real SSL inspection can affect other MuleSoft endpoints one at a time depending on what the inspection policy is configured to intercept. Endpoints seen affected in real cases include the container registry, exchange-files.anypoint.mulesoft.com, and internal ingestor/router endpoints. A bypass list that covers some but not all required endpoints will leave the application partially broken (for example, image pulls succeed but policy downloads still fail).

Résolution

DIAGNOSTIC APPROACH:
Start by testing outbound connectivity from the RTF nodes themselves:

rtfctl test outbound-network --namespace <rtf_namespace>


This exercises connectivity from inside the cluster to the required MuleSoft endpoints and is the first thing to run when validation errors appear.
Because the affected endpoints require mTLS or standard TLS, any interception or alteration of the certificate chain by a firewall or proxy will break the handshake and is visible in the certificate issuer returned.

To confirm whether certificate replacement is occurring, run the following from within the cluster environment for each endpoint in question:


openssl s_client -connect transport-layer.prod.cloudhub.io:443 -showcerts | grep -E "s:|i:"
openssl s_client -connect us1.ingest.mulesoft.com:443 -showcerts | grep -E "s:|i:"

These two are the most common examples but replace the hostnames as needed in your particular scenario.

Look at the "s:" (subject) and "i:" (issuer) lines in the output. If the issuer is the organization's own internal CA rather than the expected public MuleSoft/DigiCert CA, that endpoint is being intercepted by corporate SSL inspection. Repeat this check against any other endpoint showing a certificate-related failure, since interception is often endpoint-specific rather than blanket.
For a running application already installed, TRACE-level logging can also be enabled on the relevant client/policy scope through the Application Manager API to see exactly which outbound call is failing and against which endpoint.


Check each relevant endpoint individually and do not assume a single bypass rule covers all of them: endpoints seen affected in real cases include the container registry, exchange-files.anypoint.mulesoft.com, us1.ingest.mulesoft.com, and internal ingestor/router endpoints. A bypass list that covers some but not all required endpoints will leave the application partially broken (for example, image pulls succeed but policy downloads still fail).
For a running application already installed, you can also enable TRACE-level logging on the relevant client/policy scope through the Application Manager API to see exactly which outbound call is failing and against which endpoint.

 

RESOLUTION:
- For the transport-layer.prod.cloudhub.io mTLS false positive: no action needed. This is expected validator behavior against an mTLS-only endpoint, not a defect.
- For us1.ingest.mulesoft.com or any endpoint where the openssl check shows a replaced certificate: work with the network/security team to add that specific endpoint to the SSL inspection bypass list. Expect this to be iterative: after each bypass update, re-run rtfctl test outbound-network and re-check certificate issuers, since a round of bypass changes may miss an endpoint that only becomes visible once an earlier blocking error clears.
- If an application also shows repeated OOMKilled restarts alongside SSL or policy-download errors, treat that as a separate, compounding issue: a memory-driven crash loop will keep re-triggering failed startup sequences (including policy downloads) even after the SSL inspection issue itself is fixed. Rule this out or fix it independently.

Ressources supplémentaires

REFERENCES:

Runtime Fabric Self-Managed Network Configuration:

https://docs.mulesoft.com/runtime-manager/rtm-agent-allowlists

https://docs.mulesoft.com/runtime-fabric/latest/install-self-managed-network-configuration

Runtime Fabric Self-Managed Network Configuration (Certificate Configuration): https://docs.mulesoft.com/runtime-fabric/latest/install-self-managed-network-configuration#certificate-configuration

Exporting Telemetry Externally (Custom Log4j Appender, for the TRACE logging path): https://docs.mulesoft.com/runtime-fabric/latest/use-log4j-appender

Numéro d’article de la base de connaissances

005390311

 
Chargement
Salesforce Help | Article