Omni Gateway - mTLS Connection Reset When Routed Through an F5 Load Balancer to a Self-Hosted Gateway

Inbound mTLS (Validate Client Certificate) on a self-hosted Flex or Omni Gateway can fail with a connection reset when traffic is routed through an F5 BIG-IP load balancer, even though direct requests to the gateway succeed. This is typically caused by the F5 Virtual Server operating in full-proxy (Standard) mode rather than true Layer 4 passthrough, which prevents the client's TLS handshake and certificate from reaching the gateway intact.

 

SYMPTOM

A self-hosted Flex Gateway or Omni Gateway has inbound mTLS enabled (Validate Client Certificate under the TLS Context's Inbound Settings). Requests sent directly to a gateway node succeed, with the client certificate presented and validated. The same requests, sent through an external F5 BIG-IP load balancer in front of the gateway, fail with a TLS-layer connection reset, for example:

curl: (35) OpenSSL SSL_connect: Connection reset by peer

or an SSL handshake failure alert (for example, alert number 40). The TCP connection opens and the TLS handshake begins, but resets shortly after.

CAUSE

The F5 Virtual Server in front of the gateway is configured as a Standard type. A Standard Virtual Server is a full Layer 4-7 proxy: F5 terminates the client's TCP/TLS session on the front end and opens a separate backend TCP connection to the gateway. This breaks true mTLS passthrough, because the client's certificate and the original TLS handshake cannot cross two separate connections unless F5 is explicitly configured to re-establish its own mTLS session on the backend leg. Without that explicit re-establishment, the backend connection is rejected or reset before the gateway evaluates the client certificate.

A Performance Layer 4 (fastL4) Virtual Server, by contrast, forwards TCP packets directly between the client and the gateway without terminating or inspecting the TLS handshake, preserving true end-to-end mTLS passthrough.