Loading

How to diagnose "java.io.IOException: Remotely Closed" in HTTPS outbound calls

Дата публикации: Mar 2, 2024
Решение
Problem
 
When making an outbound HTTPS call, if the following error is seen:
 
******************************************************************************** 
Exception stack is: 
1. Remotely Closed (java.io.IOException) 
2. java.io.IOException: Remotely Closed (java.util.concurrent.ExecutionException) 
org.glassfish.grizzly.impl.SafeFutureImpl$Sync:363 (null) 
3. java.util.concurrent.ExecutionException: java.io.IOException: Remotely Closed (java.io.IOException) 
org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:274 (null) 
4. Error sending HTTP request. Message payload is of type: String (org.mule.api.MessagingException) 
org.mule.module.http.internal.request.DefaultHttpRequester:190 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html) 
******************************************************************************** 
Root Exception stack trace: 
java.io.IOException: Remotely Closed 
 
********************************************************************************
 
It's likely a SSL/TLS handshaking failure. The remote server is closing the connection but the reason is not clear.

NOTE: most of the concepts presented below apply to the deprecated HTTPS outbound endpoint but above stack trace was originated with the HTTP Requester with a TLS configuration.
 
How to diagnose
 
To find out more on this, we can: 
 
1) enable verbose SSL/TLS debug 
 
in your wrapper.conf, please add the following (please change <n> to a number that is unique for your wrapper.conf file):   
 
wrapper.java.additional.<n>=-Djavax.net.debug=ALL
 
then redeploy the application then replicate the problem. You will see what protocol is used to make the handshaking. For example:
 
*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1464074008 bytes = { 159, 22, 102, 87, 74, 8, 35, 10, 189, 119, 224, 63, 124, 101, 254, 168, 63, 201, 7, 78, 50, 26, 246, 123, 196, 134, 39, 95 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1, sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1, sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1, sect193r1, sect193r2, secp224k1, sect239k1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA224withECDSA, SHA224withRSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA, MD5withRSA
***
 
From above, we can see:
 
- TLS version
- Cipher Suites
 
2) get a comprehensive SSL/TLS report on the remote server
 
You can choose one of the following: 
  • ssllab, https://www.ssllabs.com/ssltest/index.html, if the remote site is publically accessible on internet and listens on standard port 443
  • htbridge SSL test, https://www.htbridge.com/ssl/, if the remote site is publically accessible on internet, this one does not have restrictions on port
  • SSLyze, https://github.com/iSECPartners/sslyze, if the remote site is internal
From the report, we can see what TLS protocols and Cipher Suites the remote site support. 
 
The possible causes
 
SSL/TLS version or Cipher Suites not accepted by remote server
 
If the SSL/TLS version from ClientHello is accepted by remote server, or the proposed Cipher Suites does not overlap with Cipher Suites supported by remote server, the remote server might close the connection straight away, which will end up with "Remotely Closed" error.

Note that many sites already deprecated all SSL versions and TLS 1.0 so those might not be acceptable for the remote server, even if they were in the past.
 
Solution: we can configure what SSL/TLS version and Cipher Suites to use in tls-default.conf, which locates in $ESB_HOME/conf directory. In practice, it is more common to have to change TLS protocols versions than Cipher Suites.
 
Client certificate chain is not accepted by remote server in a 2-way SSL handshaking
 
If the certificate(chain) sent by client side is not accepted/trusted by remote server, remote server will drop the connection straight away which will end up with this "Remotely Closed" error. 
 
We can confirm this if we see the following pattern from SSL/TLS debug log:
 
- server sent certificate request before ServerHelloDone
 
*** CertificateRequest
...
*** ServerHelloDone
 
- client sent certificate
 
*** Certificate chain
 
- the connection got closed
 
SNI extension not accepted by remote server
 
SNI extension is one of the extension might be included in the ClientHello message. For example:
 
Extension server_name, server_name: [host_name: name.company.com]
 
Some servers require SNI to be sent, if not, it will close the connection; while some servers require SNI not to be sent, if sent, it will close the connection. 
 
SNI is enabled by default since JDK 7, but can be turned off by adding the Java system parameter: -Djsse.enableSNIExtension=true. Note, the SNI flag is JVM level thus will affect all outbound HTTPS calls.
 
In ESB, HTTP Requester uses Grizzly library to make the HTTPS call. For ESB 3.6.0 to 3.6.2, Grizzly version 2.3.16 is used, which does not send SNI by default. This has been addressed by https://java.net/jira/browse/GRIZZLY-1737 and fixed in Grizzly lib version 2.3.19. Since ESB 3.6.3, a later version of Grizzly library is used, thus SNI is sent by default.

This issue with the Grizzly library has also been resolved in the up-to-date HTTP connectors for Mule 4
 
Номер статьи базы знаний

001118593

 
Загрузка
Salesforce Help | Article