GovCloud - March 2024 - Updates on FIPS Enforcement

FIPS 140-2 Enforcement

1.  CloudHub Application

The March update for CloudHub 1.0, scheduled for release on March 12th, 2024, will introduce Runtime 4.6 and patches for Runtime versions 4.3.0 and 4.4.0. This update is significant as it will enforce FIPS 140-2 compliance on runtime versions 4.6, 4.4 and 4.3 within the MuleSoft Government Cloud for CloudHub applications. Customers operating applications that are not compliant with FIPS standards may experience disruptions or impacts following this release.

In order to minimize any potential impact, we have made the decision to suspend monthly auto-updates during March and April of 2024. As a result, we will not be automatically patching sandbox and production environments on the specified dates. 

We highly recommend that customers perform a self-update to the 4.6 Runtime or March patch version for 4.3 and 4.4 and ensure that their applications comply with FedRAMP regulations. It's important to note that we are currently in the process of validating FedRAMP compliance for highly used connectors. The list of published connectors in exchange has been updated to differentiate between FedRAMP compliant and non-FedRAMP compliant connectors. If you use MuleSoft published connectors or partner/community developed connectors in your Mule applications, it is highly recommended that you test these applications thoroughly in your CloudHub sandbox environment in FIPS enforced mode before deploying them to production. In case you find that your application depends on a MuleSoft connector that is not FedRAMP compliant, please contact our support team. Our experts will work closely with you to provide a FedRAMP compliant version of the connector, ensuring that your application meets all necessary security requirements. If you find that your application depends on a partner published connector that may not be FedRAMP compliant, please work with your partner to make sure that the connector is FedRAMP compliant as Mulesoft is not responsible for FedRAMP compliance of partner developed connectors.

List of FIPS  compliant connectors in Exchange:

Monthly auto-updates for CloudHub in Mulesoft Government Cloud will resume in May 2024, and FIPS 140-2 compliance will continue to be enforced.

We understand that this change may have an impact on some of our customers and we apologize for any inconvenience this may cause. Please don't hesitate to reach out to us if you have any questions or concerns regarding this matter.

2.  Dedicated Load Balancers monthly patching

After the recent AMI update on November 7, 2023, it was noted that several customers experienced 504 errors with their APIs when they independently upgraded their dedicated load balancers. The primary cause of these errors is linked to the default FIPS configuration in the load balancers, which blocks the use of TLS 1.1 by client-side applications in the Runtime Manager, resulting in the 504 error. To address this issue, we recommend the following steps for our customers:

• • Ensure that all Mule applications connected to the load balancers are compatible with TLS 1.2.

  • Select the "Upstream TLS 1.2" option during the upgrade process.

  • TLS 1.2 will be enforced once the customer upgrades the DLB to the latest image. Customers will not be able to disable TLS 1.2 after the upgrade.

  • All future DLB images will have TLS 1.2 default.

Please review “GovCloud - Mar 2024 DLB upgrade” KB upgrade document.

To provide our customers with sufficient time to transition their applications to TLS 1.2 compatibility, MuleSoft will postpone the automatic patching of dedicated load balancers in the MuleSoft Government Cloud for March and April 2024. The automated patching schedule for MuleSoft Government Cloud - CloudHub Dedicated Load Balancers will commence in May 2024. The specific dates for the initial monthly automated patching are as follows:

• • Sandbox Environment: May 20-24, 2024

  • Production Environment: May 25-26, 2024

FAQ 

What can I do to prepare?

Customers are strongly encouraged to self-update their CloudHub applications with the February Date Patch release to minimize any downtime and update their mule applications, if using non-FIPs compliant applications. 

Will my CloudHub applications still have the latest OS and security patches in February? 

The March Patch will have the latest OS and security patches however it will not be auto-updated in the sandbox and production environment.

What can I check for to see if they are using non-FIPS-compliant applications?

Customers should be using PKIX algorithms and PKCS12 key stores.

How to create a key that is FIPS Validated BCFIPS keystores?

Download the following libraries into ${JAVA_HOME}/jre/lib/ext/

bcpkix-fips

bctls-fips

Bc-fips

Update the java security file to add Bouncy Castle Fips as security provider in provider section

#

# List of providers and their preference orders (see above):

#

security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider

security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS

Use Keytool to Generate the keys

keytool -genkeypair -keyalg RSA -keysize 2048 -sigalg SHA256withRSA -alias mulekey -keystore /tmp/mykeystore.p12 -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath ${JAVA_HOME}/jre/lib/ext/bc-fips-1.0.2.4.jar -storepass mule123 -deststoretype pkcs12

Key size should be at least 2048 bits.
Key Signing algorithm should be SHA256withRSA or higher. 

SHA1 based keys will not work. 

You can verify this with the command 

keytool -v --list --keystore /tmp/mykeystore.p12 --storepass <keystore password> -storetype pkcs12

Keystore type: PKCS12

Keystore provider: BCFIPS

Your keystore contains 1 entry

Alias name: mulekey

Creation date: Mar 19, 2024

Entry type: PrivateKeyEntry

Certificate chain length: 1

Certificate[1]:

Owner: CN=<MASKED>, OU=Cloudhub, O=Mulesoft, L=Bellevue, ST=WA, C=US

Issuer: CN=<MASKED>, OU=Cloudhub, O=Mulesoft, L=Bellevue, ST=WA, C=US

Serial number: a6497b9d49e6cf90

Valid from: Sat Mar 16 22:53:03 UTC 2024 until: Fri Jun 14 22:53:03 UTC 2024

Certificate fingerprints:

SHA1: 12:A9:CD:9B:61:2F:34:2F:8B:1B:5F:43:D4:AB:6E:14:FD:08:16:29

SHA256: 99:7E:86:71:CD:34:33:64:D1:5E:EE:3C:7D:CB:0F:4C:27:27:AC:76:36:FD:22:CD:C1:F1:AE:63:E8:49:24:5F

Signature algorithm name: SHA256WITHRSA   <-----------------

Subject Public Key Algorithm: 2048-bit RSA key <-----------------

Version: 3

*******************************************

*******************************************

Can I roll back? 

Yes, if you self-update and run into issues, you would be able to roll back to the previous version of their CloudHub 1.0  application in the Application Manager UI or the API.

Is there an exception process for the auto-updates? 

No. There will be no customer exception in this process.

Can I opt out of this DLB upgrade?

Unfortunately, we can not delay the upgrade to your DLBs past the deadline because of technical dependencies that could impact the reliability of your DLBs.  If you have not proactively upgraded your DLBs by the deadline they will be automatically upgraded.

Does this update impact Hybrid Standalone customers ?
No, the FIPs enforcement on the 4.3, 4.4, and 4.6 runtimes is only applicable to the runtimes that are deployed in Cloudhub.