How do I validate PCE Security Certificate

SYMPTOM

Many pods within PCE communicate with each other using secure connection, even though it is within Kubernetes. For example, when fetching policies from Exchange in API Manager, API Manager attempts to make connections to Exchange load balancer, and thus, the load balancer certificate which is configured in Access Management Security section, is being passed to the API Manager pods.
Below are few issues which we have observed that are due to invalid load balancer certificates:
1. API Manager Mule4 default policies are not showing up
- For this issue, please ensure that the KB PCE does not load/show default polices has been applied if you are running on 2.0.0-2.0.2 versions of PCE. For PCE 2.0.3 or later, you may skip this step. If the KB resolves your issue, you may ignore the rest of the KB.
- If the KB does not resolve the issue, please validate using the following steps:

Perform the action in API Manager and record HAR file as per How to retrieve HTTP archive files (HAR), you should observe the request going to
```
https://<Domain>/apimanager/xapi/v1/organizations/<Org_ID>/exchange-policy-templates
```
returns empty [].

the api-platform-api logs, you may have observed the following error:

2019-10-10T00:00:00.000Z - error: <ID>
 InvalidTokenError: Invalid token: TokenExpiredError: jwt expired
    at /usr/src/app/api/middlewares/authenticationMiddleware.js:164:31
    at tryCatcher (/usr/src/app/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/main/promise.js:503:31)
    at Promise._settlePromiseAt (/usr/src/app/node_modules/bluebird/js/main/promise.js:577:18)
    at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/main/promise.js:693:14)
    at Async._drainQueue (/usr/src/app/node_modules/bluebird/js/main/async.js:123:16)
    at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/usr/src/app/node_modules/bluebird/js/main/async.js:15:14)
    at processImmediate [as _immediateCallback] (timers.js:383:17)

Access PostgreSQL within your PCE How to access the PostgreSQL Stolon cluster in PCE, and run the following query
```
psql -U stolon -h localhost api-platform
select "name" from "ExchangePolicyTemplates";
```
You should see that the result is empty.

2. When you upload the certificate in Access Management Security section, the certificate gets successfully uploaded (the successful message is shown in the UI), but after the screen refreshes, you identified that the certificate has not being changed. You may validate this symptom by the steps below:
- Nginx pods has not being restarted after the certificate gets successfully uploaded in the UI.
- If you record a HAR file How to retrieve HTTP archive files (HAR) while uploading the certificate, you observe 500 error in the HAR file.
- No error logs are observed in the cs-config pods.

If you are observing any of the above two symptoms. It is very likely that your certificate has issues, and you may consider resolve the issue using the steps below:

CAUSE

PCE certificates needs to be nginx compatible as per Configure Security for Anypoint Platform PCE. If you would like to understand what nginx accepts, you may refer to nginx official docs.

SOLUTION

1. This step is to validate, if the certificate currently in place within PCE, is acceptable or not. You may skip this step if you have not uploaded the certificate to PCE successfully.

DNS="<Insert_Your_DNS>"
echo "" | openssl s_client -servername ${DNS} -connect ${DNS}:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > public.cert.pem
curl --cacert "public.cert.pem" https://${DNS}/

Please replace <Insert_Your_DNS> with the DNS defined in Configure DNS on Anypoint Platform PCE.
If you are observing any error for the call. This means, your certificate has issues, and please follow the steps below to check what is incorrect. Most likely, you may be missing chains. You may run the following command in any of your PCE node to see if it prints out the full chain.

openssl s_client -connect <DNS>:443

Pay attention to depth=x. If depth is 1, there should be certificate chain 0, 1 similar to below:

---
Certificate chain
 0 s:/C=US/ST=California/L=Sunnyvale/O=Oath Inc/CN=*.www.test.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
---

If you would like to understand what is a valid certificate chain, you may use the third-party reference article.

2. Check if the certificate currently uploaded to PCE is the same as the intended certificate bundle. This can be retrieved from the secrets stored inside PCE. As an example for PCE 2.1.2, here are the commands:

kubectl get secrets -n default nginx-ssl -o json | jq -r '.data."cert.pem"' | base64 --decode

PCE also uses this following command to verify certificate status after running the above command and display the error in Access Management. "cert-bundle.pem" is the output in a file from above command:

cat cert-bundle.pem | curl -v --cacert /dev/stdin https://<your PCE DNS>

3. Ensure that your key is not encrypted, as when updating the certificate in Access Management, there are no password placeholder.
To validate this, run the following command:

openssl rsa -check -noout -in server.key | openssl md5

replace server.key with your key. And check if it asks for

Enter pass phrase for server.key:

If you are observing it asking you for pass phrase, please remove the pass phrase by following this reference article.

4. If you are using certificate chain, ensure that it is in the right chain format, certificate first, followed by intermediate, followed by root. you may use the third-party reference article to understand how it works. You should concatenate all certificates into one certificate. This Salesforce KB helps you to understand how a typical merged certificate looks like.
5. Ensure that the private key matches with the public cert:

1. openssl pkey -in privateKey.key -pubout -outform pem | md5
2. openssl x509 -in certificate.crt -pubkey -noout -outform pem | md5

The command is obtained from https://www.sslshopper.com/certificate-key-matcher.html. All result should return the same value. If they don't match, please re-generate your certificate.
6. The certificate should be in text format, like below:

If the certificate is in text format, then it is in PEM format.
You can read the contents of a PEM certificate (cert.crt) using the 'openssl' command on Linux or Windows as follows:

openssl x509 -in cert.crt -text

If the above command doest not return the certificate, you need to convert the certificate into PEM format for PCE to understand. You may use SSL Converter online to convert the certificate, or run the commands as listed in SSL Converter website. Pay special attention that if you are having chains, you will need to convert one by one, then concatenate the certificate as per Step 4. This Salesforce KB helps you to understand how a typical merged certificate looks like.
7. Check if the modulus of your private key and certificate are the same. This will confirm they are the right pair:

# openssl rsa -noout -modulus -in server.key | openssl md5
# openssl x509 -noout -modulus -in server.crt | openssl md5

8. Try taking PCE out of the picture and test with a simple openssl server (step 3 of this article: How to set up a minimal SSL/TLS server from the command line

9. If you are still not able to figure out what is wrong with your certificate after following the steps above, please raise a support ticket, and include the certificate you are uploading. If possible, please provide the private key as well (please generate a new pair of certificate and private key so that your real private key is not compromised, as we don't want to get hold of your real private key) so we can further assist you to check further. Otherwise, try to have your CA provide a certificate and key for an endpoint that will not be used.

NOTES:

If you just need a certificate to test on, you may generate a self-signed certificate by following How to create a self-signed certificate.

DISCLAIMER:

As certificate is owned by the customer. MuleSoft support cannot assist customer to generate certificate and cannot validate if the certificate is valid or if it is up to security standard. It is customer's responsibility to generate the certificate and ensure that the certificate is valid and up to security standard. This article only provides general guidelines to troubleshoot when there are issues with the certificates which customer is uploading to PCE, and any referenced articles/sites that are present in this KB should be used for informational purposes only. MuleSoft bears no responsibility on any of the articles/sites that this KB references to, and it is at customer's own discretion to verify and validate its certificates.

How do I validate PCE Security Certificate

SYMPTOM

CAUSE

SOLUTION

NOTES:

DISCLAIMER:

General Information

Required Cookies

Functional Cookies

Advertising Cookies

General Information

Required Cookies

Functional Cookies

Advertising Cookies

Cookie List