IP Allowlist Policy Blocks Valid Client IPs

SOLUTION

For most IP Allowlist Policies, the check is against the ['X-Forwarded-For'] header to validate the client IP address, as X-Forwarded-For is a common method for identifying the originating IP address of a client connecting to a web server through an HTTP proxy or load balancer.

However, in some cases, the HTTP header X-Forwarded-For will contain a comma-separated list of IP addresses, rather than just the expected single client IP address, as it can be tampered with by a proxy or a client. For example, the DLB/SLB appends the source IP to the values against this header, still leaving the existing ones intact.

Consequently, the header X-Forwarded-For may be presented in this format:

X-Forwarded-For: <IP1>, <IP2>, <source ip before DLB/ELB>

Equally, in maybe that the header X-Forwarded-For may be present in this format:

X-Forwarded-For: client, proxy1, proxy2

where the client is the client IP, proxy1 is the first proxy, etc.

In cases where X-Forwarded-For contains multiple IP addresses, the allowlist policy only checks the first IP ("client" in the example above). The alternative is to check against X-Real-IP HTTP Header. X-Real-IP returns the last proxy or last public IP in the network path.
 

Determining X-Forwarded-For or X-Real-IP

If you need to determine either of these, you have the following options.

1. Enable How to Enable HTTP Wire Logging for the HTTP Connector in your application. This will result in a huge amount of logging but will log all the headers of the incoming HTTP Request. 

2. Use a 3rd party application like httpbin.org example:

$ curl https://httpbin.org/get?show_env
{
  "args": {
    "show_env": ""
  }, 
  "headers": {
    "Accept": "*/*", 
    "Connect-Time": "3", 
    "Connection": "close", 
    "Host": "httpbin.org", 
    "Total-Route-Time": "0", 
    "User-Agent": "curl/7.54.0", 
    "Via": "1.1 vegur", 
    "X-Forwarded-For": "61.68.17.10", 
    "X-Forwarded-Port": "443", 
    "X-Forwarded-Proto": "https", 
    "X-Request-Id": "730093f7-a688-4c83-82a1-428ce46d3ce6", 
    "X-Request-Start": "1518572236795"
  }, 
  "origin": "61.68.17.10", 
  "url": "https://httpbin.org/get?show_env"

3. Create a Cloudhub application that prints out the headers. Following are examples of loggers:

Mule 3

<logger message="::::: X-FORWARDED-FOR :::::: #[message.inboundProperties['X-Forwarded-For']]" level="INFO" doc:name="Logger"/>
<logger message="::::: X-REAL-IP :::::: #[message.inboundProperties['x-real-ip']]" level="INFO" doc:name="Logger"/>

Mule 4

<logger message="::::: X-FORWARDED-FOR :::::: #[attributes.headers['x-forwarded-for']]" level="INFO" doc:name="Logger"/>
<logger message="::::: X-REAL-IP :::::: #[attributes.headers['x-real-ip']]" level="INFO" doc:name="Logger"/>

4. If requests go to the worker directly, both X-Forwarded-For and X-Real-IP header may not exist or could be fake ones. Neither is a good way to get the real source IP address.
Instead, you can use "http.remote.address" header in Mule3 or "remoteAddress" attribute in Mule 4.
This header/attribute is added by the HTTP/HTTPS endpoint. The header is the same as the X-Real-IP header passing DLB/SLB, it will be replaced with the real source IP by the connector. 

In Mule 3, the "http.remote.address" header contains extra "\" and port, so you need a MEL to parse it in the expression.

#[(message.inboundProperties.'http.remote.address'.substring(1,message.inboundProperties.'http.remote.address'.lastIndexOf(':'))) ]

In Mule 4, simply use #[attributes.remoteAddress] in the IP expression. Don't use any header in the expression as a client/proxy can fake them easily unless the header is from a trusted proxy.