Use Pings and Traceroutes to Troubleshoot Network Issues

Executing Traceroute and Ping in Microsoft Windows

1. On the Windows taskbar, click the Start button | select Run.
2. Type cmd in the text box.
3. Click OK. A DOS window will appear.
4. In the DOS window, type the following and press Enter:

tracert login.salesforce.com

5. To send 100 echo ping requests, type the following and press Enter:

ping -n 100 login.salesforce.com

6. Repeat steps 4 and 5 for MyDomain.

For example:
tracert MyDomainName.my.salesforce.com.
ping MyDomainName.my.salesforce.com.

Since the Windows tracert command is only a single snapshot of the network at a point in time, we recommend running the command multiple times to ensure a fair sampling of data is collected. Running the command when the slowness occurs would provide a the most useful snapshot.

After you have completed this process, take a screenshot of the command output, or copy and paste the results into a text editor or reply email. If you have an ongoing case with your internal network team or Salesforce support, these logs will help to identify the source of a potential network issue.

Salesforce transmits packets of various sizes when communicating with your machine. If you have done successful basic traceroutes and pings to salesforce.com as covered above, please follow these instructions to run more extensive packet transmission tests.

1. On the Windows taskbar, click the Start button | select Run.
2. Type cmd in the text box.
3. Click OK. A DOS window will appear.
4. In the DOS window, type each of the following, one at a time, and press Enter.

Note: Allow each command to fully process before entering the next command. In all steps, the character before 1200, 1300, and 1400 is a lowercase "dash L."

ping -f -n 25 -l 1200 login.salesforce.com
ping -f -n 25 -l 1300 login.salesforce.com
ping -n 25 -l 1400 login.salesforce.com

5. Repeat step 5 replacing the URL with your Mydomain. For example:

ping -f -n 25 -l 1200 MyDomainName.my.salesforce.com.
ping -f -n 25 -l 1300 MyDomainName.my.salesforce.com.
ping -n 25 -l 1400 MyDomainName.my.salesforce.com.

Take a screenshot of the command output once all are completed, or copy and paste the results into a text editor or reply email.

Executing Traceroute and Ping in macOS

Apple's macOS comes with a traceroute function that can be accessed via command line using the Terminal application that ships with every version of macOS.

Traceroute using Terminal.app

1. Open a terminal session using Terminal.app.
2. Run these traceroute commands:

traceroute login.salesforce.com
traceroute MyDomainName.my.salesforce.com.

3. Copy and paste the results into a text editor or reply email.

Note: To run a traceroute to your Live Agent server, use the above steps, but instead of pointing to the Salesforce instance URL, point to your Live Agent server. You can find your Live Agent server by going to Setup and typing "Live Agent Settings" in Quick Find. You will see an endpoint URL that looks similar to

"https://d.la1w1.salesforceliveagent.com/chat/rest/".

Remove "https://" and "/chat/rest/" then run a traceroute on just the "d.la1w1.salesforceliveagent.com" portion of the address.
 

Interpreting Ping Results

The output of a ping will look similar to this example pinging na17.salesforce.com below:

Reply from 96.43.151.188. Bytes=32 Time=104ms. TTL=242
Reply from 96.43.151.188. Bytes=32 Time=104ms. TTL=242
Reply from 96.43.151.188. Bytes=32 Time=104ms. TTL=242
Reply from 96.43.151.188. Bytes=32 Time=104ms. TTL=242
Reply from 96.43.151.188. Bytes=32 Time=104ms. TTL=242

Ping Statistics for 96.43.151.188:
Packets: Sent = 100 , Recieved = 100 Lost = 0 (0% loss),
Approximate round trip times in milli-seconds :
Minimum = 104ms, Maximum = 107ms, Average = 104ms

The above results show a fast connection with no packet loss. Indications of an issue would include:

• Packet loss of 10% or more. There was 0% loss in the above example.
• A discrepancy between minimum, maximum, and average round trip times (RTT). For the above example, the minimum RTT was 104ms, maximum was 107ms, and the average was 104ms. These results indicate there was little variance in latency over a period of time where pings were being captured. However, if the results were to instead show a minimum of 100ms and a maximum of 500ms, this would indicate there are issues with latency causing a degraded connection intermittently.

Interpreting Traceroute Results

A traceroute will show performance statistics for each hop in the network path your computer takes to reach Salesforce to help identify where issues may be occurring.

Traceroute Terminology

• Hop Number: The specific hop number in the path from the sender to the destination.

• Round Trip Time (RTT): The time it takes for a packet to get to a hop and back, displayed in milliseconds (ms). By default, tracert sends three packets to each hop, so the output lists three roundtrip times per hop. RTT is sometimes also referred to as latency. An important factor that may impact RTT is the physical distance between hops. For a more detailed and verbose description of RTT and its affects, see the Effects of Round Trip Time and Bandwidth on Performance article.

• Name: The fully qualified domain name (FQDN) of the system. Many times the FQDN may provide an indication of where the hop is physically located. If the Name does not appear in the output, the FQDN was not found. It is not necessarily indicative of a problem, if an FQDN is not found.

• IP Address: The Internet Protocol (IP) address of that specific router or host associated with the Name.

 The Components of a Traceroute

The first line of the tracert output describes what the command is doing. It lists the destination system (salesforce.com), destination IP address, and the maximum number of hops that will be used in the traceroute (30).

The first hop is the initial stop your traffic makes after it leaves your computer. It will likely be either a 10.X.X.X number or a 192.168.X.X number. These are reserved for private networks and are also fairly common further down a traceroute. Early hops in the route with these address prefixes are typically within your company's internal network. Further down the route they just indicate that traffic is going through the ISP’s internal network prior to exiting.
 

Identifying Common Issues
 

The three numbers you see are the individual times to get to that specific hop. It is important to note that these numbers do not represent the difference in time between the current hop and the previous, but rather represent the cumulative time taken up until that hop. When you look at a traceroute you are looking for the first point at which there is a high degree of variation between times (for example: 50 ms 283 ms, 29 ms) or for times that are consistently much higher than the preceding hop’s. You may also see “*” as an entry. These indicate no reply was received from the server. These are not necessarily signs of an issue, especially once you reach Salesforce. Certain networks do not reply to the packets used in traceroutes for security or prioritization reasons. Once you are inside the datacenter, Salesforce does the same. If you see one hop has one timeout, that probably is not a problem as long as the connection consistently completes. It is important to note that your “normal” traceroute and ping results will vary based on the geographic location of both your system and the datacenter. For example, if you are in Australia and are connecting to an eastern US data center a 250-300ms roundtrip time is not out of the ordinary due to the geographical distance the connection must physically travel through in undersea cables. However, a 300ms roundtrip time from Australia to the Tokyo datacenter would be unusual.

Traceroute results that show increased latency on a middle hop, which remains similar all the way through to the destination, do not indicate a network problem. A traceroute that shows dramatically increased latency on a middle hop, which then increases steadily through to the destination, can indicate a potential network issue. Packet loss or asterisks (*) on many of the middle hops may indicate a possible network issue if the destination is unreachable or latency climbs considerably in subsequent hops. A steady trend of increasing latency is typically an indication of congestion or a problem between two points in the network and it requires one or more parties to correct the problem.

For more details on interpreting traceroutes, please see this PDF titled “A Practical Guide to (Correctly) Troubleshooting with Traceroute:

https://major.io/wp-content/uploads/2012/06/RAS_Traceroute_NANOG_slides.pdf

Asterisks or "Request Timed Out" Messages

If an asterisk (*) appears for RTT, then a packet was not returned within the expected timeframe. One or two asterisks for a hop do not necessarily indicate packet loss at the final destination.

When three asterisks appear, you will see a "Request timed out" message on one or more hops in the path. This does not necessarily indicate a network or ISP issue, and most of the time means that those hops are not prioritizing ICMP (ping and traceroute) packets.

Since the destination was able to be reached, this indicates there was no forwarding loss at any of the middle hops, and the consistent and low latency level at the end of the connection supports a conclusion of there being no network issues.

You may receive three asterisks followed by the “Request timed out" for the following reasons:

• ICMP packet deprioritization.
• The destination’s firewall or other security device is blocking the request.
• There could be a problem on the return path from the target system. Remember the round trip time measures the time it takes for a packet to travel from your system to a destination system and back. The forward route and the return route often follow different paths. If there is a problem on the return route, it may not be evident in the command output.
• There may be a connection problem at that particular system or the next system.

Odd Routing Decisions

ISP’s are constantly making updates to their networks such as tweaking routing decisions and adding new lines to keep their network healthy and to optimize around certain traffic patterns. Occasionally, these changes could route your requests to Salesforce via a less than ideal path. For example, if you are accessing a Salesforce instance in North American from California, but are being routed through Singapore and back before reaching the application, you will certainly see increased load times. You can identify these routing issues in the traceroute by examining the name of hops, which ISPs tend to label with location-based information and a spike in RTT of 100-200 over the span of 1 hop which is typical behavior of an intercontinental hop.

If you are based in the same region as your data center (North America for NAXX instances, Asia for APXX, Europe/Africa for EUXX) and you happen to notice a large jump in your traceroutes, you can go to http://ipddress.com and enter the troublesome IP address to find out the geographic location it is located in. If you notice this type of issue please contact your ISP as they control the route your request takes to Salesforce.

Contact Salesforce Support

If you are unable to pinpoint any networking issues, please send all the information you have collected using the steps above to Salesforce support and we will take a closer look at the issue. In addition to the traceroute and ping described above, you will also want to include login access, the steps to reproduce the slowness you are seeing, and the name of your Internet Service Provider (ISP) as well as whether your ISP has provided any insight into the latency.

NOTE:
Hyperforce uses Network Load Balancers (NLBs), which don’t support pings (and by proxy tracert). On NLBs, ICMP Packets except for Type 3 (Destination Unreachable) are considered “Unintended traffic” and are not forwarded to any targets. 

Troubleshoot Network issues in Hyperforce