Effects of Round Trip Time and bandwidth on performance

When diagnosing system performance issues the System Performance team will likely ask you for ping and tracert results in order to determine the overall speed of your connection and what path your requests take to and from the Salesforce data center. In most instances this information is more significant than the bandwidth of your connection when it comes to performance issues.
To explain why we ask/don't ask for this information, it is helpful to start with a few definitions and analogies.

Key Definitions

Round Trip Time (RTT) – This is the cumulative time that it takes for any data packet to leave your computer, and then travel to the data center and back. The round trip time has a theoretical minimum time of:
(round trip distance) / (speed of light in the communication medium)
For modern fiber optic cables this is approximately 122,500 miles/sec (about 4.5 times around the planet per second). In reality we will never be able to achieve this speed since the data has to make hops along the way, which induces delays as the signals are processed and relayed.
Bandwidth – This refers to the maximum cumulative size of all files being sent through your connection in any given second.
TCP (Transmission Control Protocol) – The communication protocol used to reliably send data between your computer and Salesforce servers. TCP requires acknowledgment of each data packet before the next one is sent, meaning RTT directly affects how quickly each exchange completes.
ISP (Internet Service Provider) – The company that provides your internet connection (for example, Comcast, AT&T, or a corporate network provider). Your ISP's routing and infrastructure can significantly affect your RTT to Salesforce data centers.

Why RTT Matters More Than Bandwidth for Salesforce

One good way to describe the interaction of the two terms is to think of a traffic system.
In this analogy the Round Trip Time (RTT) is the time it takes you to get from one traffic light to the next. This would account for the time needed for the light to change (similar to the processing time at each relay station) and the speed limit (data transfer speed — normally not a limiting factor). Bandwidth in this analogy is how many lanes the road has. If you imagine that each request you make is a car or cars on the road, a big file would be a car that takes up multiple lanes.
As with a real traffic system, occasionally traffic jams will occur at the lights, forcing you to wait. In a computer network this is what we call "Blocking." Using this analogy, what we are looking for when we ask for a traceroute is essentially the route that your car takes and how fast you get through each light, while a ping tells us how long the entire round trip takes for the car to leave and come back.
To further build out this analogy, imagine that you are building a house. Your foreman (your computer/browser) is on a job site with a pre-made foundation and a set of blueprints, but needs lumber to build the frame of the house, so sends a courier (data request packet) to the hardware store (Salesforce) to find and buy (the time the server spends processing the request) the right lumber (the HTML page). The foreman waits for the courier to return before the construction crew can get to work.
During the construction phase the foreman sends the courier out again, but this time to pick up drywall, siding, and shingles (the CSS information). Once that is done the foreman again sends the courier out to pick up any paint/finish/carpet/tile (images/animation) that is needed. Since these can be rather bulky your courier may either have to make more than one trip or take a bigger car to pick them up. Once that is done the owners of the house may request to have the yard landscaped and a sprinkler system added (JavaScript and components), so again the courier is sent out.
That example, while basic, is fairly accurate.
Most files that Salesforce sends are rather small so the request doesn't need to take up many "lanes." Some requests take longer to process, like dashboards or reports, but the resulting data packets (what is sent back to the job site) are still rather small. Furthermore, if you have a fairly complex internal network (think a large residential community) most of the time spent for each courier trip can be attributed to your courier just trying to get to a main road. On a traceroute, the intersection to the main road is normally the line that shows XXX.XXX.0.1. Since each page request can take many back and forth trips depending on the complexity of the page, the short time at each hop can add up significantly. For example, if it takes 200ms just to make a round trip to and from the main road and a typical page load can require 5 or more requests, that means that your internal network has added more than a second to your page load time.
Finally, if you and all your coworkers are trying to upholster a number of different houses at the same time, all of your respective couriers will spend more time waiting at stop lights than they will on traveling. It is only in this specific scenario that bandwidth is relevant in the discussion of performance issues.

What to Do If You Have High RTT

If you suspect high RTT is impacting Salesforce performance, run a ping and traceroute (tracert on Windows, traceroute on Mac/Linux) to the Salesforce data center. These results help the System Performance team identify where delays are occurring along the network path. See the Resolution section below for step-by-step instructions, and the Additional Resources section for the full guide.