Guide: Network Troubleshooting between Tableau and external devices

1. Examine the error

There are different avenues to investigate depending on the actual wording of the error. For example, "timed out" means the client waited a certain amount of time before giving up waiting for a response from the server. Whereas "connection refused" means the server actively disallowed communication. Understanding the error will point the investigation in a different direction. Here are some examples:

Connection Timed Out

This means communication between client and destination was likely lost or not received on time. This error may be caused by:

• • • Network path blocked or disconnected
    • Connection Throttling
    • Network latency was too large (busy network)
    • Intermediate devices (proxies, load balancers, etc) taking too long to forward traffic
    • Other processes scanning traffic waited too long to respond (Endpoint software)
    • Server is overwhelmed or unable to respond in a timely manner

Connection Refused

Communication was received, but access was denied. Here are some areas to focus on:

• • • Was the server Tableau was attempting to reach running?  Example: process was down or hung
    • The server configuration could be wrong
    • Firewall is blocking the port used for communication
    • Credentials for access are incorrect
    • Proxy requires authentication, or is blocking traffic from certain clients
    • Outbound traffic is disallowed at the client side (antivirus, endpoint software)
    • The network path or DNS name is sending traffic to an incorrect device (Rare, but possible)

Connectivity Error/Exception

General exceptions cover a wide range of issues, but here are some important elements to consider that may not be listed above:

• • • Network allowlist/blocklists
    • SSL Certificates evaluated with errors or not trusted
    • Local network connectivity is down (Home internet goes out)
    • VPN has lost connection or is improperly configured
    • Network being unreachable (Rare, but ISP fiber outages between networks can cause issues from time to time)

A clear understanding of the error received allows for more targeted troubleshooting of the issue. A connection that times out is very different from a connection refusal. A connection refused gets a response, but a timeout does not. 

2. Narrow the scope of the issue

Isolating the issue to the fewest variables is important for speeding up time to diagnose the problem. When multiple devices, software, users, etc. are involved, it is best to start by eliminating variables. But it's also necessary to know how wide the problem reaches to determine what is most likely to be the cause. Hence, these questions are important to answer:

1. How many users are affected? 
2. Are all of the resources affected (Does one client have issues with multiple servers)?
3. Does this occur when they are at a specific location? (At home, at a specific office, etc)
4. How often does the issue occur?( All the time or intermittent? If intermittent, is there a pattern?)
5. What devices can Tableau bypass, and does the issue still occur? i.e.
    - Proxies
    - Endpoint Software
    - Load Balancers
    - Running a client locally on the server machine.
6. Were there any recent changes to the network environment?
7. Were there changes to access, or the machines?

Once some of the devices involved are identified, bypassing those devices to test if the issue still occurs is the best way to eliminate variables. For example, if a user cannot reach a database from Tableau Desktop at home on a VPN, and another user also can't reach it in the office, then the VPN location and likely the network path aren't related to the root cause of the issue.

3. Engage necessary teams

In a situation where there's a connection timeout or other network exception, knowing which teams to engage is necessary. Here's a general list of teams and/or resources who need to be engaged to track down a Network issue:

1. Client OS manager (IT admin, etc.) is always a good start.
2. Network Administrator
3. Database 
4. Server administrator (This is both the software and hardware admins)
5. 3rd party support (for external or publicly accessible servers)
6. ISP (internet service providers) for either the client (someone working from home) or the server (IT/Infrastructure at a physical location)
7. VPN administrators (or any other software that reads or diverts traffic, such as Antivirus)

Sometimes the manager of a particular server could be multiple people. For example, say there is an issue reaching an on-premise MS SQL Server database. Then both the DBA, as well as a network / infrastructure manager, may need to be engaged to fully cover both the database and the host operating system it the database runs on. If the host is a virtual machine hosted on a platform like AWS, involving the AWS administrator or connecting with AWS support may be necessary as well.

4. Tools for Network Traffic Analysis

When diving deeper into the technical causes of communication failures, it's important to have the right tools on hand. Below are categories of tools and their primary uses.

Command line tools:

There are a lot of basic network connectivity tools that can be run from any machine's command line.

netstat

nslookup

ping

telnet

traceroute

mtr / mytraceroute

ipconfig / ifconfig

Advantages:

• • • • Most are available on most operating systems.
      • With some knowledge, these tools get some immediate scoping information very quickly.
      • Gives straightforward information/answers.

Disadvantages:

• • • • These tools can easily be misunderstood. Ping does not test specific ports. Further, some servers will not respond to ping (ICMP) requests, but is otherwise reachable.
      • Some of these may not be installed by default, and may need a 3rd party tool or package manager to install them.

A Note About Ping

Ping can be misleading, for a number of reasons. There are a lot of reasons we can either give a false positive, a false negative, or just be plain misleading.

• • • •  ICMP may be disabled on a host that is otherwise up, leading to non-response even though it is perfectly reachable (False negative).
      • Pinging by host name (or perceived IP Address) may hit a load balancer or proxy (false positive)
      • If the correct IP address is stolen by the wrong host (False positive)
      • Ping does not check specific ports being open, only that the host is up and responding to ICMP requests. (Would be False Positive, but the understanding of ping is wrong here not the ping itself)
         

Lastly, ping time is not always representative of the machine, but more often of the transport network between them. A long ping time could be the network more likely than the server itself. 

Network Capture tools:

Here are some examples of network applications recommended for finding and troubleshooting issues:

Fiddler

Decrypts and captures http/https web requests. Good for looking for communication between a web server and client and is run from the client machine. It does not capture traffic for communication protocols other than HTTP, and also can bypass errors in SSL evaluation.
 

Advantages:

• • • • • Offers an easy way to see web traffic between a client and server.
        • Provides detailed information about web requests, such as content.
        • Can provide insight into metadata like cookies that may affect how the content is sent

Disadvantages:

• • • • • Can include sensitive information.
        • Can be a security risk (as it is a SSL proxy, it decrypts traffic)
        • Fiddler can sometimes fix issues if SSL certs are the problem.

Browser Dev tools / HAR files:

Browser network traces are possible as well. They can gather the same information as a Fiddler, but can be more difficult to configure properly. They also may not collect all of the information fiddler collects, so some specific information necessary to diagnose the issue may be lost. But an advantage is that it can see what the browser sees, so requests can be gathered without decrypting traffic through a proxy. SSL proxy could fix issues with an otherwise bad server cert, as detailed here:

https://www.telerik.com/blogs/help!-running-fiddler-fixes-my-app-

Access logs (or other application-native logs):

Web applications, load balancers, etc. that send and receive web requests often have access logs, making them useful for reviewing information about requests to and from a client. Often, when a proxy or load balancer is the issue, the request in the access logs is either missing or has a different response code than the request hitting an application. If a load balancer reports a timeout, but the same request shows a completion on Tableau logs, it's likely the proxy has a lower limit set for request time.

Access logs will also generally have important metadata about requests, such as the source IP, the response code, the time taken to process the request, and any headers included with the request (i.e. X-Forwarded-For, X-Forwarded-Proto, etc.)

Other applications or platforms may also have logs that track incoming and outgoing requests. 

Wireshark:

Wireshark is an extremely technical tool that captures network traffic at the packet (and deeper) level. Wireshark is a great tool for seeing very specific information about the network traffic, but has a lot of disadvantages as well.

Advantages:

• • • • • Very detailed information about each individual packet of data sent and received by a machine can be gathered. This can be conclusive evidence of receiving (or not receiving) specific traffic.
        • Multiple wireshark captures from different points on the network path captured at the same time can verify where communication is being lost.
        • Wireshark has many aggregation tools to look at data size, conversations, and handshakes that aren't possible in any other tool.
        • Can be run on almost any platform (Mac, Windows, Linux, or intermediary devices)

Disadvantages:

• • • • • Wireshark is inherently a security risk, as it captures all raw traffic to and from a machine.
        • Captures can often be large unless properly filtered, especially when run for a long time, or on a host that is busy
        • Wireshark will not automatically decrypt traffic, so extra steps to capture SSL session keys are needed.
        • Wireshark is a very complex and powerful tool, but using it effectively requires in-depth knowledge of network technology and concepts, as well as specifics on different communication protocols.

5.  Reading Between the Lines

Network transport issues can be very complicated, and be disrupted for very technical reasons. It is important to ensure you have subject matter experts with technical knowledge of both network concepts, but also the specific infrastructure of your organization to solve these kinds of issues. But it also involves investigation work.

Here are a few tips:

1. 1. Sometimes the absence of something is evidence of something. If something cannot be found, then it's possible (but not inherently proven) that it doesn't exist.
   2. It may be helpful to compare a working example of something to a non-working one.
   3. Eliminating variables (devices, network path, etc) early ensures that you're trying to solve the simplest problem. It can also quickly confirm an active piece of the issue. If bypassing a load balancer works, then it doesn't prove the load balancer is the issue, but it proves it's involved in the root cause.
   4. Hold involved parties accountable to the areas of the network connection that they preside over. If the traffic can reach the server, but the server does not respond, then anyone with expertise on the server machine or software will need to investigate at that point.