Tableau Server Upgrade & Recovery Framework

1.0 Purpose and Guiding Principles

The objective of this document is to provide customers with a standardized methodology for performing Tableau Server upgrades with predictable outcomes and protected data integrity.

This document defines the expected behavior, recommendations, and escalation actions for customers performing Tableau Server upgrades.

Primary Goals:

1. Minimize customer risk during upgrades.

2. Preserve recovery options at all times.

3. Reduce prolonged outages.

4. Ensure predictable, repeatable upgrade outcomes.

Guiding Principles:

1. Recovery integrity: Recovery options must be preserved until stability is confirmed.

2. Fail-fast mentality: Early rollback or restore is preferred over prolonged remediation.

3. Gold Standard: Blue-Green upgrades represent the safest deployment model.

4. Time is a critical factor: Extended instability during an upgrade reduces available recovery options and limits safe rollback paths.

Table of Contents

• 1.0 Purpose and Guiding Principles
• 2.0 Strong Recommendation: Blue-Green Upgrades
  • 2.1 Standard Guidance for Customers
  • 2.2 Prerequisites: The "Ready for Green" Checklist
  • 2.3 Performance Comparison: Log & Metric Capture
  • 2.4 Rollback Strategy (Green Environment — Post-Upgrade)
• 3.0 In-Place Upgrades: When Blue-Green Is Not Feasible
  • 3.1 Backup Validation: Assets and Prerequisites
  • 3.2 Restore Testing Options (Risk Mitigation)
  • 3.3 Operational Constraints & Checkpoints
  • 3.4 The Recovery Strategy (The Exit Plan)
• 4.0 Actions When an Issue Is Discovered During an Upgrade
  • 4.1 Immediate Data and Log Preservation
  • 4.2 Primary Recommendation: Roll Back or Restore Immediately
  • 4.3 Critical Decision: Obliterate vs. Troubleshoot
• 5.0 Escalation and Decision Points
• 6.0 Key Takeaways and Core Principles

2.0 Strong Recommendation: Blue-Green Upgrades

Blue-Green upgrades are strongly recommended and should be the default approach for Tableau Server upgrades.

The Blue-Green model is the primary recommended approach for all customers. It involves standing up a parallel environment (Green) running the new version while the existing production environment (Blue) remains unchanged.

Tableau recommends the Blue-Green approach for mission-critical environments where downtime must be minimized and a safety net for immediate rollback is required.

2.1 Standard Guidance for Customers

• Near-Instant Rollback (The Safety Net): If a critical issue occurs after production cutover, traffic can be immediately switched back to the original Blue environment.
• Zero to Minimal Downtime: Because the Green environment is pre-built and tested, the final cutover is usually just a DNS or load balancer switch.
• Risk isolation: The upgrade process does not impact the active Blue environment, ensuring upgrade-related risks are isolated and customers using the Blue environment are not affected.
• Validation without impact: Customers can run full User Acceptance Testing (UAT) on the production-sized Green cluster while the Blue environment continues to serve business operations.
• Infrastructure Modernization: The Blue-Green approach allows addressing the need to upgrade the underlying OS (e.g., Windows Server 2022 or Linux RHEL 9) with minimal downtime. In this method, Tableau Server is installed and configured on newly provisioned infrastructure with the updated OS, and existing data is restored from backup to enable a smooth transition.

2.2 Prerequisites: The "Ready for Green" Checklist

The Green environment must match or exceed the Blue environment in resource capacity and Tableau Server cluster topology distribution. If the Green environment is under-provisioned, there is a risk of immediate performance degradation following DNS cutover from the Blue production environment to the upgraded Green environment.

Granular Topology Checklist

• Identical Specifications: The Green environment should mirror the Blue environment in capacity (CPU, RAM, disk) to ensure performance parity.
• Process Parity: Match all process counts (e.g., VizQL Server, Backgrounder) using tsm topology list-nodes -v on the Blue environment. (https://help.tableau.com/current/server/en-us/cli_topology_tsm.htm)
• VizQL Server: Typically configured as 1 instance per 4 physical cores. Avoid over-subscription.
• Coordination Service (Zookeeper): If the Blue environment uses three or more nodes, ensure a Coordination Service ensemble is deployed in Green:
  tsm topology deploy-coordination-service -n node1,node2,node3

• Backgrounder Capacity: Ensure the number and distribution of Backgrounder processes in the Tableau cluster match those in the Blue Production environment to avoid a backlog of extract refreshes immediately after cutover.
• Repository Type: If the Blue environment uses an external repository (e.g., AWS RDS or Azure SQL), the Green environment must be configured with the same external repository setup.
• CPU Affinity: In virtualized environments (e.g., VMware, Azure, AWS), dedicated CPU affinity is strongly recommended. If the Blue environment uses dedicated cores but the Green environment runs on shared or burstable instances, extract refresh times may increase significantly.
• Storage Throughput: Ensure storage performance matches the Blue environment in terms of IOPS (Input/Output Operations Per Second).
  • File Store Node: Require high disk throughput.
  • Repository Node: Require low latency (SSD or premium-tier disks).
• External Dependencies: Ensure the Green environment has the same firewall access, SSL certificates, and database drivers as the Blue environment.
• The "Run As" Service Account: The Green environment must use the same domain service account as the Blue environment.
• External SSL: Do not simply copy the existing .crt and .key files from the Blue environment to the Green environment. Because the Green environment may use a different hostname or IP during staging, a new Certificate Signing Request (CSR) must be generated and a new SSL certificate obtained for the Green cluster.

  Ensure all newly generated SSL files are stored in a secure, centralized location on the Green primary node, as original private keys may not be available on the Blue production environment.

  • Hostnames Change during Upgrades: External SSL certificates are explicitly tied to Fully Qualified Domain Names (FQDNs). Since the Green environment typically uses a temporary staging URL, the Blue environment certificate will not be valid and may result in browser security warnings.
  • The missing key constraint: Once SSL is configured in Tableau Server via TSM, the original .key files are often not retained on disk for security reasons. Private keys cannot be exported from TSM. If they are not securely backed up externally, they cannot be recovered, and new certificates must be generated.
  • Security Hygiene: Regenerating certificates and keys during a Blue-Green transition ensures cryptographic material aligns with the new infrastructure and maintains a clean security posture.
• Mutual SSL / SAML: If SAML is used for Single Sign-On, the SAML Entity ID and/or metadata must be updated in the Identity Provider (e.g., Okta, Azure AD) during cutover. This is required because the application URL typically changes between Blue and Green environments and may require a new trust relationship.
• Licensing & Server ATR: Use Server ATR (Authorization-to-Run) to avoid activation conflicts when running Blue and Green environments simultaneously with the same license key. If using ATR, ensure the lease can be transferred appropriately between environments.
  • Maintenance & subscription validation: Verify that the customer’s Tableau maintenance or subscription is active and valid for the target upgrade version date.
  • Verify licensing capacity: Ensure sufficient test or non-production activations are available to support staging, validation, and parallel Blue-Green environments.
• Blue Environment: We recommend a minimum a week burn-in period (post-deployment period used to validate stability and performance under real workloads) before decommissioning the old environment. It will be also helpful to facilitate a comparison between old and upgrade environments in case of post upgrade performance issues.

Green Deployment Workflow

Phase 1: Preparation (The Blue Production environment side)

1. Change Freeze: Announce a period where no new workbooks or data sources should be published.
2. Generate Backup: tsm maintenance backup -f blue_backup.tsbak -d
3. Export Configuration: tsm settings export -f blue_configs.json
4. Topology Export: Document the output of tsm topology list-nodes -v.

Recommended Practice: We highly recommend securing and storing your backup files in an isolated network file share completely outside your blue/green deployment infrastructure.

Phase 2: Building Green

1. Install Tableau Server: Install the new version of Tableau Server on the Green hardware, replicating the topology of the Blue environment to ensure consistency across nodes and services.
2. Initialize TSM: During setup, ensure you use the same Identity Store (Local or AD) as the Blue environment.
3. Restore Data: Move the .tsbak to the Green server and run:
   tsm maintenance restore -f blue_backup.tsbak
4. Import Settings: Import the settings file: tsm settings import -f blue_configs.json
   Note: Some manual configuration (like SSL keys or specific paths) may still be required.
5. Apply Changes: tsm pending-changes apply will require a Tableau Server restart.

Phase 3: Validation & Testing

1. Disable Schedules: Immediately disable all extract refreshes and subscriptions on the Green server to prevent "double-emailing" or redundant DB load.
2. Smoke Testing: Verify that critical dashboards load and that the "Run As Service" account has proper permissions.
3. UAT: Have a small group of power users access the Green environment (via its direct IP/temporary URL) to validate data.

Phase 4: The Cutover

1. Stop Blue: Prevents users from publishing data during the final transition.
2. Final Sync / Final Restore: If the Blue environment stayed active for long, take a final "delta" backup and restore it to Green.
3. Traffic Switch: Update your DNS CNAME to point your production URL (e.g., tableau.company.com) to the Green server's IP. Alternatively: Swap the target group in your Load Balancer.
4. Re-enable Schedules: Turn the extract refreshes and subscriptions back on in the Green (now Production) environment.

2.3 Performance Comparison: Log & Metric Capture

The following comparison matrix identifies the recommended diagnostic tools for validating Tableau Server performance after an upgrade. The objective is to establish a performance baseline in the pre-upgrade Blue environment and compare it with the upgraded Green environment to identify regressions, resource bottlenecks, or changes in user experience.

This comparison ensures there are no performance regressions in key Tableau Server services such as VizQL, Backgrounder, Hyper, Flow Processing, and Gateway after the upgrade.

A baseline must first be established in the Blue environment, followed by the same measurements in the Green environment using identical workbooks, extracts, and workloads.

Performance Comparison Matrix

Recommended Performance Validation Tools

Recommended Validation Workflow

For a comprehensive upgrade validation, Tableau recommends combining multiple tools rather than relying on a single diagnostic utility. A typical workflow includes:

1. Capture a baseline before the upgrade:
   1. Generate log bundles using tsm maintenance ziplogs.
   2. Run LogShark against the baseline logs.
   3. Export Resource Monitoring Tool metrics (if available).
   4. Record dashboard performance using Performance Recording.
   5. Capture infrastructure metrics using TabMon.
2. Execute the upgrade.
3. Repeat the same workload after the upgrade:
   1. Execute the same dashboards and extract refreshes.
   2. Run identical TabJolt or Replayer workloads.
   3. Generate a second LogShark workbook.
   4. Capture another ziplogs bundle.
   5. Compare RMT metrics and Slow Views.
4. Compare the following key performance indicators:
   • Dashboard Load Time
   • Query Execution Time
   • Backgrounder Job Duration
   • Hyper Extract Performance
   • Tableau Prep Flow Runtime
   • CPU and Memory Utilization
   • HTTP Request Latency
   • Slow Views
   • Concurrent User Throughput (TPS)
   • Error Rate (HTTP 4xx/5xx and Tableau process errors)

2.4 Rollback Strategy (Green Environment — Post-Upgrade)

If a critical issue is identified in the Green (post-upgrade) environment, a controlled rollback to the Blue (previous stable) environment must be performed to restore service stability.

Green → Blue Rollback Action

• Redirect traffic from the Green environment back to the Blue environment using DNS update or Load Balancer failover.
• This ensures users are immediately routed back to the last known stable and validated production environment.

Post-Rollback Stabilization Steps (Green Environment)

After rollback (Green → Blue):

• Disable all scheduled jobs in the Green environment to prevent unintended processing, including:
  • Extract refreshes
  • Subscriptions
  • Flow executions
  • Backgrounder tasks
• Preserve logs and system state in Green for post-incident analysis (do not purge immediately).

Backup and Restore Requirements for Environment Switches

To ensure data consistency and prevent configuration drift during environment switching, the following rules apply:

• Blue → Green Promotion (Initial Cutover or Re-deployment)
  Before switching production traffic:
  • A fresh backup must be taken from the Blue environment
  • This backup must be restored into the Green environment
  • This ensures Green is fully aligned with Blue before it becomes production
• Green → Blue Rollback (After Extended Green Operation)
  If a rollback is required due to a critical issue identified in the Green environment after extended production use, validate Blue environment readiness:
  • A new backup must be taken from Green environment
  • This backup must be restored into the Blue environment
  • This ensures Blue is synchronized with any: New content changes, User updates, Extract refresh schedules or Configuration changes that occurred while Green was active.
• Future Environment Switching (Blue ↔ Green Iterations)
  For any subsequent switch between environments:
  • Always perform a fresh backup of the active production environment
  • Restore it into the target environment before switching traffic
  • This ensures both environments remain consistent, fully synchronized and safe for failover.

Please note: The environment receiving production traffic must always be restored from a recent, validated backup of the active production environment.

3.0 In-Place Upgrades: When Blue-Green Is Not Feasible

If a Blue-Green upgrade cannot be performed due to hardware capacity limitations or budget constraints for a parallel environment, an in-place upgrade is used as an alternative approach for upgrading Tableau Server.

In addition, a sandbox restore should be performed by restoring a full production Tableau Server backup to a small single-node development instance to validate the integrity of the .tsbak file and ensure it is not corrupted.

3.1 Backup Validation: Assets and Prerequisites

1. In an in-place upgrade, the backup is the only "undo" button. Do not proceed until the backup restore is verified on your sandbox instance. (Create a temporary, single-node VM, install the new version, and restore the current production full backup.)
2. Run-as-Service account password, the TSM admin credentials, and the SSL certificates backed up externally.
3. A JSON export of the current topology & configuration (tsm settings export -f settings.json).
4. Asset Freeze: A "Change Freeze" is in effect. Any workbook published after the backup but before the upgrade will be lost if you have to roll back.

3.2 Restore Testing Options (Risk Mitigation)

If a full-scale mirror environment is unavailable, lightweight testing methods such as a single-node VM-based restore test are used to ensure the restore process is functional.

• Backup Restore Testing: Create a temporary single-node VM with specifications similar to the Blue production environment (active repository node VM). Restore the backup on this VM and validate data integrity and basic site mapping.
  • Goal: Confirm the Tableau Repository (PostgreSQL) and File Store initialize correctly. The sandbox instance won’t match production specifications so don’t expect a match in performance. At this point you are just verifying that the backup can be used to restore a new environment successfully.
• External Asset Check: If there is an External File Store or External Repository (AWS RDS/Azure SQL), be sure you have a synchronized snapshot of those services that matches the timestamp of your Tableau Server backup. (External File Store Docs and External Repo Docs)

3.3 Operational Constraints & Checkpoints

The following constraints are "Go/No-Go" factors. If these are not met, the upgrade should be rescheduled.

3.4 The Recovery Strategy (The Exit Plan)

Should the upgrade encounter a fatal error after the "Point of No Return," the following protocol is triggered:

1. Environment Cleanup: Run the obliterate script to remove the corrupted/failed installation.
2. Clean Install: Install the Target Version of Tableau Server.
3. Restore: Use the Validated Backup to restore all content, users, and permissions.

The "Obliterate" Protocol

When the decision is made to wipe and reinstall, follow these steps to ensure the next attempt is "clean":

1. Verify Backup: Confirm you have the .tsbak and the settings.json file stored outside the Tableau server VMs.
2. Run Obliterate with Flags:
   tableau-server-obliterate.cmd -a -y -y -y
   Note: The -a flag is critical as it removes the data directory, preventing old, potentially compromised configuration files from being picked up by the reinstall.
3. The Reboot Requirement: You must reboot the OS after obliteration to clear pending file rename operations and hung memory locks. Failing to reboot often leads to the same failure on the second attempt.

4.0 Actions When an Issue Is Discovered During an Upgrade

4.1 Immediate Data and Log Preservation

Goal: Capture the "crime scene" before it’s wiped by a rollback or restore.

Log Sets & Collection: To ensure Tableau Support can actually diagnose the root cause, you need to gather more than just the standard Ziplogs.

1. Primary Tool: Use the tsm maintenance ziplogs command.
2. Collection Timing: Execute immediately after the failure message appears and before running any obliterate or restore commands.
3. Required Data Points:
   • The "Ziplogs" Archive: Run tsm maintenance ziplogs -f <filename>.zip to include historical logs.
   • Upgrade Logs: Specifically capture the app-upgrade.log found in the tableau server logs directory of the new version's data folder.
   • Environment Snapshot: Record the OS system / Application event logs at the time of failure.

Note: If the TSM controller is unresponsive, manually compress & collect the C:\ProgramData\Tableau\Tableau Server\data\tabsvc\logs folder (Windows) or /var/opt/tableau/tableau_server/data/tabsvc/logs (Linux).

4.2 Primary Recommendation: Roll Back or Restore Immediately

Goal: Minimize "Mean Time to Recovery" (MTTR) by prioritizing availability over curiosity.

The Decision Matrix

4.3 Critical Decision: Obliterate vs. Troubleshoot

The Tableau Server Upgrade Recovery Framework must determine whether a failed upgrade is recoverable or whether the environment has reached a state where reinstallation and restoration from backup is the recommended recovery strategy.

While many upgrade failures can be resolved through targeted remediation (see Scenario 3 – Recoverable Upgrade Failures), certain failures indicate corruption of core platform components such as the Coordination Service, Tableau Services Manager (TSM), or Repository. In these situations, continued troubleshooting may significantly increase production downtime without improving the likelihood of a successful recovery.

The objective of this decision point is to identify failures where Tableau no longer has a reliable operational state, making an obliterate and clean installation followed by restore from backup the recommended recovery path.

Please Note: An obliterate operation permanently removes the Tableau Server installation and all local configuration. Before performing this operation, verify that a valid Tableau Server backup (.tsbak) and any required external assets are available.

Scenario 1: Fatal "Point of No Return" Errors

The following failures indicate that Tableau Server may no longer be recoverable through standard troubleshooting procedures. Review app-upgrade.log, tabadminagent.log, tabadmincontroller.log, and relevant TSM logs to confirm the failure state.

Scenario 2: Upgrade Retry Decision ("Three-Strike Rule")

Repeatedly executing the upgrade script after an unrecoverable failure rarely changes the outcome and can increase recovery time.

Scenario 3: Recoverable Upgrade Failures (No Reinstallation Required)

Some upgrade failures occur after the Tableau Server binaries and services have already been successfully upgraded. In these cases, the server remains in a recoverable state and does not require an obliterate, reinstall, or restore from backup.

More Details: Troubleshoot Tableau Server Install and Upgrade

5.0 Escalation and Decision Points

Trigger Points for Escalation: Escalate immediately when any of the following are observed:

• Upgrade logs (app-upgrade.log) indicate a failed or incomplete upgrade.
• Active Directory or authentication errors prevent service continuity.
• Upgrade scripts fail validation or return critical errors.
• Environment constraints (e.g., disk space, CPU, or network access) prevent recovery or completion of the upgrade.
• Repeated failed upgrade attempts result in an unstable server state.
• TSM services fail to start after upgrade, or key processes do not initialize.
• Timeout or service initialization issues occur across clustered nodes.

6.0 Key Takeaways and Core Principles

1. The "Blue-Green" Gold Standard
   • Blue-Green upgrades are the safest and preferred approach.
   • Parallel environments ensure zero impact on production, allowing for full validation before cutover.
   • This method is the only supported and recommended approach for OS migrations and cloud infrastructure transitions.
2. Safeguarding Your Data Assets (Verified Backup Readiness)
   • Backups are critical safeguards but are only effective when properly verified.
   • Maintaining reliable backups ensures data assets remain protected and recoverable in all scenarios.
   • Upgrades require a full Tableau Server backup (.tsbak), and all associated asset keys must be securely stored.
   • Integrity checks using tsm maintenance restore in a test environment are mandatory to confirm backup viability before proceeding.
3. "Return to Service" (RTS) Priority
   • Restoring service availability takes precedence over root cause analysis (RCA) during an outage. RCA and log analysis can be performed after service restoration.
4. The Point of No Return
   • If upgrade scripts fail, services remain unstable for 2–3 hours, or troubleshooting becomes prolonged, a rollback to the last stable version should be initiated.
5. Clean Restoration Approach:
   • Instead of attempting to repair a corrupted state, perform a clean recovery:
   • Run the obliterate script to remove the failed installation
   • Reinstall the stable Tableau Server version
   • Restore from the verified “golden” backup
6. Time-Critical Actions
   • Swift intervention prevents the rotation or overwrites of critical logs (e.g., ziplogs, tabadmincontroller.log) which are essential for Tableau Support diagnostics.
7. Continuous Improvement (The Post-Mortem)
   • Progressive Accountability: Each upgrade is a learning opportunity and should not be treated as “fix and forget”.
   • Honest Analysis: If a recovery takes longer than planned, a formal post-mortem should be conducted to identify the cause.
   • Updating the Playbook: Findings from the post-mortem are incorporated into Pre-Flight Checklists and Disaster Recovery (DR) runbooks.