Loading

Apache Log4j2 Vulnerability (Log4shell) - Tableau Server Mitigation Steps

Publiceringsdatum: Apr 13, 2024
Beskrivning

Disclosed vulnerabilities allow for remote code execution in products that use the Log4j Apache library.

Cause

Security vulnerabilities CVE-2021-44228 & CVE-2021-45046 in Apache Log4j 2 library.

Lösning

Option 1: Update Tableau

 

For customers with active maintenance, if you have not updated from an impacted version (any product release prior to December 15, 2021), or have updated to the December 15, 2021 product release, please update to one of the newer releases:

IMPORTANT NOTE: After updating to the December 19, 2021 product releases, or newer, there may be cases where legacy files from previously configured processes are present on the file system. For example, after making topology changes to change process counts or moving processes to another Tableau Server node. Versions of log4j-core-2.15 or earlier are safe to delete. These files are never accessed as part of Tableau Server operation and are safe to remove if desired. Alternatively, redeploying the services will also clean up the directories and deploy the updated files (see Configure Node Online Help Guide).
 
The December 15, 2021 Tableau Product releases updated the Log4j2 files to version 2.15. There may be diagnostic or auxiliary components still remaining. We have mitigated these outstanding components with configuration changes that disable the vulnerable JNDI lookup functionality.
The December 19, 2021 Tableau Product release, has integrated the Log4j 2.16 release, which disables JNDI Lookup by default. This action addresses both CVE-2021-44228 & CVE-2021-45046.
  • Tableau Server 2021.4.2+, 2021.3.6+, 2021.2.7+, 2021.1.10+, 2020.4.13+
By updating to the product releases from December 19, 2021, you are addressing the security issues currently identified in CVE-2021-44228 & CVE-2021-45046

 

Option 2: Please execute the mitigation steps detailed in Option 2 if:

You have updated to the product release from December 15, 2021, and cannot update to a newer release (out of maintenance, outside of a company update window, etc.).
You are on an impacted version (any product version released prior to December 15, 2021) and cannot update to a newer release.

NOTE:
  • These steps are not meant to be executed on December 19, 2021 product releases, which updated log4j to 2.16.
  • These mitigation steps are for Tableau Server versions v2020.1 - v2021.4 prior to the December 19, 2021 product release and newer releases.

These instructions remove the jndilookup.class from vulnerable jars. There are two components that don’t work with the removal of that function and we replace those specific components jars with 2.16. Python is required to run the script. The script ensures we find all instances of the JndiLookup.class file even inside nested java libraries. 

Tableau Server

NOTE:
  • Plan for at least an hour of downtime as we need to scan every java archive to ensure effective remediation. Please do not deviate from the steps by attempting to manually modify any log4j files. While these steps have been tested on all supported major versions as well as 2020.1-2020.3, Tableau suggests snapshotting your server and taking a backup before starting any manual modifications.
  • If you have updated to the product releases from December 19, 2021, or newer, you do not need to take actions as we have disabled JNDI lookup for these versions.

Tableau Server for Windows

To better view the steps showed in the below video, expand the section. Note: the video has no sound.

Step 1: On each node perform the following


1. Download and install Python 3.10: https://www.python.org/downloads/

2. Under “Download the latest version for Windows,” select “Download Python 3.10.1”
 
Direct link: Download Windows Installer (64-bit) https://www.python.org/ftp/python/3.10.1/python-3.10.1-amd64.exe

3. Go to the location you downloaded it to, Ex: C:\users\<username>\Downloads

4. Right-click on Python-3.10.1-amd64 and select “Run as administrator”

5. When Setup launches, select “Add Python 3.10 to path”

6. Select either "Install now" or "Customize the installation" to configure alternate install path if needed

If you see the error, “This installation is forbidden by system policy,” re-run from downloads using right-click “Run as Administrator”

7. At the end of the installation select “Disable path length limit”

8. Download the required Python script and Log4j version 2.16 jars for Windows in the remove_jndi.zip attached to this article, which contains:

  • log4j-api-2.16.0.jar
  • log4j-core-2.16.0.jar
  • log4j-slf4j-impl-2.16.0.jar
  • remove_jndilookup_win.py
8a. Open a new administrative command prompt to ensure Python is in the path

8b. Create the directory c:\remove_jdni by entering the following command
 
mkdir c:\remove_jndi
 
8c. Move above four files into this directory. Note: This directory C:\remove_jndi will be referenced as the location of these files in the instructions below
 
If this is the first node on which you are executing these instructions, stop Tableau Server:
 
tsm stop
 

Step 2: Stop the administrative services 

1. After Tableau Server has stopped, stop the administrative services on each node by running the stop-administative-services.cmd from within \packages\scripts.<version> directory. By default this is:
 
Example path “C:\Program Files\Tableau\Tableau Server\packages\scripts.<version>”

2. Change directory to the scripts<version> directory inside the packages folder within your installation:
 
The command below uses an environment variable to resolve to any version's specific path.

cd "%TABLEAU_SERVER_INSTALL_DIR%\packages\scripts.%TABLEAU_SERVER_DATA_DIR_VERSION%"

3. Run the stop administrative services script:
 
a. stop-administrative-services.cmd

b. Run the remove_jndilookup script to remove the jndilookup classes from the Tableau server location you installed to:
 
python.exe c:\remove_jndi\remove_jndilookup_win.py "C:\Program Files\Tableau\Tableau Server"

Please don’t add a “\” to the end of the path, or you may hit an error.

At the end of each run of remove_jndilookup_win.py, you will see logging with the files the JndiLookup.class was removed from. Example:

“Findings:

Detected org/apache/logging/log4j/core/lookup/JndiLookup.class in C:\Program Files\Tableau\Tableau Server\<path to file>“

c. If installed to the default location, you need to run a second time:
 
python.exe c:\remove_jndi\remove_jndilookup_win.py "c:\ProgramData\Tableau\Tableau Server"
 
Please don’t add a “\” to the end of the path, or you may hit an error.


For versions 2020.1 through 2020.3 and December maintenance releases dated December 15, 2021 (2021.4.1, 2021.3.5, 2021.2.6, 2021.1.9, 2020.4.12) skip to Step 4, Start Administrative Services
 

Step 3: For versions 2020.4 and newer, except for the December Maintenance releases, copy the Log4j 2.16 files into the elasticsearch and solr directories once the Python script has been run (twice if installed to the default location).


1. Find your elastic search directory within your installation
 
a. Example: C:\Program Files\Tableau\Tableau Server\packages\elasticsearch.20213.21.0917.1006

b. cd "%TABLEAU_SERVER_INSTALL_DIR%\packages\elasticsearch.%TABLEAU_SERVER_DATA_DIR_VERSION%\lib"

2. While inside the elasticsearch.%TABLEAU_SERVER_DATA_DIR_VERSION%\lib directory, copy the log4j-core-2.16.0.jar and the log4j-api.2.16.0.jar to the current directory
 
NOTE: The instructions below use a period to mean ”current directory“.

a. copy c:\remove_jndi\log4j-core-2.16.0.jar .

b. copy c:\remove_jndi\log4j-api-2.16.0.jar .

3. While still inside the elasticsearch.%TABLEAU_SERVER_DATA_DIR_VERSION%\lib directory, remove the old versions
 
a. del log4j-api-2.11.1.jar

b. del log4j-core-2.11.1.jar

4. Then change to the plugins\search-guard7 directory under “C:\Program Files\Tableau\Tableau Server\packages\elasticsearch.%TABLEAU_SERVER_DATA_DIR_VERSION%”
cd "%TABLEAU_SERVER_INSTALL_DIR%\packages\elasticsearch.%TABLEAU_SERVER_DATA_DIR_VERSION%\plugins\search-guard-7"
 
5. Remove the old jar:
 
del log4j-slf4j-impl-2.11.1.jar

6. Copy the new jar to the old jar’s name:
 
copy c:\remove_jndi\log4j-slf4j-impl-2.16.0.jar log4j-slf4j-impl-2.11.1.jar
 
7. Change directory to the packages\solr7.%TABLEAU_SERVER_DATA_DIR_VERSION%\server\lib\ext folder

cd "%TABLEAU_SERVER_INSTALL_DIR%\packages\solr7.%TABLEAU_SERVER_DATA_DIR_VERSION%\server\lib\ext"
 
8. Copy all three 2.16.0 jars to the current working directory (represented by a period):

copy c:\remove_jndi\log4j*.jar .
 
9. There is one additional jar to delete in the elasticserver directory; in a default installation the path to the jar is under ProgramData. Change directory to: “C:\ProgramData\Tableau\Tableau Server\data\tabsvc\services\elasticserver_0.%TABLEAU_SERVER_DATA_DIR_VERSION%\plugins\search-guard-7”
 
 
cd "%TABLEAU_SERVER_DATA_DIR%\data\tabsvc\services\elasticserver_0.%TABLEAU_SERVER_DATA_DIR_VERSION%\plugins\search-guard-7"
 
 
10. Delete the log4j-slf4j-impl-2.11.1.jar
 
del log4j-slf4j-impl-2.11.1.jar

Recommended step: Verify that there are no remaining JndiLookup.class files earlier than version 2.16 in your product.

1. Run the remove_jndilookup with the “--dryrun” flag to instruct the script to scan the directories, but do not take action on the files:

python.exe c:\remove_jndi\remove_jndilookup_win.py --dryrun "C:\Program Files\Tableau\Tableau Server" > verification.txt

python.exe c:\remove_jndi\remove_jndilookup_win.py --dryrun "c:\ProgramData\Tableau\Tableau Server" >> verification.txt


2. If the "Findings:" section of the dry run output in verification.txt contains no entries, this means it did not find any more jars that contained the JndiLookup.class


Step 4: Start administrative services, apply a configuration change to ensure our changes are propagated, and then start the server.

 
1. On each node, start the TSM administrative services:
 
a. cd "%TABLEAU_SERVER_INSTALL_DIR%\packages\scripts.%TABLEAU_SERVER_DATA_DIR_VERSION%"
 
b. start-administrative-services.cmd
 
NOTE: Once every Tableau Server node has started the administrative services, we need to make a configuration change to regenerate the war files:

2. Back on the initial node, identify the current value for gateway.timeout:
 
tsm configuration get -k gateway.timeout
 
3. Increment the value of gateway.timeout by 1:
 
tsm configuration set -k gateway.timeout -v <current value +1, example 7201 if set at default>

4. Apply the pending changes
 
tsm pending-changes apply

5. Start Tableau Server
 
tsm start
 
The JdniLookup.class files should now be removed from all .jar and .war files on Tableau Server.

Tableau Server for Linux

DO NOT RUN THE BELOW STEPS AS root. Use sudo instead.

NOTE: These instructions assume Python is installed on each node in the cluster.

Step 1: On each node perform the following


1. Download the required Python script and Log4j version 2.16 jars for Linux in the remove_jndi_linux.tar.gz attached to this article which contains:
  • log4j-api-2.16.0.jar
  • log4j-core-2.16.0.jar
  • log4j-slf4j-impl-2.16.0.jar
  • remove_jndilookup.py
2. Unzip remove_jndi_linux.tar.gz
 
sudo tar -xzvf remove_jndi_linux.tar.gz
 
3. Make a directory named /elasticJars in your home directory and move the 3 jar files to the /elasticJars directory:
 
mkdir ~/elasticJars
mv ./log4j* ~/elasticJars

4. Set the permissions of the jar files in elasticJars to -rw-r--r--
 
sudo chmod -R 644 ./elasticJars/

5. Set the permissions of the remove_jndilookup.py script to Execute

sudo chmod 744 ./remove_jndilookup.py

6. Stop Tableau Server

tsm stop
 
7. Ensure the environment variable TABLEAU_SERVER_DATA_DIR_VERSION is available in your environment.

env | grep TABLEAU_SERVER_DATA_DIR_VERSION

7a. If the previous command does not return a version, run the following command to set the variable for this session

export $(cat /etc/opt/tableau/tableau_server/environment.bash | grep TABLEAU_SERVER_DATA_DIR_VERSION | xargs)
 
7b. Verify TABLEAU_SERVER_DATA_DIR_VERSION is now available in your environment

env | grep TABLEAU_SERVER_DATA_DIR_VERSION
 

Step 2: Stop the administrative services for each node

NOTE: Repeat all of the following steps on each node. These steps assume the default installation location: /opt/tableau/tableau_server/packages
 

1. Stop the administrative services

sudo /opt/tableau/tableau_server/packages/scripts.$TABLEAU_SERVER_DATA_DIR_VERSION/stop-administrative-services

2. Run the remove_jndilookup.py script to remove the JndiLookup.class files from the jars in the packages directory.

sudo ./remove_jndilookup.py /opt/tableau/tableau_server/packages

NOTE: Once the remove_jndilookup.py script completes, run the script a second time against the data directory. By default /var/opt/tableau/tableau_server.
 

3. First, make a copy of the script in the home directory of the unprivileged user, which is named tableau by default

sudo cp ./remove_jndilookup.py /var/opt/tableau/tableau_server

4. Change ownership of the script to the tableau user and group
 
 
sudo chown tableau:tableau /var/opt/tableau/tableau_server/remove_jndilookup.py
 

5. Start a session as the unprivileged user

sudo su -l tableau

6. Run the script to remove the JndiLookup.class files from the jars in the data directory

./remove_jndilookup.py /var/opt/tableau/tableau_server

7. Exit the Tableau shell

exit
 

For versions 2020.1 through 2020.3 and December maintenance releases dated December 15, 2021 (Tableau 2021.4.1, 2021.3.5, 2021.2.6, 2021.1.9, 2020.4.12) skip to Step 4, Start Administrative Services
 

Step 3: For versions 2020.4 and newer, except for the December maintenance releases, copy the Log4j 2.16 files into the elasticsearch directory once the python script has been run twice.

 
1. On each node, copy the jars from ~/elasticJars to the destinations and remove the old jars

sudo cp ./elasticJars/log4j-api-2.16.0.jar /opt/tableau/tableau_server/packages/elasticsearch.$TABLEAU_SERVER_DATA_DIR_VERSION/lib/

sudo cp ./elasticJars/log4j-core-2.16.0.jar /opt/tableau/tableau_server/packages/elasticsearch.$TABLEAU_SERVER_DATA_DIR_VERSION/lib/

sudo cp ./elasticJars/log4j-slf4j-impl-2.16.0.jar /opt/tableau/tableau_server/packages/elasticsearch.$TABLEAU_SERVER_DATA_DIR_VERSION/plugins/search-guard-7/log4j-slf4j-impl-2.11.1.jar

sudo rm /var/opt/tableau/tableau_server/data/tabsvc/services/elasticserver_0.$TABLEAU_SERVER_DATA_DIR_VERSION/plugins/search-guard-7/log4j-slf4j-impl-2.11.1.jar

sudo rm /opt/tableau/tableau_server/packages/elasticsearch.$TABLEAU_SERVER_DATA_DIR_VERSION/lib/log4j-api-2.11.1.jar

sudo rm /opt/tableau/tableau_server/packages/elasticsearch.$TABLEAU_SERVER_DATA_DIR_VERSION/lib/log4j-core-2.11.1.jar
 

Recommended step: Verify there are no remaining JndiLookup.class files earlier than version 2.16 in your product. 

You may then run the remove_jndilookup with the --dryrun flag to have the script scan the directories, but not take any action on the files:

1. Run the remove_jndilookup.py script with the --dryrun flag against the install directory to have the script scan the directory, but not take any action on the files.

sudo ./remove_jndilookup.py --dryrun /opt/tableau/tableau_server/packages > verification.txt

NOTE: Once the remove_jndilookup.py script completes, run the script a second time against the data directory to validate the data directory. By default /var/opt/tableau/tableau_server.

2. First, make a copy of the script in the home directory of the unprivileged user, which is named tableau by default

sudo cp ./remove_jndilookup.py /var/opt/tableau/tableau_server

3. Change ownership of the script to the tableau user and group

 sudo chown tableau:tableau /var/opt/tableau/tableau_server/remove_jndilookup.py

5. Start a session as the unprivileged user

sudo su -l tableau

6. Run the script with the --dryrun flag against the data directory to have the script scan the directory, but not take any action on the files.

./remove_jndilookup.py –dryrun /var/opt/tableau/tableau_server > datadirverification.txt

If the "Findings:" section of the dry run output in verification.txt and datadirverification.txt contains no entries then it did not find any more jars that contained the JndiLookup.class
 

7. Exit the Tableau shell

exit
 

Step 4: Start Administrative Services

 
1. On each node start the administrative services:

sudo /opt/tableau/tableau_server/packages/scripts.$TABLEAU_SERVER_DATA_DIR_VERSION/start-administrative-services

NOTE: Once every Tableau Server node has started the administrative services we need to make a configuration change to regenerate the war files:

2. 2. Back on the initial node, identify the current value for gateway.timeout:

tsm configuration get -k gateway.timeout

3. Increment the value of gateway.timeout by 1:

tsm configuration set -k gateway.timeout -v <current value +1, example 7201 if set at default>

4. Apply the pending changes
 
 
tsm pending-changes apply
 

5. Start Tableau Server

tsm start
 
The JdniLookup.class files should now be removed from all .jar and .war files on Tableau Server
 

Option 3: Mitigate the vulnerability by setting the ‘System Environment Variable’ for Tableau Server versions v2020.4 and newer (for Tableau Server only).

NOTE: This is a temporary and partial mitigation for CVE-44228 that will modify the environment variable for all Java processes on the machine.
 

Linux Instructions


1. Stop Tableau Server

tsm stop

2. Stop the TSM administrative services. Run the following script:

sudo /opt/tableau/tableau_server/packages/scripts.<version>/stop-administrative-services

3. Change to the unprivileged user (the default user is tableau)

sudo su -l tableau

4. Create the log4j.conf file with the variable to mitigate the vulnerability

echo LOG4J_FORMAT_MSG_NO_LOOKUPS=true >> ~/.config/systemd/tableau_server.conf.d/log4j.conf

5. Exit the Tableau shell. Run the following command:

exit

6. Start the TSM administrative services. Run the following script:

sudo /opt/tableau/tableau_server/packages/scripts.<version>/start-administrative-services

7. Repeat steps 2 through 6 for each node in the cluster if running a multi-node deployment
 

8. Start TSM

tsm start

9. After restart run the command pgrep -l run- to obtain the pid of one of the Tableau processes
 

10. Run sudo strings /proc/<pid>/environ where <pid> is one of the pids returned from step 9
 

11. Verify LOG4J_FORMAT_MSG_NO_LOOKUPS=true exists in the output of step 10
 
 

Windows Instructions

 

1. Open an Administrative PowerShell window
 

2. On the initial node, stop Tableau Server

tsm stop

3.On each node stop the TSM administrative services. Run the following script:

C:\Program Files\Tableau\Tableau Server\packages\scripts.<version>\stop-administrative-services.cmd

4. Create a system environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS and set it to true, a shortcut to do this with PowerShell is:
 
[System.Environment]::SetEnvironmentVariable(‘LOG4J_FORMAT_MSG_NO_LOOKUPS’,‘true’,[System.EnvironmentVariableTarget]::Machine)

5. Ensure steps 3 and 4 were completed on each node in the cluster if running a multi-node deployment
 

6. On each node start the TSM administrative services. Run the following script:

C:\Program Files\Tableau\Tableau Server\packages\scripts.<version>\start-administrative-services.cmd

7. On the initial node, start Tableau Server

tsm start

8. Open an administrative cmd window and verify the environment variable exists by running the command set. In the command output the variable LOG4J_FORMAT_MSG_NO_LOOKUPS=true should exist
Ytterligare resurser
Please see Salesforce Trust Site for more information detailing Tableau Cloud status.
For more information, see:
Knowledge-artikelnummer

001534225

Bilagor

remove_jndi.zip

1850 KB

remove_jndi_linux.tar.gz

1846 KB

 
Laddar
Salesforce Help | Article