How to send Mule CloudHub application log entries to Splunk

In this document, we are going to explain the potential methods for dealing with log forwarding to Splunk.

1. Using the Log4j2 Appender with TCP or UDP

In order to enable the log4j appender with TCP or UDP, you will need to:

• Disable CloudHub Logs as explained in Integrate with Your Logging System Using Log4j.
• After that, please add the following snippet to your log4j file

<Appenders>
    <Socket name="socket" host="127.0.0.1" port="15000">
        <PatternLayout pattern="%p: %m%n" charset="UTF-8"/>
    </Socket>
</Appenders>

• You will need to create a Splunk Enterprise TCP input from Data > Data inputs > TCP. Please follow Get data from TCP and UDP ports. For further detail, please see the steps listed in Enable logging to TCP inputs in your Java project. These are the steps given by Splunk for Log4j2 in general, and so not all steps may be applicable to Mule.

2. Using Log4j2 Appender with HTTP and token (please check here before using this method: CloudHub | Recommended Log Appenders for Third-Party Log Aggregator Services

This is accomplished in a similar way to the steps above under "Using Log4j2 Appender with TCP or UDP".

• Disable CloudHub Logs as explained in Integrate with Your Logging System Using Log4j.
• However, for your log4j2.xml file, please modify the Appenders section to use the following snippet.

<Appenders>
...
            <Http name="HttpAppender" url="https://your.log.server.com/api/logs?api-version=2016-04-01" method="POST">       
              <Property name="token" value="insert_token_here" />    
            </Http>     
...
</Appenders>

• Again, add the org.slf4j and org.apache.logging.log4j dependencies to the log4j2.xml for the application.

3. The Splunk Appender for Log4j

In order to use the Splunk appended, you will need to

• Disable CloudHub Logs as explained in Integrate with Your Logging System Using Log4j.
• Add the Splunk Appender to the log4j2.xml of the application
• WARNING: batch_size_count should be higher if your app is doing heavy logging so that you send all log lines once they reach a threshold instead of "1" where you make a HTTP request for every single log line.

<SplunkHttp
    name="Splunk"
    url="https://yourinstance.splunkcloud.com:<port>/"
    token="YOUR-TOKEN" 
    batch_size_count="1"
    disableCertificateValidation="true"/>

     Be note that the port in URL above is the default port number. Please check with Splunk admin to ensure the port being used. 
     Here's an example of SplunkHttp setting with default port 8088.

<SplunkHttp name="Splunk" url="https://<instanceID>.splunkcloud.com:8088" token="<token>" index="demoindex"  batch_size_count="1" disableCertificateValidation="true"/>     

<SplunkHttp name="Splunk" url="https://13.xxx.xxx.xxx:8088" token="<token> index="demoindex1"  batch_size_count="1" disableCertificateValidation="true"/>

• You need to ensure the Splunk dependency is added to applications pom.xml. The latest version is available on GitHub

<dependency>
     <groupId>com.splunk.logging</groupId>
    <artifactId>splunk-library-javalogging</artifactId>
    <version>x.x.x</version>
</dependency>

• And add the packages for "com.splunk.logging" and "org.apache.logging.log4j" in your log4j2.xml's Configuration tag:

<Configuration status="INFO" name="cloudhub" packages="com.splunk.logging,org.apache.logging.log4j">

<Configuration status="INFO" name="cloudhub" packages="com.mulesoft.ch.logging.appender,com.splunk.logging,org.apache.logging.log4j">

• Ensure the following repository is added to the pom.xml

<repository> 
        <id>splunk-artifactory</id> 
        <name>Splunk Releases</name> 
        <url>https://splunk.jfrog.io/splunk/ext-releases-local</url> 
</repository>

• Add Splunk appender reference to AsyncRoot configuration.

<AsyncRoot level="INFO">
...
    <AppenderRef ref="Splunk"/>
</AsyncRoot>

4. Pushing log entries programmatically

Please note that you will need to develop and support this application yourself. An attached example is provided, but the example is provided as a reference for your own usage and is not to be considered a MuleSoft product. The attached example should be considered as a custom solution and as such is not supported by MuleSoft Global Support.

It is possible, but not suggested, that you can use a method that decouples the application from the logs. Basically, it consists of an independent Mule application that gets the logs from CloudHub and sends them to Splunk. With this approach, we don’t need to modify the application that needs to send logs to Splunk, and won’t affect its performance while logging. Instead, you can use another application that uses the CloudHub API will retrieve all the desired logs to send them to Splunk using TCP API or HTTP API. This could be a Mule application or any other custom application if you prefer. Again, there is the need to create a Splunk Enterprise TCP input from Data > Data inputs > TCP, see Get data from TCP and UDP ports.

First, you will need to define the frequency to retrieve application/s logs. For this reason, we defined a Poll component as the Input of our main flow with a Fixed Frequency schedule. Then we need to obtain the access token to consume CloudHub Enhanced Logging API; using the HTTP Transport we can do a POST to /accounts/login using two required query parameters: username and password.

The connector configuration needs the following settings:

Protocol: HTTPS
Host: anypoint.mulesoft.com
Port: 443

From the response, we need to get the bearer for future requests. If you only want to log to Splunk a specific application, you can hardcode its domain. However, if you are planning to move all your application’s logs to Splunk you can use HTTP Request again with the GET method this time using path /cloudhub/api/v2/applications and the following two headers: Authorization (‘Bearer’ + access token obtained before) and X-ANYPNT-ENV-ID (The Environment ID as described in How to get a list of applications via Anypoint platform API)

After that, we need to iterate each result (with the “For Each” component) and using the application domain get the logs for that application.

We use an Object Store to get the last record processed for each application and don’t send to Splunk the same log twice. Using a HashMap we put the following parameters: upperId (empty), lowerId (obtained from Object Store), descending (false), and the limit (a limit number that will be highly correlated with your logging level and the frequency of this application to retrieve logs, i.e., 1000). Using the Object to JSON transformer and this JSON will be the input of another HTTP Transport using POST. As parameters, we will have 3 headers:

Authorization (‘Bearer’ + access token obtained before)
X-ANYPNT-ENV-ID (Environment ID)
Content-Type (application/json).

If there are results (new logs) increase the number of logs retrieved to update lowerId, so that in the next iteration the same logs are not retrieved again.

Finally, we need to push these logs to Splunk, we used Batch Module for this with two steps:

• one to parse the logs (this must be done based on your needs)
• one to send them to Splunk with a TCP outbound endpoint using as host your Splunk IP and as port your Splunk Port.

With regard to disabling the CloudHub Logs.

To allow you to disable the CloudHub log for your organization, you will need to open a support case via the Help Center, which will allow you to disable the CloudHub Logs in the specific CloudHub Application. As disabling CloudHub (as explained in Integrate with Your Logging System Using Log4j) removes the ability to perform CloudHub Logging, we recommend you also include the Log4j2CloudhubLogAppender to your log4j configuration for supportability purposes. 

Required java package dependencies

You will find that you need to add the following dependencies to your pom.xml. We recommend you seek the latest versions of these

<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>x.x.x</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>x.x.x</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>x.x.x</version>
</dependency>

Disclaimer

This article involves products and technologies which do not form part of the MuleSoft product set. Technical assistance for such products is limited to this article.