Proactive Monitoring a Specific On-Prem Mule Application Log for Errors

GOAL

This script is a targeted tool designed to monitor a single application's log file (e.g., demo-hello.log) for specific, user-defined error patterns. When a configured error message appears in the log, the script automatically captures diagnostic information (a series of thread dumps and a heap dump) from the Mule runtime hosting the application.

This approach is ideal for troubleshooting intermittent or hard-to-reproduce issues within a specific application without the overhead of full system-level monitoring.

PROCEDURE

Prerequisites:

• A Linux or macOS-based environment.
• A working Java with its bin directory in the system's PATH.
• The following JDK tools must be available: jps, jstack, jmap.
• The user running the script must have read permissions on the target application LOG_FILE.
• The user running the script must have write permissions in the OUTPUT_DIR.

Steps for Implementation & Use

1. Save the Script: Save the code to a file named app_monitor.sh.
2. Make it Executable: Open a terminal and run chmod +x app_monitor.sh.
3. Configure Variables: Edit the script and update the following required variables in the "Configuration" section:
   • LOG_FILE: The full path to the specific application log you want to monitor (e.g., /opt/mule/logs/my-specific-api.log).
   • OUTPUT_DIR: The directory where all logs and diagnostic dumps will be saved.
   • SEARCH_PATTERNS: Customize the array with the exact error strings from your application that should trigger a diagnostic capture. Please refer to "Note on SEARCH_PATTERNS".
   • CAPTURE_LIMIT: Set how many times the script should capture dumps before it automatically stops.
4. Run the Script: To run it in the background so it continues after you close your terminal, use
   nohup: nohup ./app_monitor.sh &
5. Monitor its Output: You can view the script's own log messages (e.g., when a pattern is detected) by running:
   tail -f /path/to/your/OUTPUT_DIR/monitorOut.log
6. Stop the Script: Find the script's PID (ps -ef | grep app_monitor.sh) and send it a kill signal (kill <PID>). The script's cleanup trap will automatically handle shutting down all related processes. If the script is unresponsive, use
   kill -9 <PID>

Note on SEARCH_PATTERNS
This array is crucial. Only specify necessary and unique error strings. If the patterns are too generic, the CAPTURE_LIMIT will be reached on non-critical issues. Once the limit is hit, the monitoring will stop, and you will lose the chance of collecting data (heap and thread dumps) at the right time for a more important event.

Attachment: monitor_app_v2.sh.zip

Critical Usage Guidelines

• Full Responsibility: The user assumes all responsibility for the use and impact of this script.
• Mandatory Testing: This script must be thoroughly tested in a non-production (lower) environment before being considered for use in any production environment.
• Required Customization: The user is required to review and modify the script's configuration (file paths, error patterns, thresholds) to fit their specific requirements and system architecture. Do not run this script unmodified.

Disclaimer
This script is provided as an EXAMPLE ONLY and is not officially supported. It is the end user's full responsibility to understand, test, and adapt the script for their specific environment. Run this script at your own risk. The provider of this script disclaims all responsibility for any consequences resulting from its use.