When a Java application runs out of memory, the JVM generates a heap dump file (
.hprof). By default, Tomcat writes this file to its working directory (often /usr/share/tomcat/default on the / root partition). Because heap dumps can easily exceed several MegaBytes to Gigabytes, they can quickly fill up the root filesystem to 100% and cause system instability.
If you have identified a large heap dump file as the cause of high disk usage on your / partition, follow these steps to redirect future heap dumps to a dedicated mount point (e.g., /app/heapdumps).
Prerequisites
Create a backup: Take a snapshot of the server before modifying system configurations.
Verify storage: Ensure your custom mount point (such as /app) has sufficient free space to hold large .hprof files.
Steps:
- Create the target directory and set permissions
Create a dedicated directory on your custom mount point and assign ownership to thetomcatuser so the JVM has permission to write to it:# mkdir -p /app/heapdumps # chown -R tomcat:tomcat /app/heapdumps
Move or remove existing heap dump files
Locate the existing.hproffile in your Tomcat directory (typically/usr/share/tomcat/default) and either move it to the new mount point for analysis# mv /usr/share/tomcat/default/java_pid*.hprof /app/heapdumps/- Locate the Tomcat environment configuration file
Find the file whereCATALINA_OPTSorJAVA_OPTSis defined. Depending on your Linux distribution, this is usually located in/usr/share/tomcat/service/tomcat.conf, or inside the systemd service unit file, look for the path defined for tomcat.conf (/etc/systemd/system/tomcat.service). Look for the existing flag:-XX:+HeapDumpOnOutOfMemoryError
- Configure the custom heap dump path
Add the-XX:HeapDumpPathparameter right next to-XX:+HeapDumpOnOutOfMemoryErroralong with other parameters:CATALINA_OPTS="$CATALINA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/app/heapdumps
Reload systemd and restart Tomcat
# systemctl daemon-reload # systemctl restart tomcat
Verify the configuration
Confirm that the Tomcat process is running with the updated JVM flags:
ps -ef | grep -i java
Look for -XX:HeapDumpPath=/app/heapdumps in the output.
You can verify this parameter added,
[root@jetpack-app-prod01 default]# ps -ef | grep -i java
tomcat 1419458 1 14 14:37 ? 00:00:37 /usr/lib/jvm/java-11-openjdk-11.0.25.0.9-7.el9.x86_64/bin/java -classpath /usr/share/tomcat/default/bin/bootstrap.jar:/usr/share/tomcat/default/bin/tomcat-juli.jar -Djcifs.smb.client.responseTimeout=120000 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/app/heapdumps -XX:-OmitStackTraceInFastThrow -Djava.security.egd=file:/dev/./urandom -Dlogging.config=/usr/share/tomcat/default/conf/log4j2.xml -Dcatalina.base=/usr/share/tomcat/default -Dcatalina.home=/usr/share/tomcat/default -Djava.io.tmpdir=/usr/share/tomcat/default/temp -Djava.security.disableSystemPropertiesFile=true -Djava.util.logging.config.file=/usr/share/tomcat/default/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx15550m org.apache.catalina.startup.Bootstrap start
Comments
0 comments
Please sign in to leave a comment.