I’ve been exploring replacing Sun’s JVM with JRockit as a means to improving the uptime of our Struts-based web applications running under Tomcat. The problem I’ve been having is this. Repeatedly reloading our Struts-based web application (something we do frequently during development) causes Tomcat to stop responding to requests after about 10-ish reloads. Other techniques for updating a single webapp running under a live Tomcat instance – undeploy/deploy, stop/start – exhibit the same symptoms. The only out is to restart the entire Tomcat instance – disrupting all the other fully functional webapps deployed there.
Initially Tomcat logs were of no help to me but after upgrading Tomcat to 5.5 (from 5.0.28) and Sun’s JDK to 1.5.0_10 (from 1.5.0_05) I started getting Tomcat log entries that suggested the problem.
java.lang.OutOfMemoryError: PermGen space
That’s something Google can sink its teeth into and it quickly spit out the digested remains of the other poor saps that have fallen prey to this. The likely problem of this memory leak is due to incomplete garbage collection of the Classloader such that classes persist in Sun’s JVM PermGen memory heap – and therefore for the life of the JVM.
The simplest workaround (short of tracking down the sources of memory leaks in the app itself) suggested to replace Sun’s JVM with BEA’s JRockit. JRockit doesn’t have a PermGen heap so there can be no leaking there.
I set up a Tomcat instance for comparing Sun’s JVM and BEA’s Jrockit (actually, I borrowed an instance I had set up for testing Tomcat failover and clustering – but that’s for another posting). And started with the following command to use Sun’s JVM.
sudo /usr/local/apache-tomcat/bin/jsvc \\
-user root \\
-home $JHOME \\
-wait 10 \\
-pidfile /var/run/jsvc_Failover.pid \\
-outfile /usr/local/tomcat_instances/Failover/logs/catalina.out \\
-errfile /usr/local/tomcat_instances/Failover/logs/catalina.err \\
-cp $JHOME/lib/tools.jar:/usr/local/apache-tomcat/:/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/commons-logging-api.jar \\
Stopping that process and changing JHOME on launch switches to using Jrockit:
The following is my testing routine for triggering a PermGen error. It simply loops over calling my web application then calling tomcat’s manager to reload the app. Accessing my application is necessary to trigger the error. I can reload the webapp dozens of times (before I abort the testing) without problem if I don’t actually use it.
do curl ‘http://127.0.0.1:9380/tapp/’ &>/dev/null; \\
curl -n “http://127.0.0.1:9380/manager/reload?path=/tapp”; \\
Depending on the size of the JVM’s maximum Java heap size (typically -Xmx256) I get around 8 invocations with Sun’s JVM before the Tomcat instance crashes. With JRockit I’ve gone up to 125 reloads without failure before stopping the testing.
Sadly there is a show stopper preventing me from switching to JRockit in my production system. I am unable to run Tomcat as a non-root user with JRockit (note the -user root in the launch command above). Attempting to switch user (-user tomcat) results in the error:
[WARN ] OS message: Permission denied (13)
04/02/2007 22:38:26 10337 jsvc.exec error: Cannot create Java VM
04/02/2007 22:38:26 10336 jsvc.exec error: Service exit with a return value of 1
An strace -f on the process shows that the non-root user is unable to read /proc/self/maps (this is on Red Hate EL4, kernel 2.6.9-42.0.3.ELsmp, x86_64). Sun’s JVM gives a related error
but runs anyway.
I can’t say if we have other memory leaks that will sink our dinghy eventually but JRockit so far has a lot of promise for stemming the tide. I only need to get Tomcat+JRockit to run as a non-privileged user before I roll it out in a QA site for further testing and then into production.