Port Forwarding Through the Looking Glass

If you don’t know where you are going, any road will take you there.
- Lewis Carroll

My production servers reside behind a perimeter firewall in a data center. A minimal set of ports are open to the world, notably port 22 for sshd and port 80 for the Apache webservers which proxy requests to one of several Tomcat instances. The Tomcat ports are blocked at the data center’s perimeter firewall which means no direct access to Tomcat’s manager interfaces. But that’s OK, there are several options for reaching the Tomcat manager from outside the data center. I’ll glance over three options and then delve into a fourth option that is the gooey center of this posting.

JRockit Liftoff

I reported yesterday being unable to use the JRockit JVM when launching Tomcat with jsvc -user tomcat. I put strace -f on the process and came up with

[pid 13655] open(”/proc/self/maps”, O_RDONLY) = -1 EACCES (Permission denied)

That wasn’t surprising, I expected a /proc read problem based on my previous research, but this nailed it. strace has help me out on a few other occasions (even though I barely understand the output). I really must remember to use it sooner in the troubleshooting process.

Anyway, I delved into the source code for jsvc (jsvc-unix.c) where I was introduced to the world of Linux capabilities. Long story short, I implemented CAP_DAC_READ_SEARCH in the relevant macros, permitting the switched user to read /proc (and any other file on the system for that matter).

Here is my patch of jsvc-unix.c (from jsvc source shipped with the Apache Tomcat 5.5.20 distribution).

— jsvc-unix.c.dist 2007-02-05 22:34:01.000000000 -0500
+++ jsvc-unix.c 2007-02-05 23:41:18.000000000 -0500
@@ -115,12 +115,15 @@
#define CAPSMAX (1 << CAP_NET_BIND_SERVICE)+ \
(1 << CAP_DAC_READ_SEARCH)+ \
(1 << CAP_DAC_OVERRIDE)
-/* That a more reasonable configuration */
+/* That a more reasonable configuration.
+ CAP_DAC_READ_SEARCH permits reading /proc/self */
#define CAPS (1 << CAP_NET_BIND_SERVICE)+ \
+ (1 << CAP_DAC_READ_SEARCH)+ \
(1 << CAP_SETUID)+ \
(1 << CAP_SETGID)
/* probably the only one Java could use */
-#define CAPSMIN (1 << CAP_NET_BIND_SERVICE)
+#define CAPSMIN (1 << CAP_NET_BIND_SERVICE)+ \
+ (1 << CAP_DAC_READ_SEARCH)
static int set_caps(int caps)
{
struct __user_cap_header_struct caphead;

With patched jsvc in service, Tomcat happily runs with JRockit under a non-privileged user.

Related:

$ man 7 capabilities
BEA JRockit

Postscript
I’m reposting the pre-patch error from catalina.err here for Google to index. Maybe this post will be discovered and help someone encountering the same problem.

05/02/2007 23:13:16 12004 jsvc.exec error: Cannot create Java VM
05/02/2007 23:13:16 12003 jsvc.exec error: Service exit with a return value of 1
[WARN ] pthread_getattr_np failed in psiGetStackInfo for 0×2a9557b960.
[WARN ] OS message: Permission denied (13)

Crashing Tomcat

I’ve been exploring replacing Sun’s JVM with JRockit as a means to improving the uptime of our Struts-based web applications running under Tomcat. The problem I’ve been having is this. Repeatedly reloading our Struts-based web application (something we do frequently during development) causes Tomcat to stop responding to requests after about 10-ish reloads. Other techniques for updating a single webapp running under a live Tomcat instance - undeploy/deploy, stop/start - exhibit the same symptoms. The only out is to restart the entire Tomcat instance - disrupting all the other fully functional webapps deployed there.

Initially Tomcat logs were of no help to me but after upgrading Tomcat to 5.5 (from 5.0.28) and Sun’s JDK to 1.5.0_10 (from 1.5.0_05) I started getting Tomcat log entries that suggested the problem.

java.lang.OutOfMemoryError: PermGen space

That’s something Google can sink its teeth into and it quickly spit out the digested remains of the other poor saps that have fallen prey to this. The likely problem of this memory leak is due to incomplete garbage collection of the Classloader such that classes persist in Sun’s JVM PermGen memory heap - and therefore for the life of the JVM.

The simplest workaround (short of tracking down the sources of memory leaks in the app itself) suggested to replace Sun’s JVM with BEA’s JRockit. JRockit doesn’t have a PermGen heap so there can be no leaking there.