You are currently browsing the category archive for the ‘Tomcat’ category.

Having successfully enabled Hudson security with authentication against an LDAP server, I wanted to tweak the configuration to use a secure connection.

I tried using the ldaps scheme, ‘ldaps://ds1.crashingdaily.com’, as per the inline documentation but was confronted with the error “Syntax of this field is SERVER or SERVER:PORT”. I puzzled over this message for awhile, triple checked the syntax, consulted the support forums and then the source code to confirm that the syntax was indeed supported. Finally, I ignored the error, saved the configuration and tried a login. Then Hudson spit out a helpful exception in its Tomcat log:

hudson.security.AuthenticationProcessingFilter2
onUnsuccessfulAuthentication
INFO: Login attempt failed
org.acegisecurity.AuthenticationServiceException: Unable to connect to
LDAP server; nested exception is javax.naming.CommunicationException:
simple bind failed: ds1.crashingdaily.com:636 [Root exception is
javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target]; nested exception is
org.acegisecurity.ldap.LdapDataAccessException: Unable to connect to
LDAP server; nested exception is javax.naming.CommunicationException:
simple bind failed: ds1.crashingdaily.com:636 [Root exception is
javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target]
	at
org.acegisecurity.providers.ldap.LdapAuthenticationProvider.retrieveUser
(LdapAuthenticationProvider.java:238)

The key phrase to me there was ‘unable to find valid certification’. I suspected the Java classes used to negotiate the ldap connection were balking at my self-signed certificate coming from the ldap server. This wasn’t really a failure of Hudson specifically but rather a common issue for many language libraries and SSL applications. Typically the solution is as simple as pointing the library and/or application at the Certificate of Authority (CA) that was used to self-sign the server’s SSL cert. A quick Google search turned up sufficient information for me to import our CA into a keystore for Tomcat’s JVM.

For starters, I made a copy of the cacerts that ships with the JDK. This keystore contains a set of common, trusted CA certs. I don’t need them for Hudson but they may come in handy for future use.

$ cp $JAVA_HOME/jre/lib/security/cacerts /usr/local/tomcat/conf/

Now, I need to get the CA certificate for the ldap server. I have that file around here somewhere but rather than digging it up it’s simple to just get it from the ldap server using openssl.

$ openssl s_client -connect ds1.crashingdaily.com:636 -showcerts

This openssl command prints a list of certificates from the server, the last certificate is the CA. I copy that last certificate to a file, my-ca.crt, and import it into the cacerts keystore.

$ $JAVA_HOME/bin/keytool -import -alias myca -keystore cacerts \
-trustcacerts -file my-ca.crt

The -alias is not too important, it just has to be unique in the keystore. You can list the keys in the store with

$ $JAVA_HOME/bin/keytool -list -keystore  cacerts

Then I add a system property for Tomcat’s JVM that specifies the trustStore as my new cacert. For example, this can be added to the CATALINA_OPTS environment variable if you use the catalina.sh script that ships with Tomcat.

-Djavax.net.ssl.trustStore=/usr/local/tomcat/conf/cacerts

I restart the Tomcat instance for my Hudson installation and now Hudson is able to authenticate over a secure channel.

Related:

Apache Tomcat 5.5 SSL Configuration HOW-TO
Troubleshooting javax.net.ssl.SSLHandshakeException

Advertisements

Today I compiled a 32-bit jsvc daemon tool linked to a 32-bit Java JDK on a 64-bit machine. Turns out I didn’t need it but am recording my notes in case someone else does need it.

I’m using the jsvc source that is bundled with Tomcat and Sun’s Java SE Development Kit 6u10 (jdk-6u10-linux-i586.bin).

$ JAVA_HOME=/usr/java/jdk1.6.0_10-32bit

$ cd /usr/local/apache-tomcat-5.5.23/bin/jsvc-src

$ CFLAGS=-m32 CPPFLAGS=-m32 CCASFLAGS=-m32 LDFLAGS="-L/usr/lib -L/lib" \
./configure --build=i686-pc-linux-gnu

$ make

make will error with

gcc -L/usr/java/jdk1.6.0_10-32bit/lib -L/usr/lib -L/lib -ldl -lpthread jsvc-unix.o 
libservice.a -o ../jsvc /usr/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl /usr/bin/ld: skipping incompatible /usr/lib/libdl.a when searching for -ldl ....

Adjust that failing gcc command with the -m32 switch and compile manually.

$ cd native/
$ gcc -m32 -L/usr/java/jdk1.6.0_10-32bit/lib -L/usr/lib \
 -L/lib -ldl -lpthread jsvc-unix.o libservice.a -o ../jsvc

There should now be a 32-bit jsvc binary in the jsvc-src directory.

$ file jsvc
jsvc: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped

Tip of the hat to a Tomcat mailing list thread started by Rob Tanner

If you don’t know where you are going, any road will take you there.
– Lewis Carroll

My production servers reside behind a perimeter firewall in a data center. A minimal set of ports are open to the world, notably port 22 for sshd and port 80 for the Apache webservers which proxy requests to one of several Tomcat instances. The Tomcat ports are blocked at the data center’s perimeter firewall which means no direct access to Tomcat’s manager interfaces. But that’s OK, there are several options for reaching the Tomcat manager from outside the data center. I’ll glance over three options and then delve into a fourth option that is the gooey center of this posting.

Read the rest of this entry »

I reported yesterday being unable to use the JRockit JVM when launching Tomcat with jsvc -user tomcat. I put strace -f on the process and came up with

[pid 13655] open(“/proc/self/maps”, O_RDONLY) = -1 EACCES (Permission denied)

That wasn’t surprising, I expected a /proc read problem based on my previous research, but this nailed it. strace has help me out on a few other occasions (even though I barely understand the output). I really must remember to use it sooner in the troubleshooting process.

Anyway, I delved into the source code for jsvc (jsvc-unix.c) where I was introduced to the world of Linux capabilities. Long story short, I implemented CAP_DAC_READ_SEARCH in the relevant macros, permitting the switched user to read /proc (and any other file on the system for that matter).

Here is my patch of jsvc-unix.c (from jsvc source shipped with the Apache Tomcat 5.5.20 distribution).

— jsvc-unix.c.dist 2007-02-05 22:34:01.000000000 -0500
+++ jsvc-unix.c 2007-02-05 23:41:18.000000000 -0500
@@ -115,12 +115,15 @@
#define CAPSMAX (1 << CAP_NET_BIND_SERVICE)+ \
(1 << CAP_DAC_READ_SEARCH)+ \
(1 << CAP_DAC_OVERRIDE)
-/* That a more reasonable configuration */
+/* That a more reasonable configuration.
+ CAP_DAC_READ_SEARCH permits reading /proc/self */
#define CAPS (1 << CAP_NET_BIND_SERVICE)+ \
+ (1 << CAP_DAC_READ_SEARCH)+ \
(1 << CAP_SETUID)+ \
(1 << CAP_SETGID)
/* probably the only one Java could use */
-#define CAPSMIN (1 << CAP_NET_BIND_SERVICE)
+#define CAPSMIN (1 << CAP_NET_BIND_SERVICE)+ \
+ (1 << CAP_DAC_READ_SEARCH)
static int set_caps(int caps)
{
struct __user_cap_header_struct caphead;

With patched jsvc in service, Tomcat happily runs with JRockit under a non-privileged user.

Related:

$ man 7 capabilities
BEA JRockit

Postscript
I’m reposting the pre-patch error from catalina.err here for Google to index. Maybe this post will be discovered and help someone encountering the same problem.

05/02/2007 23:13:16 12004 jsvc.exec error: Cannot create Java VM
05/02/2007 23:13:16 12003 jsvc.exec error: Service exit with a return value of 1
[WARN ] pthread_getattr_np failed in psiGetStackInfo for 0x2a9557b960.
[WARN ] OS message: Permission denied (13)

I’ve been exploring replacing Sun’s JVM with JRockit as a means to improving the uptime of our Struts-based web applications running under Tomcat. The problem I’ve been having is this. Repeatedly reloading our Struts-based web application (something we do frequently during development) causes Tomcat to stop responding to requests after about 10-ish reloads. Other techniques for updating a single webapp running under a live Tomcat instance – undeploy/deploy, stop/start – exhibit the same symptoms. The only out is to restart the entire Tomcat instance – disrupting all the other fully functional webapps deployed there.

Initially Tomcat logs were of no help to me but after upgrading Tomcat to 5.5 (from 5.0.28) and Sun’s JDK to 1.5.0_10 (from 1.5.0_05) I started getting Tomcat log entries that suggested the problem.

java.lang.OutOfMemoryError: PermGen space

That’s something Google can sink its teeth into and it quickly spit out the digested remains of the other poor saps that have fallen prey to this. The likely problem of this memory leak is due to incomplete garbage collection of the Classloader such that classes persist in Sun’s JVM PermGen memory heap – and therefore for the life of the JVM.

The simplest workaround (short of tracking down the sources of memory leaks in the app itself) suggested to replace Sun’s JVM with BEA’s JRockit. JRockit doesn’t have a PermGen heap so there can be no leaking there.

Read the rest of this entry »

Categories

January 2019
M T W T F S S
« May    
 123456
78910111213
14151617181920
21222324252627
28293031  

Latest del.icio.us

Advertisements