Extending Selenium server with custom commands

In this posting I document extending Selenium server with a custom javascript method, getAllRadios(), to return all radio button elements in a page. It’s based on the getAllButtons() that is stock in Selenium.

I will demonstrate two ways to add this custom method. The first strategy is to add the custom method to the native javascript files that ship with Selenium server. This is not the recommended way to extend Selenium but it is an almost guaranteed way to be sure the server picks up my new method and that frees me to focus on getting my client syntax right for calling the new method using the client’s doCommand() method.

Method 1 – selenium-browserbot.js + selenium-api.js

Unpack the server jar into a temporary directory to get access to the selenium-browserbot.js and selenium-api.js files.

[20:54 20090906 crashing@server /selenium-server-1.0.1/temp]
$ jar xf ../selenium-server.jar

The new custom method is added to core/scripts/selenium-browserbot.js. It is a slightly modified version of getAllButtons() which is defined in the same file.

BrowserBot.prototype.getAllRadios = function() {
var elements = this.getDocument().getElementsByTagName('input');
var result = [];
for (var i = 0; i < elements.length; i++) {
if (elements[i].type == 'radio') {
result.push(elements[i].id);
}
}
return result;
};

and then the new method is exposed to the Selenium API in core/scripts/selenium-api.js, again modeling after getAllButtons() in selenium-api.js.

Selenium.prototype.getAllRadios = function() {
return this.browserbot.getAllButtons();
};

Now rebundle the updated javascript into a new server jar, replacing the original.

[21:01 20090906 crashing@server /selenium-server-1.0.1/temp]
$ jar cmf META-INF/MANIFEST.MF ../selenium-server.jar .

Method 2 – user-extension.js

The better way to extend Selenium is to add methods to the user-extensions.js file. This avoids screwing around with the native server jar. I initiall had some trouble getting this working, hence the Method 1 approach, but after a bit of futzing I finally got this to work.

Selenium.prototype.getAllRadios = function() {
var elements = this.browserbot.getDocument().getElementsByTagName('input');
var result = [];
for (var i = 0; i < elements.length; i++) {
if (elements[i].type == 'radio') {
result.push(elements[i].id);
}
}
return result;
};

This is basically the same method used in selenium-browserbot.js. The important changed bits are (1) the method is attached to the Selenium object instead of the BrowserBot (Selenium.prototype rather than BrowserBot.prototype) and (2) calling getDocument() on the browserbot instance.

I start the Selenium server with the -userExtensions option pointing to the user-extensions.js file.

java -jar \
/selenium-server-1.0.1/selenium-server.jar \
-userExtensions /user-extensions.js

Client Coding

For either of the above methods of extending the Selenium server, I call this new method in my client code with the doCommand().

/** incomplete code **/

proc = new HttpCommandProcessor(seleniumServerHost,
seleniumServerPort, brower, seleniumServerStartUrl);
selenium = new DefaultSelenium(proc);

selenium.open(url);

// radio buttons returned as comma-delimited strings
String allRadios = proc.doCommand("getAllRadios", null);

// alternatively, get radio buttons as a String array
String[] radios = proc.getStringArray("getAllRadios", null);
for (String radio : radios) {
System.out.println(radio);
}

To call custom doCommand()‘s, it’s necessary to instantiate with DefaultSelenium(proc) to inject the HttpCommandProcessor into the DefaultSelenium object.

Related:

Select the ids of radio buttons in a page using selenium
User-Extensions

Downsizing Google’s search input box

Google has made its search input box bigger and its font size larger. Excuse me, they S-U-P-E-R-sized! it. It’s not a mistake, indeed Google actually seems proud of it.

Fortunately, Safari and Firefox empowers mere-mortal users to fix this eyesore with a custom stylesheet. Simply add the following to your userContent.css and restart the web browser. Additional general information on userContent.css, and all the spiffy things you can do with it, is available from the links below.

/* Google input text box */
.lst {
font-size:13px ! important;
}
/* Google search type-ahead drop down */
.gac_m td {
font-size:13px ! important;
line-height:120% ! important;
}
/* Google Search/I'm Feeling Lucky buttons */
.lsb {
font-size:11px ! important;
height:auto ! important;
}
/* Google Search/I'm Feeling Lucky buttons in type-ahead drop down*/
.gac_sb {
font-size:11px ! important;
height:auto ! important;
}

With the aforementioned CSS, the submit buttons will still appear in the type-ahead drop down menu. Add the following style to remove them all together.

/* Turn off Google Search/I’m Feeling Lucky buttons in type-ahead drop down*/
.gac_sb {
display:none;
}

Power to the people.

Related Sites

Customizing Mozilla
Better Ad Blocking for Firefox, Mozilla, Camino, and Safari
Google Chrome Issue 2393: Support user stylesheet

Initing a new remote git repository with existing files

This how I took an existing directory of files on my desktop and put it into a new git repository on a remote server. I found a number of how-tos online but none worked for me – certainly this was because I’m such a newbie with git that I wasn’t sufficiently understanding what I was being told to do. The following works for me; clearly this isn’t the only way or the best way to accomplish the task. This is mostly a note to self.

Create an empty directory on the remote server to hold the repository

[11:44 20090902 crashing@server ~]
$ mkdir -p /var/local/git/repos/CrashTesting

Intialize the empty repository

[11:44 20090902 crashing@server /var/local/git/repos/CrashTesting]
$ git --bare init
Initialized empty Git repository in /var/local/git/repos/CrashTesting/

Now I have the necessary repository components.

[11:44 20090902 crashing@server /var/local/git/repos/CrashTesting]
$ ls
branches config description HEAD hooks info objects refs

On my desktop, in the existing directory of files, init the directory

[11:30 20090902 crashing@desktop ~/Desktop/testws]
$ git init
Initialized empty Git repository in /Users/crashing/Desktop/testws/.git/

This created a .git directory with git control files.

Next, I tell my local desktop repository about the remote repo. The remote repo is given the short name origin.

[11:46 20090902 crashing@desktop ~/Desktop/testws]
$ git remote add origin ssh://server.crashingdaily.com/var/local/git/repos/CrashTesting

Then, I place my local files under version control.

[11:42 20090902 crashing@desktop ~/Desktop/testws]
$ git add .

Now I can commit the local files to the local repository.

[11:45 20090902 crashing@desktop ~/Desktop/testws]
$ git commit -a -m 'initialize repo'
[master (root-commit) 7871087] initialize repo
23 files changed, 500 insertions(+), 0 deletions(-)
create mode 100755 build.properties
create mode 100755 build.xml

Finally, push the master branch to the remote repository named origin.

[11:46 20090902 crashing@desktop ~/Desktop/testws]
$ git push origin master
Counting objects: 44, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (31/31), done.
Writing objects: 100% (44/44), 1.65 MiB, done.
Total 44 (delta 1), reused 0 (delta 0)
To ssh://server.crashingdaily.com/var/local/git/repos/CrashTesting
* [new branch] master -> master

Scheduling jobs with at and cgi scripts

The at command can be used in CGI scripts to remotely trigger non-recurring jobs on a web server. This is especially useful for long running jobs that clients don’t need to wait on and jobs that need to persist across web daemon restarts.

My initial simple attempts to schedule a job with at in a CGI script executed by an Apache webserver were met with the failure response “This account is currently not available“. That message being the output of /sbin/nologin which is the shell value set for the apache user in /etc/password.

The fix is to set the SHELL environment variable to a proper shell when executing at. Here’s a simple, proof-of-concept CGI script.

#!/usr/bin/perl
use CGI;
my $cgi = new CGI;
print $cgi->header;

system(‘echo “/usr/bin/env > /tmp/apacheenv” | SHELL=/bin/bash at now’);

Calling the CGI script triggers the at scheduling. In this case, the execution of /usr/bin/env is immediate and successful.

$ cat /tmp/apacheenv
SERVER_SIGNATURE=
SHELL=/bin/bash
HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_7; en-us) AppleWebKit/528.18.1 (KHTML, like Gecko) Version/4.0 Safari/528.17
SERVER_PORT=80

Related:

at man page

Securing Hudson LDAP authentication

Having successfully enabled Hudson security with authentication against an LDAP server, I wanted to tweak the configuration to use a secure connection.

I tried using the ldaps scheme, ‘ldaps://ds1.crashingdaily.com’, as per the inline documentation but was confronted with the error “Syntax of this field is SERVER or SERVER:PORT”. I puzzled over this message for awhile, triple checked the syntax, consulted the support forums and then the source code to confirm that the syntax was indeed supported. Finally, I ignored the error, saved the configuration and tried a login. Then Hudson spit out a helpful exception in its Tomcat log:

hudson.security.AuthenticationProcessingFilter2
onUnsuccessfulAuthentication
INFO: Login attempt failed
org.acegisecurity.AuthenticationServiceException: Unable to connect to
LDAP server; nested exception is javax.naming.CommunicationException:
simple bind failed: ds1.crashingdaily.com:636 [Root exception is
javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target]; nested exception is
org.acegisecurity.ldap.LdapDataAccessException: Unable to connect to
LDAP server; nested exception is javax.naming.CommunicationException:
simple bind failed: ds1.crashingdaily.com:636 [Root exception is
javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target]
	at
org.acegisecurity.providers.ldap.LdapAuthenticationProvider.retrieveUser
(LdapAuthenticationProvider.java:238)

The key phrase to me there was ‘unable to find valid certification’. I suspected the Java classes used to negotiate the ldap connection were balking at my self-signed certificate coming from the ldap server. This wasn’t really a failure of Hudson specifically but rather a common issue for many language libraries and SSL applications. Typically the solution is as simple as pointing the library and/or application at the Certificate of Authority (CA) that was used to self-sign the server’s SSL cert. A quick Google search turned up sufficient information for me to import our CA into a keystore for Tomcat’s JVM.

For starters, I made a copy of the cacerts that ships with the JDK. This keystore contains a set of common, trusted CA certs. I don’t need them for Hudson but they may come in handy for future use.

$ cp $JAVA_HOME/jre/lib/security/cacerts /usr/local/tomcat/conf/

Now, I need to get the CA certificate for the ldap server. I have that file around here somewhere but rather than digging it up it’s simple to just get it from the ldap server using openssl.

$ openssl s_client -connect ds1.crashingdaily.com:636 -showcerts

This openssl command prints a list of certificates from the server, the last certificate is the CA. I copy that last certificate to a file, my-ca.crt, and import it into the cacerts keystore.

$ $JAVA_HOME/bin/keytool -import -alias myca -keystore cacerts \
-trustcacerts -file my-ca.crt

The -alias is not too important, it just has to be unique in the keystore. You can list the keys in the store with

$ $JAVA_HOME/bin/keytool -list -keystore  cacerts

Then I add a system property for Tomcat’s JVM that specifies the trustStore as my new cacert. For example, this can be added to the CATALINA_OPTS environment variable if you use the catalina.sh script that ships with Tomcat.

-Djavax.net.ssl.trustStore=/usr/local/tomcat/conf/cacerts

I restart the Tomcat instance for my Hudson installation and now Hudson is able to authenticate over a secure channel.

Related:

Apache Tomcat 5.5 SSL Configuration HOW-TO
Troubleshooting javax.net.ssl.SSLHandshakeException

Hudson LDAP group authorization

Enabling security to my Hudson installation was a fairly straight-foward task although it did take a little trial and error to get it all straight. I chose to authenticate against our existing LDAP directory. One of the things that caught me off guard at first was the ‘User search base’ and ‘Group search base’. These settings are Relative Distinguished Names (ou=People), not the full Distinguished Name (ou=People,dc=crashingdaily,dc=com). Not a big deal; in fact I could have (should have, but I like to tinker) just left these fields empty and Hudson would have figured it out on its own.

Here is my almost final configuration (my final configuration makes use of StartTLS for a secure connection to the LDAP server – more on that in a later post).

The Manager DN is an account that has read-only search permissions. This is required because our LDAP directory is not searchable with anonymous binds.

The LDAP group configuration took me a bit more effort and debugging. I have an HudsonAdmin LDAP group in my directory. The DN for it is cn=HudsonAdmin,ou=Groups,dc=crashingdaily,dc=com

To get this work I had to upgrade to the 1.261 release which adds uniqueMember, memberUid to the existing member in the search filter (my HudsonAdmin group has ‘uniqueMember’ entries). Incidentally, one of the things I love about Hudson is when I run into a bug or missing feature I can usually count on it be addressed in a newer release – and upgrading to a new release is quick and painless.

A not-so-obvious aspect to the configuration is that, when adding an LDAP group, Hudson wants the group name in all uppercase and with ‘ROLE_‘ prefixed to it. So, ‘cn=HudsonAdmin,...‘ becomes ‘ROLE_HUDSONADMIN‘. I did find this requirement documented after a little Google searching, it’s just not spelled out in the admin form’s help.

Tip: When screwing around with your Hudson Authorization, make frequent backups of the working $HUDSON_HOME/config.xml so if you make a mistake and lock yourself out you can restore a working configuration from backup, reload Hudson and be back in business. The Hudson documentation does include a note about recovering from a lockout by setting useSecurity to false in the config.xml file but I find that doing that causes a loss of all your security settings and you’ll have to re-enter all the LDAP values and Authorization settings.

Compiling a 32-bit jsvc on a 64-bit machine

Today I compiled a 32-bit jsvc daemon tool linked to a 32-bit Java JDK on a 64-bit machine. Turns out I didn’t need it but am recording my notes in case someone else does need it.

I’m using the jsvc source that is bundled with Tomcat and Sun’s Java SE Development Kit 6u10 (jdk-6u10-linux-i586.bin).

$ JAVA_HOME=/usr/java/jdk1.6.0_10-32bit

$ cd /usr/local/apache-tomcat-5.5.23/bin/jsvc-src

$ CFLAGS=-m32 CPPFLAGS=-m32 CCASFLAGS=-m32 LDFLAGS="-L/usr/lib -L/lib" \
./configure --build=i686-pc-linux-gnu

$ make

make will error with

gcc -L/usr/java/jdk1.6.0_10-32bit/lib -L/usr/lib -L/lib -ldl -lpthread jsvc-unix.o 
libservice.a -o ../jsvc
/usr/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libdl.a when searching for -ldl
....

Adjust that failing gcc command with the -m32 switch and compile manually.

$ cd native/
$ gcc -m32 -L/usr/java/jdk1.6.0_10-32bit/lib -L/usr/lib \
 -L/lib -ldl -lpthread jsvc-unix.o libservice.a -o ../jsvc

There should now be a 32-bit jsvc binary in the jsvc-src directory.

$ file jsvc
jsvc: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped

Tip of the hat to a Tomcat mailing list thread started by Rob Tanner

Fronting Apache with Nginx

I’ve been exploring using Nginx to front our Apache websites. I found a fair amount of documentation online but most of it was for nginx on top of a backend application running on the same host – so, lots of examples of load balancing among various ports on 127.0.0.1. In my case I have name-based Apache virtual hosts running at different IP addresses. Also my goal is not to set up automatic load balancing – our session-based application will not work well with it – but rather I want to be able to rapidly and manually switch a web address to a different machine so I can perform maintenance on or updates to the offline systems.

To summarise, I want the publically accessible crashingdaily.com website to be a proxy server to our internal, Apache name-based virtual hosts w1 or w2. I want to be able to choose which of w1 or w2 services the client’s request.

Changing the Run-As User for Fedora Directory Server

I have some installations of Fedora Directory Server (FDS) running as the user nobody. It is generally preferred that services like FDS run under a dedicated user and certainly not the nobody account. The user is configured during an initial scripted interactive setup but I wanted to change the user for the existing installations. I could not find a how-to for doing this (though did not look very hard) so I did a new installation on a beater box and during setup configured it to run as user ldapperuser and group ldappergroup. I used this installation to experiment with reconfiguring the user.

I shutdown the slapd service and changed ownership of the files owned by ldapperuser and/or ldappergroup.

find /opt/fedora-ds -user ldapperuser | xargs chown ldap
find /opt/fedora-ds -group ldappergroup | xargs chgrp ldap

By grep’ing for the silly user and group names I found these text files needed to be edited to change the user and/or group.

/opt/fedora-ds/admin-serv/config/console.conf 
/opt/fedora-ds/shared/config/ssusers.conf
/opt/fedora-ds/slapd-pepper/config/dse.ldif

I restarted the slapd process and confirmed in a process list that it was running as the new user.

Finally, I exported the entire directory to an ldif file and grep’ed it for the user and group names. There I found that I needed to change the nsSuiteSpotUser attribute in "cn=slapd-pepper, cn=Fedora Directory Server, cn=Server Group, cn=pepper.crashingdaily.com, ou=crashingdaily.com, o=NetscapeRoot"

I made this change to nsSuiteSpotUser via the Admin console.

There may very well be an official way to change the user for an installed FDS but this brute force method is simple enough.

SSH SOCKS Proxy From Behind a Gateway

Lifehacker a nice write up on Firefox web browsing with SOCKS proxies. The tip about network.proxy.socks_remote_dns was new to me and I will have to play with that sometime. Safari, my primary browser, seems to resolve DNS requests at the proxy by default so that saves me the hassle in the meantime.

One of the take home messages of the Lifehacker entry is that you can run “ssh -D 1080 server.remotehost.com” on your workstation, then configure Firefox (as well as most other browsers) to use a SOCKS proxy at localhost port 1080. This provides encrypted communications between your workstation and server (great when your workstation is on an untrusted wireless network) and for masquerading as the server (useful when accessing websites that are behind a firewall or that restrict access by IP address).

Very simple, extremely handy. But what if you want to use remote server that is behind a firewall and only accessible via a gateway machine?

----------------               -------------         -------------
| workstation  |               |           |         |  server   |
|              | --------------|  gateway  | ------- |           |
|(web browser) |               |           |         |  (SOCKS)  |
----------------               -------------         -------------

In that case you have to tunnel through the gateway to get to the SOCKS server running on the server. In this post I’m going to walk though building up the ssh command that will achieve such a tunnel. I will then present an alternate, more generic method.