Grepping nonprinting characters

Here is a tip summarized from comp.unix.shell

The problem:

> For the following TAB-delimited records, I want to count number of
> records with column-2 == -1   (should be 2)
> ===== file.txt ======
> AAA    -1    2008-07-14
> BBB    -14   2008-07-15
> CCC    -20   2008-07-16
> DDD    -1    2008-07-16
> ===========
> I tried:
>   grep -c -- "-1\t" file.txt
> which is not working

A solution offered for ksh93/zsh/bash shells:

grep -c -- $'-1\t' file.txt

I will add another alternative

grep -c '-1^V^I' file.txt

Where ^V^I means type ctrl-v and ctrl-i to enter a tab character.

Related:

Advanced Bash-Scripting Guide Example 34-1
Insert ASCII Control Characters in Text

perl pie in your face

I frequently use Perl’s in place file editing from the command line. What I didn’t consider until it bit me today is that the file ownership can change using this method.

Here’s the original file, owned by tomcat_6 and only readable by user and group.

$ ls -l web.xml
-rw-rw----  1 tomcat_6 tomcat 49384 Jul 10 11:38 web.xml

I belong to the tomcat group, so have write permissions to the file. The enclosing directory is also tomcat group writable. The importance of this is noted below.

Using the ‘perl pie’ one-liner to make an in place edit if the file:

$ perl -p -i -e 's;<session-timeout>\d+</session-timeout>;\
<session-timeout>1440</session-timeout>;' web.xml

Now the file is owned by me and my default group.

$ ls -l web.xml
-rw-rw----  1 crashing  daily 49384 Jul 10 21:55 web.xml

Most critically, now the file is no longer readable by the tomcat processes. This little change prevented my Tomcat server from starting. Ouch.

sed is a little nicer. It changes the owner but not the group.

$ sed -i 's;<session-timeout>.*</session-timeout>;\
<session-timeout>1445</session-timeout>;g' web.xml

$ ls -l web.xml 
-rw-rw----  1 crash tomcat 49385 Jul 10 22:44 web.xml

Neither the Perl nor the sed one-liners work if the directory is not writable because Perl and sed require unlinking the original file and replacing it with a new version.

The winner for both maintaining file ownership and working if the directory is not writable is ed.

$ ed - "web.xml" <<EOF
,s;<session-timeout>[[:digit:]]*</session-timeout>;\
<session-timeout>1440</session-timeout>;
w
EOF

ed truly does an in place edit. Nice. If only I could remember the syntax.

Hashing the executables - a look at hash and type

Here is a brief look at the Linux commands, type and hash.

To start, I open a new terminal and check for the location of the foo executable using the which command. (foo is a trivial shell script I concocted for illustrative examples in this posting.)

$ which foo
/usr/local/bin/foo

which tells me the location of the executable file in my $PATH but that is not necessarily what will be executed when I call foo on the command line. To learn that, I use the type command.

$ type foo
foo is /usr/local/bin/foo

type reports how a word will be interpreted if used as command name. In this case, it is telling me that using foo as a command will execute the file /usr/local/bin/foo. That happens to be what which also reported, but, as we will see in the following examples, that will not always be the case.

I can run the foo command to print its release version.

$ foo --version
foo release 1.0

Consider if I install a new foo, version 2.0, in my ~/bin directory, leaving the 1.0 version in /usr/local/bin/foo. I have positioned ~/bin ahead of /usr/local/bin in my $PATH and the which command confirms the new 2.0 version is found first.

$ which foo
/home/crashingdaily/bin/foo

However, when I call for foo to be executed it still executes the old 1.0 version.

$ foo --version
foo release 1.0

Hmm, what’s going on? The which command reported the first executable found in my path but type will be more informative. It will tell us precisely what file, function, builtin, keyword or alias is associated with a given command name.

$ type foo
foo is hashed (/usr/local/bin/foo)

This is reporting that the shell has saved the meaning of foo in a hash table as /usr/local/bin/foo (the 1.0 version). Caching the command in a hash table is an optimization that saves the shell from having to search $PATH every time. A given shell does this the first time a command is run in that shell instance. The hashed values survive for the life of the shell instance. Start a new shell, e.g. by opening a new terminal or invoking a subshell, and you start a new hash table for that shell process.

The hash table can also be manually manipulated to clear or set values. Enter the hash command. This command is used to print and edit the shell’s command hash table.

The entire hash table can be cleared

$ hash -r

or you can delete a specific entry

$ hash -d foo

Having cleared the hash table, invoking foo now calls the first in my path.

$ foo --version
foo release 2.0

And this copy is now placed in the hash table.

$ type foo
foo is hashed (/home/crashingdaily/foo)

I can also print that entry of the hash table to get its value. However, do not use this as a substitute for type as it only deals with files. More on that shortly.

$ hash -t foo
/home/crashingdaily/foo

As you might imagine, if I delete the executable whose value has been hashed,

$ rm /home/crashingdaily/foo

the command is not found when I attempt to use it,

$ foo --version
-bash: /home/crashingdaily/bin/foo: No such file or directory

until I update the hash. I can clear the hash as before or I can explictly set the value like so

$ hash -p /usr/local/bin/foo foo

$ type foo
foo is hashed (/usr/local/bin/foo)

$ foo --version
foo release 1.0

Keep in mind that the hash table caches (and the hash command reports) file and path names. It does not deal with keywords, functions, builtins or aliases which may be invoked by a command name. So, type is the utility to use for learning what will execute when a command name is invoked.

To further illustrate type usage, I will define a function named foo.

$ function foo { echo 'hello world'; }

Because function names take precedence over file executables having the same name, invoking foo executes the function instead of the shell script. Now, type reports

$ type foo
foo is a function
foo ()
{
    echo 'hello world'
}

and invoking the name gives us the expected function output.

$ foo
hello world

Using type with the -a option reports all the known values of foo

$ type -a foo
foo is a function
foo ()
{
    echo 'hello world'
}
foo is /home/crashingdaily/bin/foo
foo is /usr/local/bin/foo

In summary, use type to learn what a given command name currently means to the shell - there could be more than one meaning. Use hash to manipulate cached associations of command names with files (and be aware that the command name may be associated with a non-file that takes precedence).

Additional documentation for type and hash is availble via the help command.

$ help type
type: type [-afptP] name [name ...]
    For each NAME, indicate how it would be interpreted if used as a
    command name...

$ help hash
hash: hash [-lr] [-p pathname] [-dt] [name ...]
    For each NAME, the full pathname of the command is determined and
    remembered....

History meme

History Meme

$ history|awk '{a[$2]++} END{for(i in a){printf "%5d\t%s\n",a[i],i}}'|sort -rn|head
  77  whoami
  74  tic
  72  gawk
  70  od
  70  bash
  69  shred
  66  killall
  58  sleep
  55  id
  55  play

Rude awakening

BASH Cures Cancer has yet another interesting post this week. This one, entitled “Command Substitution and Exit Status“, leverages the exit status returned from a BASH sub-shell. That alone is a useful tip but what tickled me was the author’s trick of shortening the delay in a running loop by starting a newer, shorter sleep loop that kills off the longer-running sleep process. Clever. The thought of dueling shell processes - “I’m going to sleep for a minute.”, “Oh, no you’re not!” - makes me chuckle. Sigh. I am easily amused.

Clearing environment variables in BASH scripts

When you execute a shell script it inherits all the environment variables present in the parent shell. Sometimes that can cause unintended consequences. For example, I recently ran into a situation where one of my scripts in common use by our group failed for one of the users. I eventually tracked it down to the JAVA_HOME variable that user had set in his bash profile. The user’s JAVA_HOME being inherited by the script was not set to an appropriate value. Ideally the script should have set all the environment variables it needs. However, in this case, I had failed to explicitly set JAVA_HOME in my script, an oversight masked by the fact my own JAVA_HOME was set to a valid value so the script ran normally for me.

To reduce the chance of this type of problem reoccurring with other variables I decided to clear all the inherited environment variables at the start of the script. That way any other environment dependency not defined by the script would immediately reveal itself during the script debugging phase.

I could not find a simple builtin method to run the script without the inherited environment, (aside from the impractical “env -i scriptname“), so I put a one-liner in the script to unset each variable (BASH syntax):

unset $(/usr/bin/env | egrep '^(\w+)=(.*)$' | \
  egrep -vw 'PWD|USER|LANG' | /usr/bin/cut -d= -f1);

env prints the environment’s variable=value pairs. The cut at the end splits the pairs and returns the variable name.

The first egrep filters for the variable=value pairs that start at the beginning of the line. This was necessary because GNU screen sets a multiline TERMCAP variable:

TERMCAP=SC|screen|VT 100/ANSI X3.64 virtual terminal:\
        :DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:UP=\E[%dA:bs:bt=\E[Z:\
        :cd=\E[J:ce=\E[K:cl=\E[H\E[J:cm=\E[%i%d;%dH:ct=\E[3g:\

I wanted just the first line and needed to discard the others so the cut would not split them and return bogus variable names, like ‘:DO‘, to unset.

The second egrep is optional but allows me to skip selected variables so they retain their preset values.

With this in place, if I attempt to use an environment variable in my script (or in a program called by my script) it will fail unless I explicitly set the value. This provides me the opportunity to ensure the script will behave consistently for all users on the system.

Diff two stdout streams

The diff utility reports differences between two files. If you need to find differences between one or two stdout outputs then temporary named pipes are a handy aid.

Here’s a simple example for the BASH shell to illustrate the technique. Say you have two files, A and B:

$ cat A
Tara
Dawn
Anya
Willow

$ cat B
WILLOW
ANYA
DAWN
HARMONY
TARA

The task is to find names that differ between the two lists without creating new files or editing the existing. You can do that by sorting and normalizing the letter case, then using diff to check the stdout on the fly.

$ diff -B <( sort A | tr [:lower:] [:upper:] ) <( sort B | tr [:lower:] [:upper:] )

2a3
> HARMONY

The <( ... ) syntax creates a temporary named pipe which makes the stdout of the sort | tr commands look and behave like a file, allowing diff to operate on the expected type of input.

For fun, you can see the temporary file created by the process:

dir <( sort A | tr [:lower:] [:upper:] )
lr-x------  1 crash daily 64 Mar  5 23:17 /dev/fd/63 -> pipe:[21483501]

This technique is not limited to diff. This should work for most any other command expecting a file for input.

Interestingly (frustratingly), it does not work for me in my BASH shell on Mac OS X. Does anyone know why? Update: Indeed someone does know. Unixjunkie has the answer. The following produces the expected output
on OS X but attempting diff produces no output.

cat <( sort A | tr [:lower:] [:upper:] ) <( sort B | tr [:lower:] [:upper:] )

ANYA
DAWN
TARA
WILLOW

ANYA
DAWN
HARMONY
TARA
WILLOW

Related:

Introduction to Named Pipes
Heads and Tails - an earlier posting that uses temporary named pipes for receiving stdin
Process Substitution, Advanced Bash-Scripting Guide

Heads and Tails

I was recently handed a collection of Apache web server logs to parse for statistics. The first step taken was to assess the date ranges covered by each log file. That’s a simple procedure of looking at the first and last line of the logs. Here’s a one-liner for that:

$ zcat access_log.20070709.all.gz | tee \
>(head -n1) \
>(tail -n1) &>/dev/null

192.123.89.64 - - [22/Aug/2006:15:49:37 -0400] "GET / HTTP/1.1" 200 242 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20050921 Red Hat/1.0.7-1.4.1 Firefox/1.0.7"
192.168.75.13 - - [09/Jul/2007:16:11:47 -0400] "GET /app/showXmlDataContent.do HTTP/1.1" 200 28145 "-" "Java/1.5.0_08"

head closes its filehandle after reading the requisite number of lines and that makes baby tee cry. So, I’m directing tee’s stderr to /dev/null so it masks the ‘Broken pipe‘ error. That will also mask any other tee error that could arise, but in this simple usage it’s not a concern.

Note that the >(command) syntax for a temporary pipe does not allow a space between the >(

The temporary pipe command can be piped to clean up the output:

$ zcat access_log.20070709.all.gz | tee \
>(head -n1 | cut -d' ' -f4) \
>(tail -n1 | cut -d' ' -f4) &>/dev/null

[22/Aug/2006:15:49:37
[09/Jul/2007:16:11:47

SSH SOCKS Proxy From Behind a Gateway

Lifehacker a nice write up on Firefox web browsing with SOCKS proxies. The tip about network.proxy.socks_remote_dns was new to me and I will have to play with that sometime. Safari, my primary browser, seems to resolve DNS requests at the proxy by default so that saves me the hassle in the meantime.

One of the take home messages of the Lifehacker entry is that you can run “ssh -D 1080 server.remotehost.com” on your workstation, then configure Firefox (as well as most other browsers) to use a SOCKS proxy at localhost port 1080. This provides encrypted communications between your workstation and server (great when your workstation is on an untrusted wireless network) and for masquerading as the server (useful when accessing websites that are behind a firewall or that restrict access by IP address).

Very simple, extremely handy. But what if you want to use remote server that is behind a firewall and only accessible via a gateway machine?

----------------               -------------         -------------
| workstation  |               |           |         |  server   |
|              | --------------|  gateway  | ------- |           |
|(web browser) |               |           |         |  (SOCKS)  |
----------------               -------------         -------------

In that case you have to tunnel through the gateway to get to the SOCKS server running on the server. In this post I’m going to walk though building up the ssh command that will achieve such a tunnel. I will then present an alternate, more generic method.

scp is slow? Hey, not so fast.

Last week SpikeLab.org posted a set of benchmarks for scp, tar over ssh and tar over netcat. The loser in SpikeLab’s environment was scp, coming in at roughly three times slower than tar over ssh (10m10s vs 3m18s, respectively) for a directory of two hundred 10MB files.

Interesting. I had not ever noticed scp performing any worse than ssh but then again I had never compared them directly. I decided to run my own unscientific tests on my servers to see if I’m in the same boat.

The Hardware

sender: RedHat EL4, OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003
16GB memory, 4x Dual Core AMD Opteron
receiver: Apple OSX Server, OpenSSH_4.5p1, OpenSSL 0.9.7l 28 Sep 2006
1GB memory, 2GHz PowerPC G5

These are on a 1Gb/s network. There are four router hops between them and potentially competing traffic on the network so I repeated the tests a few times during non-peak hours to minimize effects of traffic interference.

The Tests

First I created a directory of two hundred 10MB files.

$ until [ `ls|wc -l` -gt 199 ]; do let $((i=$i+1)); dd if=/dev/urandom of=$i bs=10k count=1k; done;

Then I ran a series of scripted file transfer tests, modeling after SpikeLab’s tests. The script is posted at the end of this post.

$ ./perfTests.sh

The Results

A representative result of one of the benchmarks for transferring urandom-generated files across the network is shown in Table 1.

Table 1.

command	compression	time
scp	no	251.04s
scp	ssh	262.37s
tar	no	264.99s
tar	ssh	267.34s
tar	gzip	324.88s
tar	ssh and gzip	331.32s
tar	no, blowfish encryption	279.94s
nc	no	69.45s
nc	gzip	219.53s

In contrast to SpikeLab’s results, I saw no significant difference between scp and tar over ssh. Also, in my environment the addition of gzip compression to the tar transfers had a detrimental impact on the performance. Compare that with SpikeLab’s results in which the gzip compression significantly improved the transfer rates.

I can think of a few reasons for the transfer rate differences at the two sites. Different versions or build options of SSH could affect the results. ssh/scp have a number of options that can be set in configuration files so what I’m showing as the command line execution is not telling the whole story. Those behind-the-scene configurations may be affecting results.

The effects of gzip compression I see could be explained by the randomness of the files being compressed. The Table 1 results were using files generated from the contents of /dev/urandom. If I repeat the tests with files composed uniformly of NULL characters from /dev/zero then gzip gives a marked improvement (Table 2) on the transfer times. The more random the contents of a file the less compression gzip can achieve. In fact, if the contents is fully random, as the case here, no compression can take place and the size of the compressed file will actually be larger due to gzip’s accounting overhead stored with the file. So in some cases gzip can introduce compute time overhead with no reduction in data sent over the wire. The NULL files from /dev/zero compress nicely - the 2GB directory can be compressed down to a 2MB tarball - so the bandwidth savings is substantial.

Table 2.
Trials using non-random files generated from /dev/zero

command	compression	time
scp	no	271.80s
scp	ssh	264.76s
tar	no	269.62s
tar	ssh	272.20s
tar	gzip	78.33s
tar	ssh and gzip	76.25s
tar	no, blowfish encryption	277.29s
nc	no	78.51s
nc	gzip	78.12s

Interestingly enabling compression in scp/ssh had no real effect on the NULL files although it should be using the same zlib compression algorithm and same default compression level (6) as gzip. The CPU on the receiver seems to be the limiting factor with gzip compression over netcat so no improvement was seen there. The previous ssh results were using ssh version 2. If I use ssh version 1, with and without compression, I do see a dramatic difference (Table 3).

Table 3.
SSH-1 and /dev/zero data

command	compression	time
scp	no	587.42s
scp	ssh	98.88s
tar	no	687.80s
tar	ssh	93.92s
tar	gzip	78.05s
tar	ssh and gzip	87.90s

As an aside, I tested SSH blowfish encryption which is reportedly faster than the default AES. However I saw no improvement to the transfer rate by using that algorithm (Table 1).

I think in summary all this highlights the need to benchmark specific environments and adjust accordingly. Your mileage may vary.

rsync and sudo over SSH

I do not have a root password for many of the servers I interact with so I can not SSH directly in as the root user. Also, the ssh daemons are wisely configured with ‘PermitRootLogin’ set to ‘no’ so a password would be moot anyway. I do have sudo permissions on the servers so I can connect under my username and sudo the privileged commands as needed. Glazed-eye screen-staring started when I needed to rsync a remote directory that was read-only for root. How do I get rsync to run under sudo on the remote server? I did some searching and here are some options I found.

Option 1. Set NOPASSWD in the /etc/sudoers file.

crashingdaily ALL= NOPASSWD:/usr/bin/rsync

Then use the --rsync-path option to specify the sudo wrapper.

rsync -a -e "ssh" --rsync-path="sudo rsync" crashingdaily@server.remotehost.com:/u02/data_pump_dir/ /archive

Option 2. For interactive usage, I can pre-activate sudo and then run rsync as in Option 1.

stty -echo; ssh server.remotehost.com sudo -v; stty echo

rsync -a -e "ssh" --rsync-path="sudo rsync" crashingdaily@server.remotehost.com:/u02/data_pump_dir/ /archive

The “stty -echo” and “stty echo” is used to temporarily disable the display of the keyboard input to prevent the sudo password from being displayed.

Credits: Wayne Davison and Julian Cowley

Option 3. If sudo is not available, there is possibly an option to use “su”. I was unable to get this to work. su seems to insist on a tty - I get the error ’standard in must be a tty’. (In this case I do have a root password to use with su, so that’s not an issue).

Create a wrapper script, /usr/local/bin/su-rsync, on the remote server and make it executable.

#!/bin/sh
su - -c "rsync $*"

Then call that script with the --rsync-path option.

rsync -a -e "ssh" --rsync-path=/usr/local/bin/su-rsync crashingdaily@other.remotehost.com:/u02/data_pump_dir/ /archive

Credit: Wayne Davison

Option 4. Set ‘PermitRootLogin’ to ‘yes’ on the remote server and use SSH key authentication to login directly as the root user. This isn’t really an option for me but I throw it out there for sake of completeness.

Related:

rsync

Re: how to use option for rsync

rsync using sudo via remote shell

Port Forwarding Through the Looking Glass

If you don’t know where you are going, any road will take you there.
- Lewis Carroll

My production servers reside behind a perimeter firewall in a data center. A minimal set of ports are open to the world, notably port 22 for sshd and port 80 for the Apache webservers which proxy requests to one of several Tomcat instances. The Tomcat ports are blocked at the data center’s perimeter firewall which means no direct access to Tomcat’s manager interfaces. But that’s OK, there are several options for reaching the Tomcat manager from outside the data center. I’ll glance over three options and then delve into a fourth option that is the gooey center of this posting.

pwd and symlinks

It’s the simple things in life… Sometimes you simply want to know the canonical path for your current working directory.

Here is a symbolic link to a directory.

$ ls -go /home/crashingdaily/symlink
lrwxrwxrwx 1 14 Mar 29 10:54 /home/crashingdaily/symlink -> im/a/real/path

I change to the directory using the symbolic link.

$ cd /home/crashingdaily/symlink

The ‘pwd’ bash builtin does not resolve symbolic links so reports that I’m in the ’symlink’ directory.

$ pwd
/home/crashingdaily/symlink

‘/bin/pwd’ reports the canonical path.

$ /bin/pwd
/home/crashingdaily/im/a/real/path

I can leverage this to quickly change my path from symlinked to real.

$ pwd
/home/crashingdaily/symlink

$ cd `/bin/pwd`

$ pwd
/home/crashingdaily/im/a/real/path

Also see

readlink -f .

readlink is advantageous if you need to resolve paths outside your current working directory.

Update: Sometimes the simpler things are right under your nose.

pwd -P

and

cd -P .

were sitting there all along. No need for /bin/pwd (for bash at least). RTFM, indeed.

A Workaround For Net::SFTP Hanging On Large Transfers

I ran into an issue with the Perl module Net::SFTP that drives the file transfers in a script that I use. Net::SFTP is built on top of Net::SSH::Perl, a pure Perl implementation of the SSH-2 protocol.

The issue is that after 1 GB of data is transferred the connection hangs indefinitely.

I found something similar was reported before in the Net::SSH::Perl bug tracker a couple of years ago. Though in that case the hanging was after 1 hour time rather than 1 GB data transfered. The bug report was rejected by the maintainer just the other day with the comment to “try the latest version”.

Well, I have the latest version of Net::SSH::Perl and Net::SFTP from CPAN (1.30) and still have the issue.

I did some digging.

On the server side I have

OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003

If I turn on sshd debugging I get this logged at the time of the transfer stalling:

Feb 15 23:21:55 olive sshd[19497]: debug1: need rekeying
Feb 15 23:21:55 olive sshd[19497]: debug1: SSH2_MSG_KEXINIT sent

The client Perl script using Net::SFTP with debugging on, reports:

sftp: In read loop, got 8192 offset 1069547520
sftp: Sent message SSH2_FXP_READ I:130563 O:1069555712
Warning: ignore packet type 20
channel 1: window 16358 sent adjust 16410
sftp: Received reply T:103 I:130563
sftp: In read loop, got 8192 offset 1069555712
sftp: Sent message SSH2_FXP_READ I:130564 O:1069563904

I did some more digging and found that the ‘packet type 20′ that is being ignored corresponds to the SSH2_MSG_KEXINIT sent by the server and is a request to initiate session rekeying.

The rekeying was originally something that the commercial SSH2 package did and that sometimes caused problems early on when talking to an OpenSSH clients.

That was then. This is now and it seems that OpenSSH 3.9 knows all about rekeying sessions (I think this was added in 3.6?). The OpenSSH 3.9 server I was connecting to automatically attempts rekeying with SSH-2 clients after ~1GB of data transfer and as far as I can tell there’s no configuration to turn if off. There is an poorly documented ‘RekeyLimit’ for that version that is supposed to control how much data is transfered before rekeying but it had no effect in my hands and I do not see it attempting to regulate anything in the source code (though I’m not very good at C so maybe I’m missing something).

The proper fix I suppose would be to patch Net::SSH::Perl so it implements rekeying. However, I’m not nearly well versed enough in the SSH-2 protocol to do that, so I needed to work around this issue.

After further digging around in the OpenSSH source code I discovered that the server is compiled with a list of known clients that do not support session rekeying (see client strings with SSH_BUG_NOREKEY in OpenSSH compat.c).

Net::SSH::Perl reports itself to the server as version 1.30 (which it is) as the logging from the OpenSSH server in debug mode demonstrates:

Feb 18 13:48:09 olive sshd[18746]: debug1: Client protocol version 2.0; client software version 1.30

‘1.30′ is not on OpenSSH’s compat.c list of SSH_BUG_NOREKEY clients. I could add it to the list and recompile but I don’t have admin access to the sshd server I use in production (the debug messages were from my test server). Therefore the fix has to be in the client.

So, I changed the $VERSION of Net::SSH::Perl to ‘OpenSSH_2.5.2′. The OpenSSH server recognizes it’s talking to a non-rekeying client and skips the session rekeying process. Now large transfers complete without hanging.

Process Uptime

I have a shell script to manage and report on my Tomcat instances. I wanted the ’status’ portion of the script to report on instance uptime (which, by the way, has improved significantly since switching to JRockit). The script was already reporting the PID of the parent tomcat process so I shoved in this one-liner that takes that PID and gets the elapsed time from ps. I filter the result through grep and sed to get a clean human-readable output.

uptime = `ps -o etime $PID |grep -v ELAPSED | sed ’s/\s*//g’ | sed “s/\(.*\)-\(.*\):\(.*\):\(.*\)/\1d \2h/; s/\(.*\):\(.*\):\(.*\)/\1h \2m/; s/\(.*\):\(.*\)/\1m \2s/”`

echo $uptime

The output is formated as one of days&hours, hours&minutes, minutes&seconds.

6d 08h
03h 23m
20m 56s

Anyone got a better or different way?

Copy One File To Multiple Directories

ls -d /usr/local/tomcat_instances/{InstanceA, InstanceB, InstanceC, InstanceD}/conf/Catalina/localhost | xargs -i{} cp /usr/local/tomcat_instances/Instance_Template/conf/Catalina/localhost/ROOT.xml {}

rsync with SSH Forwarding Across a Gateway

rsync -a -e “ssh gateway.remotenet.org ssh server.remotenet.org” :/logs /sync/logs

That is all.

Mounting Filesystems with SSH Forwarding Across a Gateway

sshfs is wickedly handy for mounting remote directories on your local filesystem. Recently I needed to mount the /logs directory off a remote server so a program on my workstation could process log files in /logs.

The textbook command to do that would be:

[me@workstation]$ sshfs server.remotenet.org:/logs /mnt/svrlogs

The tricky part in this particular case is that the server is on a private network so my workstation can not directly access it. I’m required to first ssh to a gateway machine and then ssh to the server.

----------------               -------------         -------------
|workstation   |               |           |         |  server   |
|              | --------------|  gateway  | ------- |           |
|/mnt/svrlogs  |               |           |         |   /logs   |
----------------               -------------         -------------

I found three ways to work with this scenario. I’d love to hear of more ways and get feedback on these.

Crashing Tomcat

I’ve been exploring replacing Sun’s JVM with JRockit as a means to improving the uptime of our Struts-based web applications running under Tomcat. The problem I’ve been having is this. Repeatedly reloading our Struts-based web application (something we do frequently during development) causes Tomcat to stop responding to requests after about 10-ish reloads. Other techniques for updating a single webapp running under a live Tomcat instance - undeploy/deploy, stop/start - exhibit the same symptoms. The only out is to restart the entire Tomcat instance - disrupting all the other fully functional webapps deployed there.

Initially Tomcat logs were of no help to me but after upgrading Tomcat to 5.5 (from 5.0.28) and Sun’s JDK to 1.5.0_10 (from 1.5.0_05) I started getting Tomcat log entries that suggested the problem.

java.lang.OutOfMemoryError: PermGen space

That’s something Google can sink its teeth into and it quickly spit out the digested remains of the other poor saps that have fallen prey to this. The likely problem of this memory leak is due to incomplete garbage collection of the Classloader such that classes persist in Sun’s JVM PermGen memory heap - and therefore for the life of the JVM.

The simplest workaround (short of tracking down the sources of memory leaks in the app itself) suggested to replace Sun’s JVM with BEA’s JRockit. JRockit doesn’t have a PermGen heap so there can be no leaking there.