Missing RPM package conflicts when installing updates

While making some custom RPMs and then remaking them with altered build steps, I stumbled across the behaviour that rpm will let you install updated packages even when an older package is already installed, provided that none of the files in the new package have changed.

Here’s a demonstration using a simple spec file (Listing 1) for an RPM package that installs a README file in /tmp/rpmtest/. Note the %install process creates the README file with some text in it.

Build the RPM package from the spec file:

$ rpmbuild -bb rpmtest.spec 

and install it:

$ sudo rpm -i /scratch/RPM/RPMS/x86_64/rpmtest-1-1.x86_64.rpm

Now, the single README file is installed:

$ ls -l /tmp/rpmtest/README 
-rw-r--r-- 1 cdaily cdaily 10 Nov 14 16:39 /tmp/rpmtest/README

Now we’ll update the rpmtest.spec file to build Version 1, Release 2. Then we change the rpm’s install process but not the README contents. For example, in the rpmtest.spec change the defattr, line 22, from

%defattr(-, %(%{__id_u} -n), %(%{__id_u} -n), -)

to

%defattr(-, root, root)

and change the spec file’s Release to

Release: 2

(line 4).

The generated README file will be the same, only the file ownership changes between releases.

Build as before and try to install.

$ sudo rpm -ivh /scratch/RPM/RPMS/x86_64/rpmtest-1-2.x86_64.rpm

That worked. I’m able to install Version 1, Release 2 and it does not conflict with Release 1.
The README is now owned by root.

$ ls -l /tmp/rpmtest/README 
-rw-r--r-- 1 root root 10 Nov 14 16:47 /tmp/rpmtest/README

I have both packages installed:

$ rpm -qa rpmtest
rpmtest-1-1
rpmtest-1-2

And the README is owned by both packages:

$ rpm -qf /tmp/rpmtest/README 
rpmtest-1-1
rpmtest-1-2

The same happens if you increment the Version. As long as the checksums for all the package files are unchanged, the rpm install will proceed.

Remove them both:

$ sudo rpm -e rpmtest-1-1 rpmtest-1-2

Let’s see what happens when the change a package file. In the rpmtest.spec file, change line 16

echo "This is a readme" > $RPM_BUILD_ROOT/tmp/rpmtest/README

to

echo VERSION %{version} RELEASE %{release} > $RPM_BUILD_ROOT/tmp/rpmtest/README

That is, the README contents will now have the version and release numbers and so will change with each release.

After building and installing Release 1, then building Release 2, I am unable to install Release 2.

$ sudo rpm -i /scratch/RPM/RPMS/x86_64/rpmtest-1-2.x86_64.rpm
	file /tmp/rpmtest/README from install of rpmtest-1-2.x86_64 conflicts with file from package rpmtest-1-1.x86_64

This time because the README has changed between releases, rpm reports a conflict.

In contrast to installing, upgrading a package release (rpm -U rpmtest-1-2.x86_64.rpm ) will replace Release 1 with Release 2 even when no files have changed. The upgrade option will also install the rpm if an earlier version/release is not already installed, so if you routinely use the upgrade command instead of the install, you don’t have to be concerned with ending up with multiple package release installations.

In summary, rpm only reports install conflicts if files change, it doesn’t make any comparisons of Version or Release numbers.

This seemed odd to me at first but the flexibility it provides does make sense after I thought about it more. So what if multiple packages own the same file? It happens. One common case is with packages for multiple architectures. Consider the neon library as a random example. I have versions installed for both the i386 and x86_64 architectures. The documentation is the same between the two, only the libneon.so shared objects differ and these are installed in different directories so there’s no conflict.

On the other hand, the observation that a second package installation can change file ownership and permissions of the first package could conceivably be a problem.

Related:

Maximum RPM

Persisting ssh connections while changing networks

I recently re-discovered autossh. I’ve been using it to persist mounted sshfs volumes but am only just now realizing it’s the scratch for some of my other itches.

With it I can

autossh -M50000 -t -D1080 -Nf crashingdaily.com

to keep a seemingly persistent SOCKS proxy running.

Or

autossh -M50000 -t crashingdaily.com 'screen -aAdR autossh'

to maintain an interactive session.

The value that I missed earlier is that the connections are maintained (re-established, actually) even if I change networks – which my laptop and I often do.

Jon’s View has a nice posting on using autossh for SOCKS proxying.

And Dread Pirate PJ has a good instruction for combining autossh and screen for persistent interactive sessions.

autossh is available for OS X via Darwin Ports and Fink and for Linux as source code or as binary packages from repositories for most modern Linux distributions.

Scheduling jobs with at and cgi scripts

The at command can be used in CGI scripts to remotely trigger non-recurring jobs on a web server. This is especially useful for long running jobs that clients don’t need to wait on and jobs that need to persist across web daemon restarts.

My initial simple attempts to schedule a job with at in a CGI script executed by an Apache webserver were met with the failure response “This account is currently not available“. That message being the output of /sbin/nologin which is the shell value set for the apache user in /etc/password.

The fix is to set the SHELL environment variable to a proper shell when executing at. Here’s a simple, proof-of-concept CGI script.

#!/usr/bin/perl
use CGI;
my $cgi = new CGI;
print $cgi->header;

system(‘echo “/usr/bin/env > /tmp/apacheenv” | SHELL=/bin/bash at now’);

Calling the CGI script triggers the at scheduling. In this case, the execution of /usr/bin/env is immediate and successful.

$ cat /tmp/apacheenv
SERVER_SIGNATURE=
SHELL=/bin/bash
HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_7; en-us) AppleWebKit/528.18.1 (KHTML, like Gecko) Version/4.0 Safari/528.17
SERVER_PORT=80

Related:

at man page

Grepping nonprinting characters

Here is a tip summarized from comp.unix.shell

The problem:

> For the following TAB-delimited records, I want to count number of 
> records with column-2 == -1   (should be 2) 
> ===== file.txt ====== 
> AAA    -1    2008-07-14 
> BBB    -14   2008-07-15 
> CCC    -20   2008-07-16 
> DDD    -1    2008-07-16 
> =========== 
> I tried: 
>   grep -c -- "-1\t" file.txt 
> which is not working

A solution offered for ksh93/zsh/bash shells:

grep -c -- $'-1\t' file.txt 

I will add another alternative

grep -c '-1^V^I' file.txt

Where ^V^I means type ctrl-v and ctrl-i to enter a tab character.

Related:

Advanced Bash-Scripting Guide Example 34-1
Insert ASCII Control Characters in Text

perl pie in your face

I frequently use Perl’s in place file editing from the command line. What I didn’t consider until it bit me today is that the file ownership can change using this method.

Here’s the original file, owned by tomcat_6 and only readable by user and group.

$ ls -l web.xml

-rw-rw----  1 tomcat_6 tomcat 49384 Jul 10 11:38 web.xml

I belong to the tomcat group, so have write permissions to the file. The enclosing directory is also tomcat group writable. The importance of this is noted below.

Using the ‘perl pie’ one-liner to make an in place edit if the file:

$ perl -p -i -e 's;<session-timeout>\d+</session-timeout>;\
<session-timeout>1440</session-timeout>;' web.xml

Now the file is owned by me and my default group.

$ ls -l web.xml

-rw-rw----  1 crashing  daily 49384 Jul 10 21:55 web.xml

Most critically, now the file is no longer readable by the tomcat processes. This little change prevented my Tomcat server from starting. Ouch.

sed is a little nicer. It changes the owner but not the group.

$ sed -i 's;<session-timeout>.*</session-timeout>;\
<session-timeout>1445</session-timeout>;g' web.xml

$ ls -l web.xml 

-rw-rw----  1 crash tomcat 49385 Jul 10 22:44 web.xml

Neither the Perl nor the sed one-liners work if the directory is not writable because Perl and sed require unlinking the original file and replacing it with a new version.

The winner for both maintaining file ownership and working if the directory is not writable is ed.

$ ed - "web.xml" <<EOF
,s;<session-timeout>[[:digit:]]*</session-timeout>;\
<session-timeout>1440</session-timeout>;
w
EOF

ed truly does an in place edit. Nice. If only I could remember the syntax.

Hashing the executables – a look at hash and type

Here is a brief look at the Linux commands, type and hash.

To start, I open a new terminal and check for the location of the foo executable using the which command. (foo is a trivial shell script I concocted for illustrative examples in this posting.)

$ which foo
/usr/local/bin/foo

which tells me the location of the executable file in my $PATH but that is not necessarily what will be executed when I call foo on the command line. To learn that, I use the type command.

$ type foo
foo is /usr/local/bin/foo

type reports how a word will be interpreted if used as command name. In this case, it is telling me that using foo as a command will execute the file /usr/local/bin/foo. That happens to be what which also reported, but, as we will see in the following examples, that will not always be the case.

I can run the foo command to print its release version.

$ foo --version
foo release 1.0

Consider if I install a new foo, version 2.0, in my ~/bin directory, leaving the 1.0 version in /usr/local/bin/foo. I have positioned ~/bin ahead of /usr/local/bin in my $PATH and the which command confirms the new 2.0 version is found first.

$ which foo
/home/crashingdaily/bin/foo

However, when I call for foo to be executed it still executes the old 1.0 version.

$ foo --version
foo release 1.0

Hmm, what’s going on? The which command reported the first executable found in my path but type will be more informative. It will tell us precisely what file, function, builtin, keyword or alias is associated with a given command name.

$ type foo
foo is hashed (/usr/local/bin/foo)

This is reporting that the shell has saved the meaning of foo in a hash table as /usr/local/bin/foo (the 1.0 version). Caching the command in a hash table is an optimization that saves the shell from having to search $PATH every time. A given shell does this the first time a command is run in that shell instance. The hashed values survive for the life of the shell instance. Start a new shell, e.g. by opening a new terminal or invoking a subshell, and you start a new hash table for that shell process.

The hash table can also be manually manipulated to clear or set values. Enter the hash command. This command is used to print and edit the shell’s command hash table.

The entire hash table can be cleared

$ hash -r

or you can delete a specific entry

$ hash -d foo

Having cleared the hash table, invoking foo now calls the first in my path.

$ foo --version
foo release 2.0

And this copy is now placed in the hash table.

$ type foo
foo is hashed (/home/crashingdaily/foo)

I can also print that entry of the hash table to get its value. However, do not use this as a substitute for type as it only deals with files. More on that shortly.

$ hash -t foo
/home/crashingdaily/foo

As you might imagine, if I delete the executable whose value has been hashed,

$ rm /home/crashingdaily/foo

the command is not found when I attempt to use it,

$ foo --version
-bash: /home/crashingdaily/bin/foo: No such file or directory

until I update the hash. I can clear the hash as before or I can explictly set the value like so

$ hash -p /usr/local/bin/foo foo

$ type foo
foo is hashed (/usr/local/bin/foo)

$ foo --version
foo release 1.0

Keep in mind that the hash table caches (and the hash command reports) file and path names. It does not deal with keywords, functions, builtins or aliases which may be invoked by a command name. So, type is the utility to use for learning what will execute when a command name is invoked.

To further illustrate type usage, I will define a function named foo.

$ function foo { echo 'hello world'; }

Because function names take precedence over file executables having the same name, invoking foo executes the function instead of the shell script. Now, type reports

$ type foo
foo is a function
foo () 
{ 
    echo 'hello world'
}

and invoking the name gives us the expected function output.

$ foo
hello world

Using type with the -a option reports all the known values of foo

$ type -a foo
foo is a function
foo () 
{ 
    echo 'hello world'
}
foo is /home/crashingdaily/bin/foo
foo is /usr/local/bin/foo

In summary, use type to learn what a given command name currently means to the shell – there could be more than one meaning. Use hash to manipulate cached associations of command names with files (and be aware that the command name may be associated with a non-file that takes precedence).

Additional documentation for type and hash is availble via the help command.

$ help type
type: type [-afptP] name [name ...]
    For each NAME, indicate how it would be interpreted if used as a
    command name...

$ help hash
hash: hash [-lr] [-p pathname] [-dt] [name ...]
    For each NAME, the full pathname of the command is determined and
    remembered....

History meme

History Meme

$ history|awk '{a[$2]++} END{for(i in a){printf "%5d\t%s\n",a[i],i}}'|sort -rn|head
  77  whoami
  74  tic
  72  gawk
  70  od
  70  bash
  69  shred
  66  killall
  58  sleep
  55  id
  55  play

Rude awakening

BASH Cures Cancer has yet another interesting post this week. This one, entitled “Command Substitution and Exit Status“, leverages the exit status returned from a BASH sub-shell. That alone is a useful tip but what tickled me was the author’s trick of shortening the delay in a running loop by starting a newer, shorter sleep loop that kills off the longer-running sleep process. Clever. The thought of dueling shell processes – “I’m going to sleep for a minute.”, “Oh, no you’re not!” – makes me chuckle. Sigh. I am easily amused.

Clearing environment variables in BASH scripts

When you execute a shell script it inherits all the environment variables present in the parent shell. Sometimes that can cause unintended consequences. For example, I recently ran into a situation where one of my scripts in common use by our group failed for one of the users. I eventually tracked it down to the JAVA_HOME variable that user had set in his bash profile. The user’s JAVA_HOME being inherited by the script was not set to an appropriate value. Ideally the script should have set all the environment variables it needs. However, in this case, I had failed to explicitly set JAVA_HOME in my script, an oversight masked by the fact my own JAVA_HOME was set to a valid value so the script ran normally for me.

To reduce the chance of this type of problem reoccurring with other variables I decided to clear all the inherited environment variables at the start of the script. That way any other environment dependency not defined by the script would immediately reveal itself during the script debugging phase.

I could not find a simple builtin method to run the script without the inherited environment, (aside from the impractical “env -i scriptname“) [ edit 8/28/2010: see Gordon’s env - /bin/bash suggestion below and you probably don’t need to read the rest of this posting ], so I put a one-liner in the script to unset each variable (BASH syntax):

unset $(/usr/bin/env | egrep '^(\w+)=(.*)$' | \
  egrep -vw 'PWD|USER|LANG' | /usr/bin/cut -d= -f1);

env prints the environment’s variable=value pairs. The cut at the end splits the pairs and returns the variable name.

The first egrep filters for the variable=value pairs that start at the beginning of the line. This was necessary because GNU screen sets a multiline TERMCAP variable:

TERMCAP=SC|screen|VT 100/ANSI X3.64 virtual terminal:\
        :DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:UP=\E[%dA:bs:bt=\E[Z:\
        :cd=\E[J:ce=\E[K:cl=\E[H\E[J:cm=\E[%i%d;%dH:ct=\E[3g:\

I wanted just the first line and needed to discard the others so the cut would not split them and return bogus variable names, like ‘:DO‘, to unset.

The second egrep is optional but allows me to skip selected variables so they retain their preset values.

With this in place, if I attempt to use an environment variable in my script (or in a program called by my script) it will fail unless I explicitly set the value. This provides me the opportunity to ensure the script will behave consistently for all users on the system.

Diff two stdout streams

The diff utility reports differences between two files. If you need to find differences between one or two stdout outputs then temporary named pipes are a handy aid.

Here’s a simple example for the BASH shell to illustrate the technique. Say you have two files, A and B:

$ cat A
Tara
Dawn
Anya
Willow

$ cat B
WILLOW
ANYA
DAWN
HARMONY
TARA

The task is to find names that differ between the two lists without creating new files or editing the existing. You can do that by sorting and normalizing the letter case, then using diff to check the stdout on the fly.

$ diff -B <( sort A | tr [:lower:] [:upper:] ) <( sort B | tr [:lower:] [:upper:] )

2a3
> HARMONY

The <( ... ) syntax creates a temporary named pipe which makes the stdout of the sort | tr commands look and behave like a file, allowing diff to operate on the expected type of input.

For fun, you can see the temporary file created by the process:

dir <( sort A | tr [:lower:] [:upper:] ) 
lr-x------  1 crash daily 64 Mar  5 23:17 /dev/fd/63 -> pipe:[21483501]

This technique is not limited to diff. This should work for most any other command expecting a file for input.

Interestingly (frustratingly), it does not work for me in my BASH shell on Mac OS X. Does anyone know why? Update: Indeed someone does know. Unixjunkie has the answer. The following produces the expected output
on OS X but attempting diff produces no output.

cat <( sort A | tr [:lower:] [:upper:] ) <( sort B | tr [:lower:] [:upper:] )

ANYA
DAWN
TARA
WILLOW

ANYA
DAWN
HARMONY
TARA
WILLOW

Related:

Introduction to Named Pipes
Heads and Tails – an earlier posting that uses temporary named pipes for receiving stdin
Process Substitution, Advanced Bash-Scripting Guide