Porting RabbitMQ Tutorials to Perl Net::RabbitMQ

I’ve begun playing with message and task queues to see how they might benefit my projects. I have several ideas, most of which are out of my immediate reach. I still have a lot to learn. I have one use case for a simple message queue that I think will be a good starter for me.

I’ve settled on RabbitMQ for the broker. It’s certainly capable of handling my simple starter project and it is a safe bet that I will be able to grow into it for a long time to come. The message producer for my first use case will be a Perl script (for reasons not worth explaining right now). I do have more flexibility with the message consumer, I’m thinking of incorporating that into a Python/Django application, but I’ll keep that in Perl as well to start with.

There are a few Perl modules available supporting RabbitMQ and AMQP. The RabbitMQ tutorials for Perl use Net::RabbitFoot. I decided to use Net::RabbitMQ because of its simplicity and lack of dependancies on other modules. It also forced me to work through the existing RabbitMQ tutorials and rewrite them – a good educational opportunity. I used the Python/pika tutorials as the source of my porting. This gave me some exposure to pika, which I will need if I later start working in Python. I’ve got my eye on Celery but I’m not ready for that level of sophistication yet.

The fruits of my labor are available at crashingdaily/rabbitmq-tutorials-perl on GitHub. My Perl ports are compatible with the official Python/pika examples. That is, you can use the Perl scripts to consume from the Python producers and vice versa.

Related Sites

Redoing RabbitMQ’s tutorial – Marco Marongiu’s rewrite of several tutorials, also using Net::RabbitMQ. His examples provided a nice a jump start for my adventure.

perl pie in your face

I frequently use Perl’s in place file editing from the command line. What I didn’t consider until it bit me today is that the file ownership can change using this method.

Here’s the original file, owned by tomcat_6 and only readable by user and group.

$ ls -l web.xml

-rw-rw----  1 tomcat_6 tomcat 49384 Jul 10 11:38 web.xml

I belong to the tomcat group, so have write permissions to the file. The enclosing directory is also tomcat group writable. The importance of this is noted below.

Using the ‘perl pie’ one-liner to make an in place edit if the file:

$ perl -p -i -e 's;<session-timeout>\d+</session-timeout>;\
<session-timeout>1440</session-timeout>;' web.xml

Now the file is owned by me and my default group.

$ ls -l web.xml

-rw-rw----  1 crashing  daily 49384 Jul 10 21:55 web.xml

Most critically, now the file is no longer readable by the tomcat processes. This little change prevented my Tomcat server from starting. Ouch.

sed is a little nicer. It changes the owner but not the group.

$ sed -i 's;<session-timeout>.*</session-timeout>;\
<session-timeout>1445</session-timeout>;g' web.xml

$ ls -l web.xml 

-rw-rw----  1 crash tomcat 49385 Jul 10 22:44 web.xml

Neither the Perl nor the sed one-liners work if the directory is not writable because Perl and sed require unlinking the original file and replacing it with a new version.

The winner for both maintaining file ownership and working if the directory is not writable is ed.

$ ed - "web.xml" <<EOF
,s;<session-timeout>[[:digit:]]*</session-timeout>;\
<session-timeout>1440</session-timeout>;
w
EOF

ed truly does an in place edit. Nice. If only I could remember the syntax.

Props to Net::SFTP::Foreign::Compat

This is a belated posting about dealings I had back in March with the slowness of Net::SFTP even with Math::BigInt::GMP installed. In short, I successfully replaced Net::SFTP with Net::SFTP::Foreign in my scripts and am now enjoying native transfer rates.

Foreign uses the system’s ssh clients instead of a pure pure implementation
as Net::SFTP does. With Net::SFTP::Foreign I reduced a 1.2GB download from 4 hours to 20 minutes.

It also doesn’t have the 1GB ssh rekeying hang that an unpatched Net::SFTP has.

There is a Compat sub-module that allows Net::SFTP::Foreign to work as a drop in replacement to Net::SFTP. In version 0.90_15 I had to comment out line 85 of Compat.pm, dontsave => !defined($remote), to get it to work. I didn’t report this to the maintainer at the time (sorry). I now see that there have been over a dozen releases of Net:SFTP::Foreign since I investigated this and I don’t know if this modification is still necessary.

I recommend Net::SFTP::Foreign where speed is important and a native client is available. But before rushing out to adopt it, do know that it prefers SSH key authentication. It won’t take passwords without some hoop jumping as described in the perldoc.

Inspecting Perl DBI Placeholders

I frequently use placeholders and bind values with Perl DBI and during debugging phases I frequently want to see what values are being bound. The bound values are available from the statement handle’s ParamValues.

%bound_values = %{ $sth->{ParamValues} }; 

Here’s a simple script to illustrate.

#!/usr/bin/perl

use DBI;
use Data::Dumper;

my $dbh = DBI->connect(
    'dbi:Oracle:cd_sid', 
    'cdaily',
    'pa55wd');

my $sql = "select 'something' from dual where 1 = ? and 2 = ?";

my $sth = $dbh->prepare($sql);
$sth->execute(1, 2);


print Dumper $sth->{ParamValues};

Will print a result like:

$VAR1 = {
          ':p1' => 1,
          ':p2' => 2
        };

This is handy but awkward to work with when there are a lot of placeholders. Is there a simple way to print the param values in the context of the sql statement? Like so,

select 'something' from dual where 1 = 1 and 2 = 2

That would be a swell thing to be able to do.

Related:

What are placeholders in DBI, and why would I want to use them?
DBI perldoc
Tim Bunce’s Advanced Perl DBI

A Workaround For Net::SFTP Hanging On Large Transfers

[update: This issue is reportedly fixed in Net::SSH::Perl 1.34]

I ran into an issue with the Perl module Net::SFTP that drives the file transfers in a script that I use. Net::SFTP is built on top of Net::SSH::Perl, a pure Perl implementation of the SSH-2 protocol.

The issue is that after 1 GB of data is transferred the connection hangs indefinitely.

I found something similar was reported before in the Net::SSH::Perl bug tracker a couple of years ago. Though in that case the hanging was after 1 hour time rather than 1 GB data transfered. The bug report was rejected by the maintainer just the other day with the comment to “try the latest version”.

Well, I have the latest version of Net::SSH::Perl and Net::SFTP from CPAN (1.30) and still have the issue.

I did some digging.

On the server side I have

OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003

If I turn on sshd debugging I get this logged at the time of the transfer stalling:

Feb 15 23:21:55 olive sshd[19497]: debug1: need rekeying
Feb 15 23:21:55 olive sshd[19497]: debug1: SSH2_MSG_KEXINIT sent

The client Perl script using Net::SFTP with debugging on, reports:

sftp: In read loop, got 8192 offset 1069547520
sftp: Sent message SSH2_FXP_READ I:130563 O:1069555712
Warning: ignore packet type 20
channel 1: window 16358 sent adjust 16410
sftp: Received reply T:103 I:130563
sftp: In read loop, got 8192 offset 1069555712
sftp: Sent message SSH2_FXP_READ I:130564 O:1069563904

I did some more digging and found that the ‘packet type 20’ that is being ignored corresponds to the SSH2_MSG_KEXINIT sent by the server and is a request to initiate session rekeying.

The rekeying was originally something that the commercial SSH2 package did and that sometimes caused problems early on when talking to an OpenSSH clients.

That was then. This is now and it seems that OpenSSH 3.9 knows all about rekeying sessions (I think this was added in 3.6?). The OpenSSH 3.9 server I was connecting to automatically attempts rekeying with SSH-2 clients after ~1GB of data transfer and as far as I can tell there’s no configuration to turn if off. There is an poorly documented ‘RekeyLimit’ for that version that is supposed to control how much data is transfered before rekeying but it had no effect in my hands and I do not see it attempting to regulate anything in the source code (though I’m not very good at C so maybe I’m missing something).

The proper fix I suppose would be to patch Net::SSH::Perl so it implements rekeying. However, I’m not nearly well versed enough in the SSH-2 protocol to do that, so I needed to work around this issue.

After further digging around in the OpenSSH source code I discovered that the server is compiled with a list of known clients that do not support session rekeying (see client strings with SSH_BUG_NOREKEY in OpenSSH compat.c).

Net::SSH::Perl reports itself to the server as version 1.30 (which it is) as the logging from the OpenSSH server in debug mode demonstrates:

Feb 18 13:48:09 olive sshd[18746]: debug1: Client protocol version 2.0; client software version 1.30

‘1.30’ is not on OpenSSH’s compat.c list of SSH_BUG_NOREKEY clients. I could add it to the list and recompile but I don’t have admin access to the sshd server I use in production (the debug messages were from my test server). Therefore the fix has to be in the client.

So, I changed the $VERSION of Net::SSH::Perl to ‘OpenSSH_2.5.2’. The OpenSSH server recognizes it’s talking to a non-rekeying client and skips the session rekeying process. Now large transfers complete without hanging.