Comments on: scp is slow? Hey, not so fast. http://crashingdaily.wordpress.com/2007/07/09/scp-is-slow-hey-not-so-fast/ admin sys a terrible thing to waste Mon, 09 Nov 2009 13:06:18 +0000 http://wordpress.com/ hourly 1 By: Euro http://crashingdaily.wordpress.com/2007/07/09/scp-is-slow-hey-not-so-fast/#comment-361 Euro Fri, 31 Oct 2008 15:16:34 +0000 http://crashingdaily.wordpress.com/2007/07/09/scp-is-slow-hey-not-so-fast/#comment-361 Thank U for the article, it's very useful. BTW, the bottleneck of your scp is the network bandwidth, so the transferring time is limited for a given file size. Using Blowfish can't change anything even though it's, theoretically, much more faster. (you're see the difference while using SCP for localhost ) Thank U for the article, it’s very useful.
BTW, the bottleneck of your scp is the network bandwidth, so the transferring time is limited for a given file size. Using Blowfish can’t change anything even though it’s, theoretically, much more faster. (you’re see the difference while using SCP for localhost )

]]> By: crashingdaily http://crashingdaily.wordpress.com/2007/07/09/scp-is-slow-hey-not-so-fast/#comment-352 crashingdaily Tue, 19 Aug 2008 14:51:23 +0000 http://crashingdaily.wordpress.com/2007/07/09/scp-is-slow-hey-not-so-fast/#comment-352 Thank you for your comment, Ben. The effects of compression on random data and tests with non-random data was covered in Tables 2 and 3 and associated discussion. I started with random data because I wanted to reproduce the tests reported by SpikeLab. Thank you for your comment, Ben. The effects of compression on random data and tests with non-random data was covered in Tables 2 and 3 and associated discussion. I started with random data because I wanted to reproduce the tests reported by SpikeLab.

]]> By: Ben Clark http://crashingdaily.wordpress.com/2007/07/09/scp-is-slow-hey-not-so-fast/#comment-351 Ben Clark Tue, 19 Aug 2008 14:20:43 +0000 http://crashingdaily.wordpress.com/2007/07/09/scp-is-slow-hey-not-so-fast/#comment-351 Of course compressing will be detrimental when you are compressing random ( /dev/urandom ) data. With a real file full of very redundant data ( try some xml files ) you will get serious improvement with compression. Random data is by definition very information dense, if it weren't then it would have redundancies ( patterns ) and so would not be random. Because the data you send is random there are no patterns for the compression algorithm to find. That is why compressing an already compressed file can make it larger - the compressed file is already information dense, and it's bytes appear pretty random. The added overhead of the next 'layer' of compression takes more space than it is worth ( all the low hanging fruit have been picked by the first layer of compression ) and so the resulting file is larger. In your test, compressing the random data before sending it over the wire may have resulted in more data being send over the wire ( unless the algoritm first tries compression, but then sends the raw file if the result was larger than the raw file which it probably does, though I don't know ). There was also the computational cost involved in doing the compression. Try the test again with some real files that are susceptible to compression, such as some xml files. ( gifs, jpegs, mp3s. .movs, .zip, .Z, .gz, and .bz2 files are already compressed and will contain little redundancy to compress out ) Of course compressing will be detrimental when you are compressing random ( /dev/urandom ) data. With a real file full of very redundant data ( try some xml files ) you will get serious improvement with compression.

Random data is by definition very information dense, if it weren’t then it would have redundancies ( patterns ) and so would not be random.

Because the data you send is random there are no patterns for the compression algorithm to find.

That is why compressing an already compressed file can make it larger – the compressed file is already information dense, and it’s bytes appear pretty random. The added overhead of the next ‘layer’ of compression takes more space than it is worth ( all the low hanging fruit have been picked by the first layer of compression ) and so the resulting file is larger.

In your test, compressing the random data before sending it over the wire may have resulted in more data being send over the wire ( unless the algoritm first tries compression, but then sends the raw file if the result was larger than the raw file which it probably does, though I don’t know ). There was also the computational cost involved in doing the compression.

Try the test again with some real files that are susceptible to compression, such as some xml files. ( gifs, jpegs, mp3s. .movs, .zip, .Z, .gz, and .bz2 files are already compressed and will contain little redundancy to compress out )

]]>