Tuesday, November 11, 2008

LZMA compression becoming the better choice.

LZMA - Lempel-Ziv-Markov chain algorithm

If you have not heard about it, certainly start reading up on it and DO use it. If your using bzip2 currently then your going to like this even more.

LZMA is currently being used in openSUSE today. Have you been wondering why the install is quicker for both openSUSE 11.X and SLE11? We now use lzma to compress the content in our rpm's. The decompression is quite a bit faster than the bzip2 that was used prior. This is just one reason why its faster, but certainly adds to it quite a bit, not to mention it makes the rpm's a bit smaller too.

LZMA has several levels of compression from 1-9. According to the man page on lzma it states the following "The first two (-1 and -2) are designed for fast compression speed. -3 .. -9 provide good to excellent compression ratio but require more CPU time and system memory. For relatively fast compression with medium compression ratio -1 is the recommended setting. It's faster than 'bzip2 --fast' and usually creates smaller files than 'bzip2 --best'. -2 makes somewhat smaller files but doubles the compression time close to what 'bzip2 --best' takes." I have been happy with using level -2 since it is close to bzip2 in speed and also creates smaller files. The standard level they are using for the openSUSE rpm's is -2. By default LZMA uses level -7 which does take quite a bit more time to compress, but you certainly will be happy with the size of the file after the compression. Amazing!!

The Bad News: Currently in the Stable releases of openSUSE and SLE tar does not support LZMA.
The Good News: The version of tar in the upcoming releases of openSUSE 11.1 and SLE 11 will support LZMA. I have also heard that the openSUSE build service also supports tar.lzma for factory builds currently.

The bad news is not so bad because there is a way to get around it of course.

Building a .tlz or .tar.lzma compresses archive pre-openSUSE 11.1 / pre-SLE 11:

with tar executing lzma:
1. tar --use-compress-program=/usr/bin/lzma -cvf some_archive_file.tlz some_archive_directory/

without tar executing lzma:
1. tar -cvf - some_files | lzma -2 > some_archive_file.tlz

Note: when you use method 1 with tar executing lzma it defaults to the -7 compression level. If you don't want to use that level then its preferred that you use the method without tar executing the compression program. Notice after the pipe you see the lzma switch with -2, which you can substitute with any number from 1-9 for your compression level.

Happy Compressing...

No comments: