[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: More (de)compress?

From: Dmitry Antipov
Subject: Re: More (de)compress?
Date: Tue, 20 Aug 2013 12:19:35 +0400
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8

On 08/19/2013 08:41 PM, Paul Eggert wrote:

* It can be faster to compress using an external program,
   since the compression can be done in parallel.  Have you
   timed your compression approach on a multicore platform,
   and compared its real time to doing it with external
   compression?  (Similarly for decompression, though I
   expect there we won't find the external program faster.)
   You might try "pigz" for compression, since it's multicore

It's faster because the buffer machinery is slower than external
compression program's input reader (at least, in case of gzip).
I tried to compress 959 small text files (~16Mb in total) with
'gzip *.txt' (0.67s), dired-compress-file (6.15s, but don't forget
about fork+exec overhead) and simple ad-hoc function using
compress-region and zlib method:

(defun compress-file (name method)
  (message "Compress %s" name)
  (let ((ext (cdr (assoc method '((zlib . "gz") (bzlib . "bz2") (lzma . 
        (buffer (find-file-literally name)))
    (when (null ext) (error "Unsupported compression method '%S'" method))
      (set-buffer buffer)
      (compress-region method)
      (delete-file (buffer-file-name))
      (rename-buffer (concat (buffer-name) "." ext))
      (write-file (concat (buffer-file-name) "." ext))

The latter version deliberately takes ~19s. Unfortunately internal
compression support can't replace calls to external programs, especially
in batch operations where we need to (de)compress multiple files at once.
But internal compression should have some advantages when we just need
to show the contents of compressed buffer (I didn't try to check this
yet, BTW).

* There seems to be quite a bit of repetition in configure.ac
   and in the C code -- each compression package does pretty
   much the same thing with respect to allocating buffers,
   saving point, etc.  Could this be factored out to simplify
   the code and make it easier to add future compression


* bzlib_detect and lzm_detect mishandle the case where the
   buffer gap is located very near the start of the buffer.

Argh, yes.

* If the buffer contains random garbage,
   (decompress-region nil 1 100000)
   signals "Unsupported decompression method", which
   isn't very clear.  It should signal something like
   "Unknown compression format".

* The functions compress-region and decompress-region
   should be defined on all platforms, even those that
   lack all compression libraries.  They'll simply return
   nil on such platforms, since they can't compress or
   decompress anything.  This simplifies the C code and
   will simplify Lisp code too.



reply via email to

[Prev in Thread] Current Thread [Next in Thread]