Re: map-file-lines

[Thierry Volpiatto]

Hi Ted!
Ted Zlatanov <address@hidden> writes:

> On Mon, 02 Feb 2009 11:20:07 -0600 Ted Zlatanov <address@hidden> wrote: 
>
> TZ> Emacs Lisp lacks a good way to iterate over all the lines of a file,
> TZ> especially for a large file.  The following code tries to provide a
> TZ> solution, concentrating on reading a block of data in one shot and then
> TZ> processing it line by line.  It may be more efficient to write this in
> TZ> C.  Also, it does not deal with cases where the first line read is
> TZ> bigger than the buffer size, and may have other bugs, but it works for
> TZ> me so I thought I'd post it for comments and criticism.
>
> Updated:
>
> - line count 0-based now, logic is cleaner
> - buffer size 128K by default
> - accept start line and count
> - abort when the lambda func returns nil
> - renamed endline to line-end for clarity
>
> Thanks
> Ted

Can you try `tve-flines-iterator' that work a little like python
iterators work.
I plan to use it in futures versions of traverselisp if it is faster than
actual code (traverselisp don't use this code actually).

,----[ C-h f tve-flines-iterator RET ]
| tve-flines-iterator is a Lisp function in `traverselisp.el'.
| 
| (tve-flines-iterator file &optional nlines startpos bufsize)
| 
| Return an iterator on `nlines' lines of file.
| `startpos' and `bufsize' are the byte options to give to
| `insert-file-contents'.
| 
| [back]
| 
| ===*===*===*===*===*===*===*===*===*===*===
| Example:
| ,----
| | ;; create an elisp-iterator object that
| | ;; record the first 1024 bytes of my .emacs
| | (setq A (tve-flines-iterator "~/.emacs.el" nil 0 1024))
| | 
| | ;; eval as many times as needed or launch it in a loop
| | (tve-next A)
| `----
`----

You can get it with hg here:
hg clone http://freehg.org/u/thiedlecques/traverselisp/

> (defun map-file-lines (file func &optional startline count bufsize)
>   (let ((filepos 0)
>       (linenum 0)
>       (bufsize (or bufsize (* 128 1024))))
>     (with-temp-buffer
>       (while
>         (let*
>             ((inserted (insert-file-contents
>                         file nil
>                         filepos (+ filepos bufsize) 
>                         t))
>              (numlines (count-lines (point-min) (point-max)))
>              (read (nth 1 inserted))
>              (done (< 1 read))
>              result line-end)
>           (dotimes (n (count-lines (point-min) (point-max)))
>               (goto-char (point-min))
>               (setq line-end (line-end-position)
>                     result (if (and startline (< linenum startline))
>                                ()
>                              (if (and 
>                                   count
>                                   (>= (- linenum startline) count))
>                                  (return)
>                                (funcall func 
>                                         (buffer-substring 
>                                          (line-beginning-position)
>                                          line-end)
>                                         linenum)))
>                     done (and done result))
>               (incf filepos line-end)
>               (forward-line)
>               (incf linenum))
>           done)))
>     linenum))
>
> ;;(map-file-lines "/tmp/test" (lambda (line num) (message "%d: %s" num line)))
> ;;(map-file-lines "/tmp/test" (lambda (line num) (message "%d: %s" num line)) 
> 100)
> ;;(map-file-lines "/tmp/test" (lambda (line num) (message "%d: %s" num line)) 
> 100 10)
>

-- 
A + Thierry Volpiatto
Location: Saint-Cyr-Sur-Mer - France