Re: How to read integers from file faster?

[Andy Wingo]

On Sat 31 Aug 2013 05:55, Darren Hoo <address@hidden> writes:

>     (define (read-test1)
>       (with-input-from-file
>           "rnd.txt"
>         (lambda ()
>           (let lp ((i (read)))
>       (if (not (eof-object? i))
>           (lp (read)))))))
>     
> scheme@(guile-user)> ,time (read-test1)
> ;; 37.348000s real time, 37.340000s run time.  0.450000s spent in GC.

What this does is read, character by character, ready to read any
S-expression.  Then when it sees a character that can start a number, it
collects the characters into a string up to the next delimiter, and
tries to parse the string as a number.  If it's not a number it's
treated as a symbol.

>     (import (ice-9 rdelim))
>     (define (read-test2)
>       (with-input-from-file
>           "rnd.txt"
>         (lambda ()
>           (let lp ((i (read-line)))
>       (if (not (eof-object? i))
>           (begin 
>             (string->number i)
>             (lp (read-line))))))))

This is faster because you have chosen the delimiter (newline) and you
are only looking for numbers.

> it only takes 1.8 seconds by using fscanf
>
>     FILE *f = fopen("rnd.txt", "r");
>     if (f == NULL) {
>       printf("open failed!");
>       exit(1);
>     }
>     long long i;
>     while (fscanf(f, "%lld", &i) != EOF) {
>       
>     }

Likewise, but here we don't cons any strings, and you're looking only
for signed integers in a particular range, not any kind of number.

> Are there any primitives in Guile that is equivalent to C's scanf?

No, for better or for worse :)

I just took a look at your program, which ran in 40s on my machine.
Under callgrind it turned out that we were doing a lot of iconv stuff
that we didn't need to do.  With some patches that are now in master,
I'm down to 6 seconds for your test.  I think that with some
optimizations we could get down to 3 seconds or so -- run Guile under
valgrind to see.

Writing a read-integer in Scheme would eventually be faster because it
knows what kinds of characters it's looking for, and really can avoid
garbage.  But for now the VM overhead is too much.  I'm working on that
but it will take longer :)

Andy
-- 
http://wingolog.org/