[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
substantial performance loss when running long-lived or computationally
Michael A. Wells
substantial performance loss when running long-lived or computationally intensive programs
Mon, 05 May 2003 22:36:26 -0500 (CDT)
The Guile interpreter comes to a near halt after making a sufficiently
large number of calls to guile built-ins which do not appropriately
maintain the 'scm_mallocated' variable (declared in "libguile/gc.c").
Although this problem is present in Guile-1.6.4, the problem dates
back to at least Guile 1.4.
There appears to be a widespread misunderstanding as to how
'scm_mallocated' should be maintained.
[Several other people have discussed this issue:
There is no comment in "libguile/gc.c" which describes what
'scm_mallocated' should contain, but by looking through the code, it
appears that 'scm_mallocated' should correspond to the total number of
bytes obtained through malloc currently held by the interpreter.
Each time a block of memory of size S is allocated, 'scm_mallocated'
should be incremented by S.
Each time a block of memory of size S is garbaged collected and/or
freed, 'scm_mallocated' should be decremented by S.
'scm_mallocated', in combination with 'scm_mtrigger' is used to
determine when the certain calls to the garbage collector (scm_igc)
When 'scm_mallocated' is greater than 'scm_mtrigger', garbage
collection is triggered. After garbage collection, if the yield is too
small, 'scm_mtrigger' is increased, relative to 'scm_mtrigger'. (See
body of 'check_mtrigger' function.)
[Garbage collections are triggered by functions other than
'scm_mtrigger', which turns out to be a good thing. If
'check_mtrigger' were the only caller of 'scm_igc', the size of the
guile image could grow very rapidly.]
Unfortunately, in some cases 'scm_mallocated' is incremented when the
memory is allocated, but scm_mallocated' is not decremented when
memory is freed.
As 'scm_mallocated' approaches 2^32, the value of 'scm_mtrigger' may be
set to a value _intended_ to be higher than 'scm_mallocated', but ends
up wrapping around to a value lower than 'scm_mallocated'.
It is also possible that 'scm_mallocated' will wrap around as well.
Once a wraparound occurs, the interpreter comes to a near halt.
I've identified a _partial_ list of code which increments but does not
libguile/filesys.c: getcwd, readline
These functions allocate memory with 'scm_must_malloc' and
'scm_must_realloc', then free memory with a call
'scm_must_free'. The call to 'scm_must_free' is not made
by the garbage collector.
While 'scm_must_malloc' and 'scm_must_realloc' increment
'scm_mallocated', 'scm_must_free' does not decrement
I have attached some scheme code which demonstrates how certain using certain
guile built-ins increment but do not decrement 'scm_mallocated'.
(use-modules (ice-9 format))
(define (tester how-many-times thunk)
; force garbage collection
; get initial value of 'scm_mallocated'
; (available using 'bytes_malloced key from call to (gc-stats)
(let ((initial-bytes-malloced (cdr (assoc 'bytes-malloced (gc-stats))))
(starting-seconds (tms:clock (times)))) ;(current-time)))
; run 'thunk' 'how-many-times'
(do ((x 0 (+ x 1)))
((= x how-many-times) #t)
(cond ((= (modulo x 1000) 0)
(let ((now (tms:clock (times))))
(display (format " ~A : ~A clock units~%" x (- now
(set! starting-seconds now))))
; force garbage collection
(let ((final-bytes-malloced (cdr (assoc 'bytes-malloced (gc-stats)))))
(format " final-bytes-malloced: ~A~%" final-bytes-malloced))
(format "initial-bytes-malloced: ~A~%" initial-bytes-malloced))
(format " final - initial: ~A~%" (- final-bytes-malloced
; libguile/posix.c -- "gethostname"
(tester 10000 (lambda () (gethostname)))
; Performance suffers when 'scm_mallocated' is very large.
; Note how quickly each 100 calls to '(gethostname)'
; run at first; compare with how long each 100 calls takes
; near end of run.
; (For me, performance grinds to a halt after the
; first 14779000 or so iterations of (gethostname).
;(tester 20000000 (lambda () (gethostname)))
; fports.c -- "fport_close"
; ports.c -- "scm_remove_from_port_table"
(with-output-to-string (lambda ()
; libguile/regex-posix.c -- "scm_regexp_exec"
(define *token* (make-regexp "^[A-Za-z0-9_]*" ))
(tester 1000 (lambda () (regexp-exec *token* "foo")))
; libguile/filesys.c "getcwd"
(tester 1000 (lambda () (getcwd)))
- substantial performance loss when running long-lived or computationally intensive programs,
Michael A. Wells <=