[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

substantial performance loss when running long-lived or computationally

From: Michael A. Wells
Subject: substantial performance loss when running long-lived or computationally intensive programs
Date: Mon, 05 May 2003 22:36:26 -0500 (CDT)

The Guile interpreter comes to a near halt after making a sufficiently
large number of calls to guile built-ins which do not appropriately
maintain the 'scm_mallocated' variable (declared in "libguile/gc.c").

Although this problem is present in Guile-1.6.4, the problem dates
back to at least Guile 1.4.

There appears to be a widespread misunderstanding as to how
'scm_mallocated' should be maintained.

[Several other people have discussed this issue:
    http://www.glug.org/snap/workbook/gc/memory.text ]

There is no comment in "libguile/gc.c" which describes what
'scm_mallocated' should contain, but by looking through the code, it
appears that 'scm_mallocated' should correspond to the total number of
bytes obtained through malloc currently held by the interpreter. 

Each time a block of memory of size S is allocated, 'scm_mallocated'
should be incremented by S.

Each time a block of memory of size S is garbaged collected and/or
freed, 'scm_mallocated' should be decremented by S.

'scm_mallocated', in combination with 'scm_mtrigger' is used to
determine when the certain calls to the garbage collector (scm_igc)
are triggered.

When 'scm_mallocated' is greater than 'scm_mtrigger', garbage
collection is triggered.  After garbage collection, if the yield is too
small, 'scm_mtrigger' is increased, relative to 'scm_mtrigger'. (See
body of 'check_mtrigger' function.)

[Garbage collections are triggered by functions other than
'scm_mtrigger', which turns out to be a good thing.   If
'check_mtrigger' were the only caller of 'scm_igc', the size of the
guile image could grow very rapidly.]

Unfortunately, in some cases 'scm_mallocated' is incremented when the
memory is allocated, but scm_mallocated' is not decremented when
memory is freed.

As 'scm_mallocated' approaches 2^32, the value of 'scm_mtrigger' may be
set to a value _intended_ to be higher than 'scm_mallocated', but ends
up wrapping around to a value lower than 'scm_mallocated'.
It is also possible that 'scm_mallocated' will wrap around as well.

Once a wraparound occurs, the interpreter comes to a near halt.

I've identified a _partial_ list of code which increments but does not
decrement 'scm_mallocated':
   libguile/filesys.c: getcwd, readline
   libguile/fports.c: fport_close
   libguile/posix.c: gethostname
   libguile/ports.c: scm_remove_from_port_table
   libguile/regex-posix.c: scm_regexp_exec

      These functions allocate memory with 'scm_must_malloc' and
      'scm_must_realloc', then free memory with a call
      'scm_must_free'.  The call to 'scm_must_free' is not made
      by the garbage collector.          

      While 'scm_must_malloc' and 'scm_must_realloc' increment 
      'scm_mallocated',  'scm_must_free' does not decrement

I have attached some scheme code which demonstrates how certain using certain
guile built-ins increment but do not decrement 'scm_mallocated'.

Michael Wells

;---snip here

(use-modules (ice-9 format))

(define (tester how-many-times thunk)
  ; force garbage collection
  ; get initial value of 'scm_mallocated'
  ;   (available using 'bytes_malloced key from call to (gc-stats)
  (let ((initial-bytes-malloced (cdr (assoc 'bytes-malloced (gc-stats))))
        (starting-seconds (tms:clock (times)))) ;(current-time)))       
    ; run 'thunk' 'how-many-times'
    (do ((x 0 (+ x 1)))
        ((= x how-many-times) #t)
      (cond ((= (modulo x 1000) 0)
             (let ((now (tms:clock (times))))
               (display (format "   ~A : ~A clock units~%" x (- now 
               (set! starting-seconds now))))        
    ; force garbage collection
    (let ((final-bytes-malloced (cdr (assoc 'bytes-malloced (gc-stats)))))
       (format "  final-bytes-malloced: ~A~%" final-bytes-malloced))
       (format "initial-bytes-malloced: ~A~%" initial-bytes-malloced))
       (format "       final - initial: ~A~%" (- final-bytes-malloced
; libguile/posix.c -- "gethostname"
(tester 10000 (lambda () (gethostname)))

; Performance suffers when 'scm_mallocated' is very large.
; Note how quickly each 100 calls to '(gethostname)'
; run at first; compare with how long each 100 calls takes 
; near end of run.
; (For me, performance grinds to a halt after the 
; first 14779000 or so iterations of (gethostname).
;(tester 20000000 (lambda () (gethostname)))

; fports.c -- "fport_close"
; ports.c -- "scm_remove_from_port_table"
(tester 1000
        (lambda () 
          (with-output-to-string (lambda () 
                                   (display "1234567890")
                                   (display "1234567890")))))

; libguile/regex-posix.c -- "scm_regexp_exec"
(define *token* (make-regexp "^[A-Za-z0-9_]*" ))

(tester 1000 (lambda () (regexp-exec *token* "foo")))

; libguile/filesys.c "getcwd"
(tester 1000 (lambda () (getcwd)))

reply via email to

[Prev in Thread] Current Thread [Next in Thread]