RE: Problem with NSAutoreleasePool in GNUstep

[JP Young]

I not be fully informed on the original issue, but I think the source of
this question has to do with an application (WarmStartServer) that will be
encoding a very large list of elements to transfer to another process in one
large message. Let's say that the size of the element list we are encoding
is about 50M in memory and we call NSArchiver to encode this. At our
application level, we don't have a loop to put a pool within (though one of
some form certainly exists within NSArchiver to walk and encode the list).
We are just making a single call with a root object that will cause a lot of
encoding to happen and generate temporary memory usage that greatly exceeds
the original size of what we are encoding. Putting a pool around the
encodeRootObject: call will be closing the gate after the memory has been
allocated :-)

I think our choices to work around this problem are:

1. Try putting autorelease pools in the encodeWithCoder: methods of our
objects that are being encoded within the list and see if this solves the
problem. This should work if the majority of the memory allocation is
happening in these methods.

2. Break up the list into smaller portions to encode and send. This may be
complicated by the fact that we sometimes want to perform this same encoding
and write it to disk instead of send it to another process. It would be much
cleaner, in both cases, to just encode one big blob.

3. Go into the NSArchiver code (or write categories for it) that will use
autorelease pools during the execution of its existing logic.

I certainly agree with everyone that has said this is not a problem with
AutoreleasePools. If there is any problem (though this is certainly not
demonstrated out by the example code), it is with NSArchiver and its
behavior with encoding large sets of objects.

Richard's and Wim's explanation of the way that memory allocation works is
spot-on. Once we hit that peak size, we will never grow anymore, unless we
have to encode an even bigger object set. The peak size itself is the issue,
and we can't easily rearrange memory to let go of all of the huge "dead
spots" that get created when we do this operation.

Ramana, please reply, especially if I am wrong about what triggered this
question.

                -- J. P.

-----Original Message-----
From: Nicola Pero [mailto:n.pero@mi.flashnet.it]
Sent: Monday, December 03, 2001 8:14 AM
To: ramana rao
Cc: Richard Frith-Macdonald; discuss GNUstep; Vijaya Bhaskar Reddy K;
Ravindra K S; Kotesh M V; Krishna Kumar; jpyoung
Subject: Re: Problem with NSAutoreleasePool in GNUstep



> Richard,
>
> thanks for the response.
>
> We tried with +freecache, still the memory used by the program is not
going
> down.
>
> I still could not get why memory usage is not coming down after [pool
> release]. In OpenStep the memory usage does come down?
>
> This problem is not with particular machine, it is coming on all machines

I tried your code ... hacked it to run on my debian gnu/linux box - and I
get -

nicola@didone:~/hh/A$ ./shared_obj/ix86/linux-gnu/gnu-gnu-gnu/memory

Dic 03 15:02:03 memory[2337] Memory Usage at the beginning: VmSize:
3928 kB

Dic 03 15:02:03 memory[2337] Memory Usage before allocation: VmSize:
4496 kB

Dic 03 15:02:05 memory[2337] Memory Usage after allocation: VmSize:
29604 kB

Dic 03 15:02:05 memory[2337] Memory Usage after removal: VmSize:
29604 kB

Dic 03 15:02:05 memory[2337] Memory Usage after release of the pool: VmSize:
8872 kB

Dic 03 15:02:05 memory[2337] Memory Usage after freeCache: VmSize:
4492 kB

so I don't see the effect you see ... memory is returned on my system.


Anyway - the suggestion you got from other people is correct - if your
application reaches a peak memory usage of 1 Gb or whatever - did I hear
correctly ?, are you sure you can't rework your application so that the
peak memory usage is less than that ?  It's going to be healthy even not
considering this question.  I think you really need to cut that peak down
if you can.

Typically, if you have a loop, you need to recreate the autorelease pool
periodically, as in

/* -*-objc-*- */

#include <Foundation/Foundation.h>
#include "OIProcessStatistics.h"

void printMemory (NSString *when)
{
  NSLog (@"Memory Usage %@: %@", when,
         [OIProcessStatistics getCurrentProcessMemoryUsage]);
}


int main (void)
{
  NSAutoreleasePool * pool = [NSAutoreleasePool new];

  printMemory (@"at the beginning");

  {
    NSAutoreleasePool * poolInner = [NSAutoreleasePool new];
    NSMutableArray *theArr = [NSMutableArray new];
    int i, j;

    printMemory (@"before allocation");

    for (i = 0, j =0; i < 100000; i++, j++)
      {
        if (j = 10000)
          {
            DESTROY (poolInner);
            poolInner = [NSAutoreleasePool new];
            j = 0;
          }
        [theArr addObject:
                  [NSString stringWithFormat:
@"Helllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllooooooo
oooooooooooooooooooooooooooooo %d",
                                           i]];
      }

    printMemory (@"after allocation");

    [theArr removeAllObjects];

    printMemory (@"after removal");

    RELEASE (theArr);
    DESTROY (poolInner);

    printMemory (@"after release of the pool");

    [NSAutoreleasePool freeCache];

    printMemory (@"after freeCache");
  }

  DESTROY (pool);
  return 0;
}

this example is not particularly meaningful because you are not creating
many temporary objects inside the loop _ in a real world application you
would probably do something inside the loop, generate a lot of temporary
objects, and releasing the pools has a greater effect on memory
performance.  I'm not sure how much obvious is what I've been saying :-)
but I thought I would give a practical example code in case you felt the
suggestion of recreating autorelease pools more often was too vague.