rdiff-backup-users
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [rdiff-backup-users] Memory usage during regressions


From: Claus-Justus Heine
Subject: Re: [rdiff-backup-users] Memory usage during regressions
Date: Sat, 06 Aug 2011 19:41:31 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20110706 Thunderbird/5.0

On 08/06/2011 07:13 PM, Robert Nichols wrote:
On 08/06/2011 08:35 AM, Claus-Justus Heine wrote:
[snip]
Then things start to slow down (swapping). It's a quite large backup
set, about
400G, with a large history. I doesn't seem to be a memory leak, as the
memory
usage stay at 3G. Just seems a little bit too much in principle.

Regression is concerned with only the two most recent sessions, so

Well, yes. So that piece of information was irrelevant, as rdiff-backups keeps the most recent snapshot literally, and the older ones as deltas.

the amount of history should be irrelevant. What is the total
number of files being backed up and the size of the uncompressed
mirror_metadata snapshot?

zcat file_statistics.{latest_timestamp}.data.gz | tr '\0' '\n' | wc -l
zcat mirror_metadata.{latest_timestamp}.snapshot.gz | wc -c

That would be more indicative of the amount of data that needs to be

Quite a bit of data:

address@hidden rdiff-backup-data $\
zcat file_statistics.2011-07-24T20:02:14+02:00.data.gz|\
   tr '\0' '\n' |\
   wc -l
2698977
address@hidden rdiff-backup-data $\
zcat  mirror_metadata.2011-07-30T05\:56\:36+02\:00.snapshot.gz |\
  wc -c
732509974

So roughly 2.5 millon files, metadata is about 700M.

kept in memory during the regression. FWIW, I'm seeing memory usage
of about 480MB during regression of a backup of about 250,000 files,

Well mine is a factor of 10 larger, roughly (good that memory consumption does not scale linearily ... ;)

Still, this seems to be insane. Reading this month's archives (and given that the last release of rdiff-backup was about '09) it seems that I would have to fix that by myself, it seems. Or live with it. Or buy a backup server with more memory ...

Of course, in principle the core-size of rdiff-backup is not a problem, on a decent OS the part of the core currently not in use would just be swapped out. There is only a problem if the program constantly traverses the allocated buffers. This is what I suspect. It takes ages (2 or 3 days) to finally finish the regression.

Thanks for your quick response!

Best,

Claus



reply via email to

[Prev in Thread] Current Thread [Next in Thread]