Re: Rm performance issue

[James Youngman]

On 9/26/07, Ken Naim <address@hidden> wrote:
> As for bad io experiences, our core production system use raw devices for
> our databases so we don't have the same issue(s), this is our production
> reporting system that gets cloned over nightly. So the process removes all
> the existing files and then writes new versions of them from a backup onto a
> file system. I have noticed the poor io performance since I came onsite but
> the unix team keeps saying everything is fine.

They probably just mean that nothing is reporting an error.   Unless
you have a performance SLA agreed with them, that is.   [If you do
have a performance SLA with them you should start by measuring how
many blocks of I/O you are issuing from your host while trying to
delete the files]

> This rm issue is causing the
> database clone process to exceed its allocated downtime window so I thought
> I'd start there.
>
> If anyone can point me to any specific information on tuning ext3 file
> system I'd appreciate it. I am googling it now.

If you are not space limited, you will find it faster in terms of wall
time to move the old files to a staging area (using either mv(1) or
rename(2)) on the same filesystem, launch a file removal process in
the background, and meanwhile create the new files (from the backup
you refer to).  This will probably get you the biggest win (assuming
there are a reasonable number of spindles to spread the I/O load).

Another option here is to issue all the unlink system calls in
parallel.  Again, this assumes that the performance limitation is due
to SAN latency rather than I/O bandwidth.

If these files are the only things on this filesystem, the ext3
journalling is gaining you nothing except a limited fsck time.   You
could try turning off journalling or moving the journal to a different
LUN.  However, if you go down this route you will need to script
something to avoid fsck on startup after unclean shutdown (e.g. by
making a new, empty, filesystem with mkfs).

Alternatively, if your production database supports point-in-time
recovery, you could just  snapshot the LUNs on the production system,
copy the transaction logs to the clone system, attach the snapshot
LUNs to it, switch the snapshots to read-write, perform point-in-time
recovery with the transaction logs, and then use the copy of the
database you have there.

James.