Very strange slowdown in memset() when running in gdb

[Andrew Athan]

All:

We have thus far been unable to reproduce the following application
behavior when running outside of gdb running inside emacs.  Therefore, I
am starting with this list for suggestions on how to proceed.



The high level description of the problem is this:  At times, when
running an application that (a) opens many sockets (b) receives
relatively high rates of traffic on those sockets (c) eventually has
many threads running [50+] and (d) mallocs several large blocks of
memory, some up to 500M or 1G ... the application will "hang" for long
periods inside memset(0) of one of those memory blocks.  It is not clear
that (a-c) are relevant, since the behavior is often exhibited in an
initialization thread ahead of starting the sockets.

The slow memset() happens most or more often when running gdb inside
emacs.  C-c C-c will take 30sec+ and sometimes up to several minutes to
return to the (gdb) prompt, and the machine will be generally slow for
some period of time during/after this.  We do not see evidence of
virtual memory paging but we are not certain we are looking in all the
right places -- hints appreciated.

The problem occurs both within stock gdb and within a gdb patched to
with this patch: 
http://sourceware.org/ml/gdb-patches/2010-04/msg00466.html.  The version
strings are respectively:

GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
GNU gdb (GDB) 7.2

Invariably, if breaking via C-c C-c we end up with the program counter
*inside* memset, which causes me to suspect that some kind of overactive
page fault situation is occurring, and that this is drastically slowing
the machine.

We have seen references to problems with memset when crossing 2G
boundaries, but this is a 64bit box with 33G of ram, so I would have
thought this is not a problem.  Further, the reference seems to be to
introduce pointers mapped to userland from a driver, which is not the
case here: 
http://lists.kernelnewbies.org/pipermail/kernelnewbies/2011-February/000760.html

Another reference, which implicates MCFG ACPI table problems
(http://lists.us.dell.com/pipermail/linux-precision/2011-February/001503.html
and https://bugzilla.redhat.com/show_bug.cgi?id=581933) also seems
unrelated, as booting with pci=nommconf does not help.

I am addressing gdb to start with because this intermittent problem does
not seem to occur outside of gdb.  My thought is that when running
inside both emacs and gdb we end up mapped into an area of physical
memory that exhibits a problem at the kernel level.  However, I am not
entirely sure whether gdb could be receiving any signals or be otherwise
interposed in the memory allocation and subsequent walk of those bytes
by memset().

I can say with a low degree of confidence that the problem occurs more
frequently when the system has higher incoming network load, though
there is no chance that all CPUs are pegged or that "lots" of active
threads are running by the time the slow call is made.  In fact, often,
the memset() occurs in an initialization thread that starts before other
threads.  This, gdb, emacs and bash are always the *only* running
processes besides stock ubuntu processes -- i.e., there is not other
work going on that is starving the CPUs.  Finally, we have disabled the
"ondemand" functionality, so all cores are running at full speed.

Another reason to suspect gdb is that we have run fairly thorough memory
tests (both "bios level" and memtester within linux) on the machine,
including writing an application that simply allocates huge chunks of
memory and memsets them.  The large memset application was run inside
gdb and inside emacs and did not exhibit the behavior.  Could this be
some kind of code offset/alignment or symbol lookup problem exhibited
only by the problem executable when loaded by gdb???

Also, this does not appear to be a problem with emacs' prompt
parsing/gud-mode/etc because the application definitely does not proceed
beyond the memset for a long period and breaking puts us within memset. 
Under these conditions it is sometimes necessary to C-z out of emacs,
and kill -9 both gdb and the application.

I am somewhat at a loss on how to debug this and do not have the
resources to run into too many dead ends.  Therefore, can anyone suggest
whether whole-system profiling such as oprofile would help catch
kmap/kunmap or other kernel / virtual-memory badness ?  Would running
gdb inside gdb be fruitful, and if so can anyone point me to functions
or areas I would look to breakpoint or otherwise monitor under such a setup?

I am unable to immediately reproduce the problem in order to provide a
stack trace of the slow thread at this moment, but will follow up when I
can.  Details of the hardware and OS below.

Thanks in advance,
A.A.



Ubuntu system with 48 AMD cores

uname -a:  Linux  2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC
2011 x86_64 x86_64 x86_64 GNU/Linux

Last /proc/cpuinfo stanza:

processor    : 47
vendor_id    : AuthenticAMD
cpu family    : 16
model        : 9
model name    : AMD Opteron(tm) Processor 6180 SE
stepping    : 1
cpu MHz        : 2500.000
cache size    : 512 KB
physical id    : 3
siblings    : 12
core id        : 5
cpu cores    : 12
apicid        : 75
initial apicid    : 59
fpu        : yes
fpu_exception    : yes
cpuid level    : 5
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc
extd_apicid amd_dcm pni monitor cx16 popcnt lahf_lm cmp_legacy svm
extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit
wdt nodeid_msr npt lbrv svm_lock nrip_save pausefilter
bogomips    : 5000.18
TLB size    : 1024 4K pages
clflush size    : 64
cache_alignment    : 64
address sizes    : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate



MemTotal:       33008392 kB
MemFree:        28624044 kB
Buffers:           59708 kB
Cached:          1302196 kB
SwapCached:         1764 kB
Active:          2700812 kB
Inactive:         369772 kB
Active(anon):    1673156 kB
Inactive(anon):    35572 kB
Active(file):    1027656 kB
Inactive(file):   334200 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      14865404 kB
SwapFree:       14861092 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:       1707772 kB
Mapped:            59296 kB
Shmem:                28 kB
Slab:             666848 kB
SReclaimable:      85424 kB
SUnreclaim:       581424 kB
KernelStack:        3128 kB
PageTables:        10576 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    31369600 kB
Committed_AS:    4772588 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      122392 kB
VmallocChunk:   34328955388 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       65920 kB
DirectMap2M:     9369600 kB
DirectMap1G:    24117248 kB