[Savannah-hackers-public] vcs: page allocation failure

[Bob Proulx]

As I spend some time looking at the logs I am seeing this type of
error reported from vcs.

  Jul  6 23:39:50 vcs kernel: [10544.316262] apache2: page allocation failure. 
order:4, mode:0x4020
  Jul  6 23:48:09 vcs kernel: [11043.052979] apache2: page allocation failure. 
order:4, mode:0x4020
  Jul  7 03:00:11 vcs kernel: [22565.420474] apache2: page allocation failure. 
order:5, mode:0x4020
  Jul  7 04:16:39 vcs kernel: [27152.541839] apache2: page allocation failure. 
order:5, mode:0x4020
  Jul  7 04:28:06 vcs kernel: [27840.380948] apache2: page allocation failure. 
order:5, mode:0x4020

Then looking further I see that various other processes, not just
apache, report the same problem.

  /var/log/kern.log.5.gz:Sep  7 18:38:02 vcs kernel: [10015736.672531] swapper: 
page allocation failure. order:5, mode:0x4020
  /var/log/kern.log.5.gz:Sep  8 17:04:05 vcs kernel: [10096499.787091] 
sv_get_authoriz: page allocation failure. order:5, mode:0x4020
  /var/log/kern.log.7.gz:Apr  8 19:59:20 vcs kernel: [8909592.237962] cvs: page 
allocation failure. order:5, mode:0x4020
  /var/log/kern.log.7.gz:Apr 11 20:52:28 vcs kernel: [9171979.662807] cvs: page 
allocation failure. order:5, mode:0x4020

Researching that finds a kernel issue discussed here:

  
http://serverfault.com/questions/236170/page-allocation-failure-am-i-running-out-of-memory

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=666021

Following the discussion I am going to try the following mitigation:

  sysctl -w vm.min_free_kbytes=65536

The default is 3420 on vcs.  On newer Linux kernels the default value
has been increased to 67584.  When we eventually upgrade this explicit
increase should not be needed anymore.  If reserving more pages
resolves the problem I will make it permanent in vm.conf which is
currently used for disabling the OOM killer.  I will add this config
to it if this works to eliminate the page allocation failures.

  /etc/sysctl.d/vm.conf

Bob

I reproduce the important parts of the above here for reference:

  
http://serverfault.com/questions/236170/page-allocation-failure-am-i-running-out-of-memory

  Debian bug 666021 seems to be a report of this same issue. The
  suggestion there is:

  #change value for this boot
  sysctl -w vm.min_free_kbytes=65536

  #change value for subsequent boots
  echo "vm.min_free_kbytes=65536" >> /etc/sysctl.conf

  http://russ.garrett.co.uk/2009/01/01/linux-kernel-tuning/ has some
  discussion of when altering this setting may be useful, reproduced
  here:

    This tells the kernel to try and keep 64MB of RAM free at all
    times. It’s useful in two main cases:

    Swap-less machines, where you don’t want incoming network traffic to
    overwhelm the kernel and force an OOM before it has time to flush any
    buffers.  x86 machines, for the same reason: the x86 architecture only
    allows DMA transfers below approximately 900MB of RAM. So you can end
    up with the bizarre situation of an OOM error with tons of RAM free.