> Dear Ralf,
>
> I am posting your requested output of commands but it is working right now.
> Do you need these when openvpn stuck ?
>
> address@hidden ~]# ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 40960
> max locked memory (kbytes, -l) 32
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 10240
> cpu time (seconds, -t) unlimited
> max user processes (-u) 40960
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> address@hidden ~]# netstat -npl | grep openvpn
> tcp 0 0
188.122.86.195:7505 0.0.0.0:*
> LISTEN 3405/openvpn
> tcp 0 0
188.122.86.195:7506 0.0.0.0:*
> LISTEN 20349/openvpn-ssl
> address@hidden ~]# cat /proc/3405/status
> Name: openvpn
> State: R (running)
> SleepAVG: 98%
> Tgid: 3405
> Pid: 3405
> PPid: 1
> TracerPid: 0
> Uid: 99 99 99 99
> Gid: 99 99 99 99
> FDSize: 64
> Groups: 99
> VmPeak: 103476 kB
> VmSize: 101688 kB
> VmLck: 0 kB
> VmHWM: 18752 kB
> VmRSS: 17548 kB
> VmData: 25768 kB
> VmStk: 88 kB
> VmExe: 512 kB
> VmLib: 9352 kB
> VmPTE: 316 kB
> StaBrk: 01b99000 kB
> Brk: 029fc000 kB
> StaStk: 7fff0a085ca0 kB
> Threads: 2
> SigQ: 0/40960
> SigPnd: 0000000000000000
> ShdPnd: 0000000000000000
> SigBlk: 0000000000000000
> SigIgn: 0000000000001000
> SigCgt: 0000000180004a03
> CapInh: 0000000000000000
> CapPrm: 0000000000000000
> CapEff: 0000000000000000
> Cpus_allowed:
> 00000000,00000000,00000000,00000000,00000000,00000000,00000000,0000000f
> Mems_allowed: 00000000,00000001
> address@hidden ~]# cat /proc/20349/status
> Name: openvpn-ssl
> State: S (sleeping)
> SleepAVG: 98%
> Tgid: 20349
> Pid: 20349
> PPid: 1
> TracerPid: 0
> Uid: 99 99 99 99
> Gid: 99 99 99 99
> FDSize: 64
> Groups: 99
> VmPeak: 88268 kB
> VmSize: 88008 kB
> VmLck: 0 kB
> VmHWM: 3880 kB
> VmRSS: 3492 kB
> VmData: 12088 kB
> VmStk: 88 kB
> VmExe: 512 kB
> VmLib: 9352 kB
> VmPTE: 192 kB
> StaBrk: 1ad42000 kB
> Brk: 1aecc000 kB
> StaStk: 7fff01807f60 kB
> Threads: 2
> SigQ: 0/40960
> SigPnd: 0000000000000000
> ShdPnd: 0000000000000000
> SigBlk: 0000000000000000
> SigIgn: 0000000000001000
> SigCgt: 0000000180004a03
> CapInh: 0000000000000000
> CapPrm: 0000000000000000
> CapEff: 0000000000000000
> Cpus_allowed:
> 00000000,00000000,00000000,00000000,00000000,00000000,00000000,0000000f
> Mems_allowed: 00000000,00000001
> address@hidden ~]#
>
> Best Regards.
>
> On Wed, Nov 3, 2010 at 12:14 PM, Ralf Lübben <
address@hidden> wrote:
> > Hi,
> >
> > very strange why the std::bad_alloc exception is thrown.
> >
> > Can you also check the process limitation with "ulimit -a" and the
> > process memory usage, e.g. with "cat /proc/<PIDs>/status", where PIDs
> > are the OpenVPN
> > processes.
> > Maybe one of the processes allocates to much memory.
> >
> > Or is it possible that there is peak memory usage on your which exceeds
> > the free memory? But about 5GB free memory seems a lot.
> >
> > I will try to catch all std::bad_alloc exceptions in the plugin, so that
> > the
> > error maybe don't cause a crash, or crashes with an appropriate error
> > message. But this will take some days.
> >
> >
> > Regards,
> > Ralf
> >
> > Am Mittwoch, 3. November 2010, um 05:08:24 schrieb Farrukh Ahmed:
> > > Dear Ralf,
> > >
> > > I have seen that there is no memory limitation.
> > >
> > > address@hidden ~]# free -m
> > >
> > > total used free shared buffers
> > > cached
> > >
> > > Mem: 3941 1547 2394 0 361
> > > 915 -/+ buffers/cache: 269 3671
> > > Swap: 2047 0 2047
> > > address@hidden ~]#
> > >
> > > Best Regards.
> > >
> > > On Wed, Nov 3, 2010 at 1:25 AM, Ralf Lübben <
address@hidden> wrote:
> > > > Hi,
> > > >
> > > > the most strange message is
> > > >
> > > > terminate called after throwing an instance of 'std::bad_alloc'
> > > > what(): St9bad_alloc
> > > >
> > > > the other messages should not cause a crash.
> > > >
> > > > Is it possible that no memory can be allocated (system memory,
> > > > process memory
> > > > limitation)? Memory is allocated for example when a user connects.
> > > > Can you check the memory for the system/process?
> > > >
> > > > Also I could try to catch all exceptions if memory if allocated, that
> > > > would identify where error occurs and it could be handled.
> > > >
> > > > Regards,
> > > > Ralf
> > > >
> > > > Am Dienstag, 2. November 2010, um 18:43:27 schrieb Farrukh Ahmed:
> > > > > Dear Ralf,
> > > > >
> > > > > Today again this got crashed and found these lines in logs.
> > > > >
> > > > > address@hidden ~]# pstack 6419
> > > > > #0 0x0000003f3f8c6240 in __read_nocancel () from /lib64/libc.so.6
> > > > > #1 0x00002b0a8360723f in IpcSocket::recvInt() () from
> > > > > /etc/openvpn/radiusplugin.so
> > > > > #2 0x00002b0a83610061 in
> > > > > AuthenticationProcess::Authentication(PluginContext*) () from
> > > > > /etc/openvpn/radiusplugin.so
> > > > > #3 0x00002b0a8360e5ec in openvpn_plugin_open_v2 () from
> > > > > /etc/openvpn/radiusplugin.so
> > > > > #4 0x0000000000441971 in ASN1_BIT_STRING_get_bit ()
> > > > > #5 0x0000000000415477 in ASN1_BIT_STRING_get_bit ()
> > > > > #6 0x000000000041834d in ASN1_BIT_STRING_get_bit ()
> > > > > #7 0x0000000000418b9d in ASN1_BIT_STRING_get_bit ()
> > > > > #8 0x0000000000426e07 in ASN1_BIT_STRING_get_bit ()
> > > > > #9 0x000000000042f203 in ASN1_BIT_STRING_get_bit ()
> > > > > #10 0x0000003f3f81d994 in __libc_start_main () from
> > > > > /lib64/libc.so.6 #11 0x00000000004064d9 in ASN1_BIT_STRING_get_bit
> > > > > ()
> > > > > #12 0x00007fff59bc06e8 in ?? ()
> > > > > #13 0x0000000000000000 in ?? ()
> > > > > address@hidden ~]# pstack 861
> > > > > Thread 2 (Thread 0x429f5940 (LWP 867)):
> > > > > #0 0x0000003f4000aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > > > /lib64/libpthread.so.0
> > > > > #1 0x00002b0445501fc5 in auth_user_pass_verify(void*) () from
> > > > > /etc/openvpn-ssl/radiusplugin.so
> > > > > #2 0x0000003f4000673d in start_thread () from
> > > > > /lib64/libpthread.so.0 #3 0x0000003f3f8d3f6d in clone () from
> > > > > /lib64/libc.so.6
> > > > > Thread 1 (Thread 0x2b04454ce3c0 (LWP 861)):
> > > > > #0 0x0000003f3f8d4358 in epoll_wait () from /lib64/libc.so.6
> > > > > #1 0x000000000040e0b1 in ASN1_BIT_STRING_get_bit ()
> > > > > #2 0x0000000000425f8a in ASN1_BIT_STRING_get_bit ()
> > > > > #3 0x000000000042f203 in ASN1_BIT_STRING_get_bit ()
> > > > > #4 0x0000003f3f81d994 in __libc_start_main () from /lib64/libc.so.6
> > > > > #5 0x00000000004064d9 in ASN1_BIT_STRING_get_bit ()
> > > > > #6 0x00007fff93e01b08 in ?? ()
> > > > > #7 0x0000000000000000 in ?? ()
> > > > > address@hidden ~]# pstack 855
> > > > > #0 0x0000003f3f8c6240 in __read_nocancel () from /lib64/libc.so.6
> > > > > #1 0x00002b0445500e6b in IpcSocket::recvInt() () from
> > > > > /etc/openvpn-ssl/radiusplugin.so
> > > > > #2 0x00002b044550ac71 in
> > > > > AuthenticationProcess::Authentication(PluginContext*) () from
> > > > > /etc/openvpn-ssl/radiusplugin.so
> > > > > #3 0x00002b04455091fa in openvpn_plugin_open_v2 () from
> > > > > /etc/openvpn-ssl/radiusplugin.so
> > > > > #4 0x0000000000441971 in ASN1_BIT_STRING_get_bit ()
> > > > > #5 0x0000000000415477 in ASN1_BIT_STRING_get_bit ()
> > > > > #6 0x000000000041834d in ASN1_BIT_STRING_get_bit ()
> > > > > #7 0x0000000000418b9d in ASN1_BIT_STRING_get_bit ()
> > > > > #8 0x0000000000425d8c in ASN1_BIT_STRING_get_bit ()
> > > > > #9 0x000000000042f203 in ASN1_BIT_STRING_get_bit ()
> > > > > #10 0x0000003f3f81d994 in __libc_start_main () from
> > > > > /lib64/libc.so.6 #11 0x00000000004064d9 in ASN1_BIT_STRING_get_bit
> > > > > ()
> > > > > #12 0x00007fff93e01b08 in ?? ()
> > > > > #13 0x0000000000000000 in ?? ()
> > > > > address@hidden ~]#
> > > > >
> > > > >
> > > > > Tue Nov 2 16:56:34 2010 RADIUS-PLUGIN: BACKGROUND ACCT: No
> > > > > accounting
> > > >
> > > > data
> > > >
> > > > > was found for
vpn.change-mon-ip.com,
92.141.115.34:60981.
> > > > > terminate called after throwing an instance of 'std::bad_alloc'
> > > > > what(): St9bad_alloc
> > > > > Tue Nov 2 16:57:30 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 16:59:47 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 16:59:51 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 16:59:54 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 17:00:00 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 17:00:18 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 17:00:21 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 17:00:25 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 17:01:39 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 17:01:41 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 17:01:42 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 17:01:45 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 17:01:49 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 17:01:51 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > > .Tue Nov 2 17:02:06 2010 RADIUS-PLUGIN: Client config file was not
> > > >
> > > > written,
> > > >
> > > > > overwriteccfiles is false
> > > > >
> > > > > Best Regards.
> > > > >
> > > > > On Mon, Nov 1, 2010 at 1:57 AM, Ralf Lübben <
address@hidden>
> >
> > wrote:
> > > > > > Hi,
> > > > > >
> > > > > > it seems that the interprocess communication between the plugin
> > > >
> > > > processes
> > > >
> > > > > > causes the crash. From the past messages it seems that the
> > > > > > process crashes after a read or write to internal sockets.
> > > > > >
> > > > > > So far it's clear to me why.
> > > > > >
> > > > > > If you increase the "verb" option to 7, the plugin will create
> > > > > > some
> > > >
> > > > debug
> > > >
> > > > > > output. I think it is also forwarded to syslog. Can you try to
> >
> > send
> >
> > > > the
> > > >
> > > > > > output to me? The plugin output has always a "RADIUS-PLUGIN" as
> > > > > > prefix.
> > > > > >
> > > > > > Regards
> > > > > > Ralf
> > > > > >
> > > > > > Am Sonntag, 31. Oktober 2010, um 19:09:05 schrieben Sie:
> > > > > > > Dear Ralf,
> > > > > > >
> > > > > > > Same thing happen with Radius Plugin Version 2.1
> > > > > > >
> > > > > > > #0 0x0000003f3f8c6240 in __read_nocancel () from
> >
> > /lib64/libc.so.6
> >
> > > > > > > #1 0x00002ae6c5cbee6b in IpcSocket::recvInt() () from
> > > > > > > /etc/openvpn-ssl/radiusplugin.so
> > > > > > > #2 0x00002ae6c5cc8c71 in
> > > > > > > AuthenticationProcess::Authentication(PluginContext*) () from
> > > > > > > /etc/openvpn-ssl/radiusplugin.so
> > > > > > > #3 0x00002ae6c5cc71fa in openvpn_plugin_open_v2 () from
> > > > > > > /etc/openvpn-ssl/radiusplugin.so
> > > > > > > #4 0x0000000000441971 in ASN1_BIT_STRING_get_bit ()
> > > > > > > #5 0x0000000000415477 in ASN1_BIT_STRING_get_bit ()
> > > > > > > #6 0x000000000041834d in ASN1_BIT_STRING_get_bit ()
> > > > > > > #7 0x0000000000418b9d in ASN1_BIT_STRING_get_bit ()
> > > > > > > #8 0x0000000000425d8c in ASN1_BIT_STRING_get_bit ()
> > > > > > > #9 0x000000000042f203 in ASN1_BIT_STRING_get_bit ()
> > > > > > > #10 0x0000003f3f81d994 in __libc_start_main () from
> > > > > > > /lib64/libc.so.6 #11 0x00000000004064d9 in
> >
> > ASN1_BIT_STRING_get_bit
> >
> > > > > > > ()
> > > > > > > #12 0x00007fff7207a588 in ?? ()
> > > > > > > #13 0x0000000000000000 in ?? ()
> > > > > > >
> > > > > > > Best Regards.
> > > > > > >
> > > > > > > On Sun, Oct 31, 2010 at 6:17 PM, Farrukh Ahmed <
> >
> >
address@hidden>
> >
> > > > > > wrote:
> > > > > > > > Dear Ralf,
> > > > > > > >
> > > > > > > > I am using syslog logging for OpenVPN and which does not
> > > > > > > > logged Radius Plugin Logs. Today i found that we again got
> > > > > > > > crashed.
> > > > > > > >
> > > > > > > > FYI,
> > > > > > > >
> > > > > > > > #0 0x0000003f3f8c6240 in __read_nocancel () from
> > > > > > > > /lib64/libc.so.6 #1 0x00002b27a7ebce6b in
> > > > > > > > std::_Rb_tree<std::basic_string<char, std::char_traits<char>,
> > > > > > > > std::allocator<char> >,
> > > > > > > > std::pair<std::basic_string<char, std::char_traits<char>,
> > > > > > > > std::allocator<char> > const, UserAcct>,
> > > > > > > > std::_Select1st<std::pair<std::basic_string<char,
> > > > > >
> > > > > > std::char_traits<char>,
> > > > > >
> > > > > > > > std::allocator<char> > const, UserAcct> >,
> > > > > > > > std::less<std::basic_string<char, std::char_traits<char>,
> > > > > > > > std::allocator<char> > >,
> > > > > > > > std::allocator<std::pair<std::basic_string<char,
> > > > > >
> > > > > > std::char_traits<char>,
> > > > > >
> > > > > > > > std::allocator<char> > const, UserAcct> >
> > > > > > > >
> > > > > > > > >::_M_insert(std::_Rb_tree_node_base*,
> >
> > std::_Rb_tree_node_base*,
> >
> > > > > > > > std::pair<std::basic_string<char, std::char_traits<char>,
> > > > > > > > std::allocator<char> > const, UserAcct> const&) () from
> > > > > > > > /etc/openvpn/radiusplugin.so
> > > > > > > > #2 0x00002b27a7ec6c71 in
> > > > > > > > AuthenticationProcess::Authentication(PluginContext*) () from
> > > > > > > > /etc/openvpn/radiusplugin.so
> > > > > > > > #3 0x00002b27a7ec51fa in User::User(User const&) () from
> > > > > > > > /etc/openvpn/radiusplugin.so
> > > > > > > > #4 0x0000000000441971 in ASN1_BIT_STRING_get_bit ()
> > > > > > > > #5 0x0000000000415477 in ASN1_BIT_STRING_get_bit ()
> > > > > > > > #6 0x000000000041834d in ASN1_BIT_STRING_get_bit ()
> > > > > > > > #7 0x0000000000418b9d in ASN1_BIT_STRING_get_bit ()
> > > > > > > > #8 0x0000000000426e07 in ASN1_BIT_STRING_get_bit ()
> > > > > > > > #9 0x000000000042f203 in ASN1_BIT_STRING_get_bit ()
> > > > > > > > #10 0x0000003f3f81d994 in __libc_start_main () from
> > > >
> > > > /lib64/libc.so.6
> > > >
> > > > > > > > #11 0x00000000004064d9 in ASN1_BIT_STRING_get_bit ()
> > > > > > > > #12 0x00007fff46ebb6d8 in ?? ()
> > > > > > > > #13 0x0000000000000000 in ?? ()
> > > > > > > >
> > > > > > > > I will try 2.1 and let you know if there will be no issue.
> > > > > > > >
> > > > > > > > Best Regards.
> > > > > > > >
> > > > > > > > On Sun, Oct 31, 2010 at 2:37 PM, Ralf Lübben <
> >
> >
address@hidden>
> >
> > > > > > wrote:
> > > > > > > >> Hello,
> > > > > > > >>
> > > > > > > >> can you also try the version 2.1 from
> > > > > > > >>
http://www.nongnu.org/radiusplugin/?
> > > > > > > >>
> > > > > > > >> The new beta mainly includes a "accounting only feature" and
> > > > > > > >> "non
> > > > > >
> > > > > > fatal
> > > > > >
> > > > > > > >> accouting" and bugfixes related to BSD.
> > > > > > > >>
> > > > > > > >> Some changes are in the function AccoutingProcess which
> > > > > > > >> seems
> >
> > to
> >
> > > > > > crash,
> > > > > >
> > > > > > > >> to find
> > > > > > > >> the code where the plugin crashes I would need the end (~100
> > > >
> > > > lines)
> > > >
> > > > > > > >> of the OpenVPN log file, maybe it gives a hint.
> > > > > > > >>
> > > > > > > >> Can you identify the event when the plugin crashes, e.g. a
> >
> > user
> >
> > > > > > > >> connects, disconnects, etc?
> > > > > > > >>
> > > > > > > >> Regards,
> > > > > > > >> Ralf
> > > > > > > >>
> > > > > > > >> Am Samstag, 30. Oktober 2010, um 14:03:49 schrieb Farrukh
> >
> > Ahmed:
> > > > > > > >> > Dear Ralf,
> > > > > > > >> >
> > > > > > > >> > can you provide me some more information:
> > > > > > > >> > > - Which version of the plugin do you use?
> > > > > > > >> >
> > > > > > > >> > I have used this v2.1a_beta1 and from CVS head.
> > > > > > > >> >
> > > > > > > >> > > - Which operating system? Is it 64 bit os?
> > > > > > > >> >
> > > > > > > >> > Yes, its CentOS 64 bit os.
> > > > > > > >> >
> > > > > > > >> > > - The plugin configuration.
> > > > > > > >> >
> > > > > > > >> > NAS-Identifier=OpenVPN-NL-01
> > > > > > > >> > Service-Type=5
> > > > > > > >> > Framed-Protocol=1
> > > > > > > >> > NAS-Port-Type=5
> > > > > > > >> > NAS-IP-Address=83.170.87.238
> > > > > > > >> > OpenVPNConfig=/etc/openvpn/server.conf
> > > > > > > >> > subnet=255.255.255.0
> > > > > > > >> > overwriteccfiles=false
> > > > > > > >> >
> > > > > > > >> > server
> > > > > > > >> > {
> > > > > > > >> >
> > > > > > > >> > acctport=1813
> > > > > > > >> > authport=1812
> > > > > > > >> > name=83.170.87.238
> > > > > > > >> > retry=1
> > > > > > > >> > wait=1
> > > > > > > >> > sharedsecret=testing123
> > > > > > > >> >
> > > > > > > >> > }
> > > > > > > >> >
> > > > > > > >> > > - And can you also increase the verbosity in OpenVPN to
> > > > > > > >> > > 7 and send me
> > > > > > > >>
> > > > > > > >> the
> > > > > > > >>
> > > > > > > >> > > OpenVPN log file? The plugin will write some debug
> > > > > > > >> > > information to it.
> > > > > > > >> >
> > > > > > > >> > I have seen log file its very huge i can not send it to
> > > > > > > >> > you
> > > >
> > > > right
> > > >
> > > > > > now.
> > > > > >
> > > > > > > >> > Best Regards.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Färrükh Ähmëd