Re: [Discuss-gnuradio] Volk machine: ssse3_32 segmentation faults (was RE: How to diagnose make test failures)

[Stamper, Brian]

Hi Tom,

>From: address@hidden [mailto:address@hidden On Behalf Of Tom Rondeau
>Sent: Tuesday, April 23, 2013 10:15 PM
>On Tue, Apr 23, 2013 at 10:37 AM, Stamper, Brian <address@hidden> wrote:
...
>> Previously I posted that I was getting the following make test failures. I'm 
>> building GNU Radio in Fuduntu (a fork of Fedora) on a 32-bit Intel Atom N270 
>> based netbook (an Asus Eee 1000).
...
>> I started running the test shell scripts individually and found that they 
>> all crashed with the same message:
>>
>> Using Volk machine: ssse3_32
>> Segmentation fault
...
>The best way to get info from here is to use gdb. In one of the QA test files 
>(the Python file that's in the source tree), add the lines:
>
>import os
>print os.getpid()
>raw_input()
>
>Do this before any other Python code. When you run it now, it will print the 
>PID of the process. In another terminal, run gdb (probably as root), and then 
>run 'attach <pid>' where <pid> is the PID printed by the program. Then press 
>'c' to continue in gdb. On the other terminal running the QA code, press enter 
>(to proceed past the 'raw_input' line). Wait until it crashes, then, in gdb, 
>type 'bt' to get a back trace. Send that along and it will help us figure out 
>where the seg fault is happening.
>
>Tom
>

So I picked on one of the failing qa_fft_filter tests:
$ vi 
/home/brian/SDR/gnuradio/gnuradio-core/src/python/gnuradio/gr/qa_fft_filter.py
$ /bin/sh 
"/home/brian/SDR/gnuradio/build/gnuradio-core/src/python/gnuradio/gr/qa_fft_filter_test.sh"
[then switch to gdb, attach, continue, await crash.]

Here's the bt:

(gdb) bt
#0 0xb6a6f7a7 in volk_32fc_x2_multiply_32fc_a_sse3 () from 
/home/brian/SDR/gnuradio/build/volk/lib/libvolk.so.0.0.0
#1 0xb6a3cc92 in get_volk_32fc_x2_multiply_32fc_a () from 
/home/brian/SDR/gnuradio/build/volk/lib/libvolk.so.0.0.0
#2 0xb6e8e539 in gri_fft_filter_ccc_generic::filter(int, std::complex<float> 
const*, std::complex<float>*) ()
from 
/home/brian/SDR/gnuradio/build/gnuradio-core/src/lib/libgnuradio-core-3.6.4.1.so.0.0.0
#3 0xb6e961a5 in gr_fft_filter_ccc::work(int, std::vector<void const*, 
std::allocator<void const*> >&, std::vector<void*, std::allocator<void*> >&) () 
from 
/home/brian/SDR/gnuradio/build/gnuradio-core/src/lib/libgnuradio-core-3.6.4.1.so.0.0.0
#4 0xb6e5fd8a in gr_sync_decimator::general_work(int, std::vector<int, 
std::allocator<int> >&, std::vector<void const*, std::allocator<void const*> 
>&, std::vector<void*, std::allocator<void*> >&) ()
from 
/home/brian/SDR/gnuradio/build/gnuradio-core/src/lib/libgnuradio-core-3.6.4.1.so.0.0.0
#5 0xb6e3fdf1 in gr_block_executor::run_one_iteration() ()
from 
/home/brian/SDR/gnuradio/build/gnuradio-core/src/lib/libgnuradio-core-3.6.4.1.so.0.0.0
#6 0xb6e62ad6 in 
gr_tpb_thread_body::gr_tpb_thread_body(boost::shared_ptr<gr_block>, int) ()
from 
/home/brian/SDR/gnuradio/build/gnuradio-core/src/lib/libgnuradio-core-3.6.4.1.so.0.0.0
#7 0xb6e5ce6b in 
boost::detail::function::void_function_obj_invoker0<gruel::thread_body_wrapper<tpb_container>,
 void>::invoke(boost::detail::function::function_buffer&) ()
from 
/home/brian/SDR/gnuradio/build/gnuradio-core/src/lib/libgnuradio-core-3.6.4.1.so.0.0.0
#8 0xb6d91b86 in boost::detail::thread_data<boost::function0<void> >::run() ()
from /home/brian/SDR/gnuradio/build/gruel/src/lib/libgruel-3.6.4.1.so.0.0.0
#9 0xb6cf924d in ?? () from /usr/lib/libboost_thread-mt.so.1.48.0
#10 0xb7608adf in start_thread () from /lib/libpthread.so.0
#11 0xb751042e in clone () from /lib/libc.so.6


That does appear to point to Volk, so I continue to look at that. I learned 
about "volk_profile" and ran it, and then tried make test again. I actually 
ended up with more failures than before, but it was interesting that it did 
something different.

When run, volk_profile builds ~/.volk/volk_config, with entries like this:
        #this file is generated by volk_profile.
        #the function name is followed by the preferred architecture.
        volk_16ic_s32f_deinterleave_real_32f_a generic
        volk_16ic_deinterleave_real_8i_a ssse3
        volk_16ic_deinterleave_16i_x2_a sse2
        volk_16ic_s32f_deinterleave_32f_x2_a sse
        ...

Where "generic", "ssse3", "sse2", etc. are all chosen by volk_profile based on 
how well your machine runs each. So I decided to try different volk_config 
versions where I set all functions to the same architecture, e.g. one version 
of volk_config with all "sse" like this:
        volk_16ic_s32f_deinterleave_real_32f_a sse
        volk_16ic_deinterleave_real_8i_a sse
        volk_16ic_deinterleave_16i_x2_a sse
        volk_16ic_s32f_deinterleave_32f_x2_a sse
        ...

Then for each version of volk_config I ran make test again. The results:
type                            results
------                  ----------
volk_profile            92% tests passed, 16 tests failed out of 192
generic                 99% tests passed, 2 tests failed out of 192
sse                             92% tests passed, 15 tests failed out of 192
sse2                            99% tests passed, 2 tests failed out of 192
sse3                            94% tests passed, 12 tests failed out of 192
ssse3                           99% tests passed, 2 tests failed out of 192

There is generally overlap in the failures, but clearly which arch I chose 
affected the outcome. The best run time was on sse2, where only these two tests 
fail:
114 - qa_ctcss_squelch
151 - qa_constellation_receiver

These two failures actually appeared with each version of volk_config. #114 I 
actually saw before, but it is not a seg fault, it is assertion errors, so I 
can continue to debug that on my own a bit. #151 is a seg fault, but I found 
that it is intermittent - often the same test passes or fails, even though I 
haven't changed anything.

tl;dr
It does appear to be an issue with configuring Volk for my system. Setting all 
functions to use "sse2" in volk_config clears up most of the problems on my 
system, but not all.

Thanks again,
Brian