Re: [Discuss-gnuradio] Segfault in volk_32fc_x2_multiply_32fc_a_avx2_fma

[devin kelly]

Calling 'info variables' (or args or locals) the last few frames didn't give me any real info so I built a copy of GR/Volk with debug symbols.  I ran the FG again, this time from GDB, here's my back trace.  In this backtrace you can see the arguments passed in each call.  I have an i7-5600U CPU @ 2.60GHz, the volk_profile is appended at the bottom.

Here's are the links for the relevant code:

https://github.com/gnuradio/volk/blob/f0b722392950bf7ede7b32f5ff60019bce7a8592/kernels/volk/volk_32fc_x2_multiply_32fc.h#L232
https://github.com/gnuradio/gnuradio/blob/master/gr-filter/lib/fft_filter.cc#L323
https://github.com/gnuradio/gnuradio/blob/222e0003f9797a1b92d64855bd2b93f0d9099f93/gr-digital/lib/corr_est_cc_impl.cc#L214

Could the problem be that nitems is 257 and num_points is 512?  Or should nitems really be 256 and not 257?

Thanks,

Devin

(gdb) bt
#0  0x00007fffdcaccb57 in volk_32fc_x2_multiply_32fc_a_avx2_fma (__P=0x3b051b0)
    at /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/avxintrin.h:835
#1  0x00007fffdcaccb57 in volk_32fc_x2_multiply_32fc_a_avx2_fma (cVector=0x3b1f770, aVector=0x3b051b0, bVector=0x3b240e0, num_points=512)
    at /local_disk/gr_3.7.9_src/volk/kernels/volk/volk_32fc_x2_multiply_32fc.h:242
#2  0x00007fffdc945a75 in __volk_32fc_x2_multiply_32fc_a (cVector=0x3b1f770, aVector=0x3b051b0, bVector=0x3b240e0, num_points=512)
    at /local_disk/gr_3.7.9_src/volk/build_debug/lib/volk.c:7010
#3  0x00007fffd3f8e360 in gr::filter::kernel::fft_filter_ccc::filter(int, std::complex<float> const*, std::complex<float>*) (this=0x3b02f40, address@hidden, address@hidden, address@hidden)
    at /local_disk/gr_3.7.9_src/gnuradio/gr-filter/lib/fft_filter.cc:323
#4  0x00007fffd42910df in gr::digital::corr_est_cc_impl::work(int, std::vector<void const*, std::allocator<void const*> >&, std::vector<void*, std::allocator<void*> >&) (this=0x3b01560, noutput_items=257, input_items=..., output_items=std::vector of length 1, capacity 1 = {...})
    at /local_disk/gr_3.7.9_src/gnuradio/gr-digital/lib/corr_est_cc_impl.cc:237
#5  0x00007fffdd064907 in gr::sync_block::general_work(int, std::vector<int, std::allocator<int> >&, std::vector<void const*, std::allocator<void const*> >&, std::vector<void*, std::allocator<void*> >&) (this=0x3b015b8, noutput_items=<optimized out>, ninput_items=..., input_items=..., output_items=...) at /local_disk/gr_3.7.9_src/gnuradio/gnuradio-runtime/lib/sync_block.cc:66
#6  0x00007fffdd02f70f in gr::block_executor::run_one_iteration() (address@hidden)
    at /local_disk/gr_3.7.9_src/gnuradio/gnuradio-runtime/lib/block_executor.cc:438
#7  0x00007fffdd06da8a in gr::tpb_thread_body::tpb_thread_body(boost::shared_ptr<gr::block>, int) (this=0x7fff83ffedb0, block=..., max_noutput_items=<optimized out>) at /local_disk/gr_3.7.9_src/gnuradio/gnuradio-runtime/lib/tpb_thread_body.cc:122
#8  0x00007fffdd062761 in boost::detail::function::void_function_obj_invoker0<gr::thread::thread_body_wrapper<gr::tpb_container>, void>::invoke(boost::detail::function::function_buffer&) (this=0x3bc3ec0)
    at /local_disk/gr_3.7.9_src/gnuradio/gnuradio-runtime/lib/scheduler_tpb.cc:44
#9  0x00007fffdd062761 in boost::detail::function::void_function_obj_invoker0<gr::thread::thread_body_wrapper<gr::tpb_container>, void>::invoke(boost::detail::function::function_buffer&) (this=0x3bc3ec0)
    at /local_disk/gr_3.7.9_src/gnuradio/gnuradio-runtime/include/gnuradio/thread/thread_body_wrapper.h:51
#10 0x00007fffdd062761 in boost::detail::function::void_function_obj_invoker0<gr::thread::thread_body_wrapper<gr::tpb_container>, void>::invoke(boost::detail::function::function_buffer&) (function_obj_ptr=...) at /usr/include/boost/function/function_template.hpp:153
#11 0x00007fffdd016cd0 in boost::detail::thread_data<boost::function0<void> >::run() (this=<optimized out>)
    at /usr/include/boost/function/function_template.hpp:767
#12 0x00007fffdd016cd0 in boost::detail::thread_data<boost::function0<void> >::run() (this=<optimized out>)
    at /usr/include/boost/thread/detail/thread.hpp:117
#13 0x00007fffdbe4f24a in thread_proxy () at /lib64/libboost_thread-mt.so.1.53.0
#14 0x00007ffff7800dc5 in start_thread () at /lib64/libpthread.so.0
#15 0x00007ffff6e2528d in clone () at /lib64/libc.so.6

Here are the locals on the last few frames:

(gdb) f 0
#0  0x00007fffdcaccb57 in _mm256_load_ps (__P=0x3b051b0) at /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/avxintrin.h:835
835       return *(__m256 *)__P;
(gdb) info locals
No locals.
(gdb) f 1
#1  volk_32fc_x2_multiply_32fc_a_avx2_fma (cVector=0x3b1f770, aVector=0x3b051b0, bVector=0x3b240e0, num_points=512)
    at /local_disk/gr_3.7.9_src/volk/kernels/volk/volk_32fc_x2_multiply_32fc.h:242
242         const __m256 x = _mm256_load_ps((float*)a); // Load the ar + ai, br + bi as ar,ai,br,bi
(gdb) info locals
y = {-4.87433296e+17, 4.59163468e-41, -3.92813517e+17, 4.59163468e-41, 5.15677835e-43, 0, 5.26888223e-43, 0}
tmp2x = {6.389921e-43, 0, -512.314453, 4.59163468e-41, 1.26116862e-44, 0, -4.87433296e+17, 4.59163468e-41}
x = {-512.314453, 4.59163468e-41, 0, 0, 2.76102662, -3.64918089, -4.92134571, -1.06491208}
yl = {4.14784345e-43, 0, 1.26116862e-44, 0, -4.87442367e+17, 4.59163468e-41, -4.87439343e+17, 4.59163468e-41}
yh = {-1674752, 4.59163468e-41, 0, 0, -1.50397414e-36, 4.59163468e-41, -3.31452625e+17, 4.59163468e-41}
tmp2 = {6.72623263e-44, 1.2751816e-43, 2.24207754e-44, 0, 7.17464814e-43, 0, -3.31440427e+17, 4.59163468e-41}
z = {0.794147611, 0, 0.263988227, 0, -0.380019426, 0, -0.953325868, 0}
number = 0
quarterPoints = 128
c = 0x3b1f770
a = 0x3b051b0
b = 0x3b240e0
(gdb) f 2
#2  0x00007fffdc945a75 in __volk_32fc_x2_multiply_32fc_a (cVector=0x3b1f770, aVector=0x3b051b0, bVector=0x3b240e0, num_points=512)
    at /local_disk/gr_3.7.9_src/volk/build_debug/lib/volk.c:7010
7010        volk_32fc_x2_multiply_32fc_a(cVector, aVector, bVector, num_points);
(gdb) info locals
No locals.
(gdb) f 3
#3  0x00007fffd3f8e360 in gr::filter::kernel::fft_filter_ccc::filter (this=0x3b02f40, address@hidden,
    address@hidden, address@hidden) at /local_disk/gr_3.7.9_src/gnuradio/gr-filter/lib/fft_filter.cc:323
323               volk_32fc_x2_multiply_32fc_a(c, a, b, d_fftsize);
(gdb) info locals
a = <optimized out>
b = <optimized out>
c = <optimized out>
i = 0
dec_ctr = 0
j = <optimized out>
ninput_items = 257

My volk profile results:

$  volk_profile -R 32fc_x2_multiply
Using Volk machine: avx2_64_mmx_orc
RUN_VOLK_TESTS: volk_32fc_x2_multiply_32fc(131071,1987)
u_avx2_fma completed in 220ms
u_avx completed in 220ms
u_sse3 completed in 240ms
generic completed in 2810ms
a_avx2_fma completed in 200ms
a_avx completed in 220ms
a_sse3 completed in 230ms
a_generic completed in 2810ms
u_orc completed in 280ms
Best aligned arch: a_avx2_fma
Best unaligned arch: u_avx2_fma
RUN_VOLK_TESTS: volk_32fc_x2_multiply_conjugate_32fc(131071,1987)
u_avx completed in 230ms
u_sse3 completed in 230ms
generic completed in 2790ms
a_avx completed in 220ms
a_sse3 completed in 230ms
a_generic completed in 2800ms
Best aligned arch: a_avx
Best unaligned arch: u_avx
Writing "/home/devin/.volk/volk_config"...

On Mon, Mar 7, 2016 at 10:38 PM, West, Nathan <address@hidden> wrote:

On Mon, Mar 7, 2016 at 10:18 PM, West, Nathan <address@hidden> wrote:

On Mon, Mar 7, 2016 at 2:32 PM, devin kelly <address@hidden> wrote:

Hello,

I've built a flowgraph (grc, python attached) that usually (but not always) produces a segfaults in volk_32fc_x2_multiply_32fc_a_avx2_fma.  The segfault occurs in the FFT filter in correlation estimator block.  I'm not sure if it's the Volk code that's causing the segfault or the GR code calling it or what I'm putting into GR.  I've got a back trace below if that helps. 

Also, at the tag gate in my flowgraph I would usually have a USRP transmitter and a USRP receiver in a before the Correlation Estimate in a separate flowgraph but since I can re-create the segfault in the flowgraph I simplified.

I'm using GR 3.7.9, Volk 1.2.1 and UHD 3.9.2 (though I don't call any UHD code here).

Thanks for any help,

Devin

$ gdb /usr/bin/python core.13408
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/python2.7...Reading symbols from /usr/bin/python2.7...(no debugging symbols found)...done.
(no debugging symbols found)...done.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `python2 ./segfault_test.py'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f11a4d064c0 in volk_32fc_x2_multiply_32fc_a_avx2_fma () from /local_disk/gr_3.7.9/lib/libvolk.so.1.2.1
Missing separate debuginfos, use: debuginfo-install python-2.7.5-34.el7.x86_64
(gdb) bt
#0  0x00007f11a4d064c0 in volk_32fc_x2_multiply_32fc_a_avx2_fma () at /local_disk/gr_3.7.9/lib/libvolk.so.1.2.1
#1  0x00007f119c1d0006 in gr::filter::kernel::fft_filter_ccc::filter(int, std::complex<float> const*, std::complex<float>*) ()
    at /local_disk/gr_3.7.9/lib64/libgnuradio-filter-3.7.9.so.0.0.0
#2  0x00007f119c4d724f in gr::digital::corr_est_cc_impl::work(int, std::vector<void const*, std::allocator<void const*> >&, std::vector<void*, std::allocator<void*> >&) () at /local_disk/gr_3.7.9/lib64/libgnuradio-digital-3.7.9.so.0.0.0
#3  0x00007f11a52b1f57 in gr::sync_block::general_work(int, std::vector<int, std::allocator<int> >&, std::vector<void const*, std::allocator<void const*> >&, std::vector<void*, std::allocator<void*> >&) () at /local_disk/gr_3.7.9/lib64/libgnuradio-runtime-3.7.9.so.0.0.0
#4  0x00007f11a527a6bd in gr::block_executor::run_one_iteration() () at /local_disk/gr_3.7.9/lib64/libgnuradio-runtime-3.7.9.so.0.0.0
#5  0x00007f11a52baf60 in gr::tpb_thread_body::tpb_thread_body(boost::shared_ptr<gr::block>, int) ()
    at /local_disk/gr_3.7.9/lib64/libgnuradio-runtime-3.7.9.so.0.0.0
#6  0x00007f11a52aebc1 in boost::detail::function::void_function_obj_invoker0<gr::thread::thread_body_wrapper<gr::tpb_container>, void>::invoke(boost::detail::function::function_buffer&) () at /local_disk/gr_3.7.9/lib64/libgnuradio-runtime-3.7.9.so.0.0.0
#7  0x00007f11a5260910 in boost::detail::thread_data<boost::function0<void> >::run() ()
    at /local_disk/gr_3.7.9/lib64/libgnuradio-runtime-3.7.9.so.0.0.0
#8  0x00007f11a40b824a in thread_proxy () at /lib64/libboost_thread-mt.so.1.53.0
#9  0x00007f11bfa38dc5 in start_thread () at /lib64/libpthread.so.0
#10 0x00007f11bf05d28d in clone () at /lib64/libc.so.6

_______________________________________________
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

I couldn't reproduce this on my machine with avx2. Your GRC flowgraph didn't draw anything, so I replaced your FG with random source(0,255) -> const. modulator -> correlation est -> sinks.

I'm wondering if a bad vector length is getting passed in by accident. Can you get the parameters to the VOLK call when it crashes? If you're not familiar with gdb do something like the following:

gdb> f 0

gdb> info variables

Also, I'd like to know what processor you have. I don't think it has anything to do with this bug, but my processor (i7-4700MQ) doesn't generally see much benefit from AVX much less AVX2. What were the times volk_profile gives you for this kernel? (use volk_profile -R 32fc_x2_multiply)