Re: [Discuss-gnuradio] Debugging ControlPort/Thrift problem (Re: costas ambiguity and correlate-and-sync block in qpsk)

[Andy Walls]

On Sun, 2016-04-03 at 13:01 -0400, Andy Walls wrote:
> >On April 3, 2016 11:31:02 AM EDT, Tom Rondeau wrote:
> >>On Sun, Apr 3, 2016 at 11:27 AM, Landsman, Arik wrote:

[snip]

> >>>> Here is some good info on how GRNURadio's control port works at a
> >>high
> >>>> level:
> >>>> https://gnuradio.org/redmine/projects/gnuradio/wiki/ControlPort
> >>>>
> >><http://redir.aspx?REF=azbe_GyxFRKBpOrbCXsEn87Wj8ET5GUm0QC3BOXvRTTOmAsr1FvTCAFodHRwczovL2dudXJhZGlvLm9yZy9yZWRtaW5lL3Byb2plY3RzL2dudXJhZGlvL3dpa2kvQ29udHJvbFBvcnQ.>
> >>>> https://gnuradio.org/doc/doxygen/page_ctrlport.html
> >>>>
> >><http://redir.aspx?REF=SxLIHILotdP8HOdDt_K6tXfxwQKnxOUFu1TiyaDWXwfOmAsr1FvTCAFodHRwczovL2dudXJhZGlvLm9yZy9kb2MvZG94eWdlbi9wYWdlX2N0cmxwb3J0Lmh0bWw.>
> >>>>
> >>>> From that second page, it appears that running two flowgraphs on
> >the
> >>>> same machine has a problem caused by Apache Thrift and GNURadio's
> >>>> configuration of the Control Port network port (default of 9090).
> >>>>
> >>https://gnuradio.org/doc/doxygen/page_ctrlport.html#ctrlport_thrift_issues
> >>>>
> >><http://redir.aspx?REF=PEreM48t5VngbsJ1nPZyY40coia_Ki9MnN28CfQhswzOmAsr1FvTCAFodHRwczovL2dudXJhZGlvLm9yZy9kb2MvZG94eWdlbi9wYWdlX2N0cmxwb3J0Lmh0bWwjY3RybHBvcnRfdGhyaWZ0X2lzc3Vlcw..>
> >>>>
> >>>> So please only run on flowgraph at a time, at first, to
> >troubleshoot
> >>>> this issue.
> >>>>
> >>>> Second, you may want to try setting the GR_CONF_THRIFT_PORT
> >>environment
> >>>> variable to something other than 9090, a port that you know is open
> >>on
> >>>> your machine, for each separate flowgraph that you run.
> >>>>
> >>>> FYI, the modulate_vector block runs a "mini"-flowgraph, before the
> >>main
> >>>> flowgraph actually runs:
> >>>>
> >>>>
> >>>>
> >>https://github.com/gnuradio/gnuradio/blob/master/gr-digital/lib/modulate_vector.cc#L59
> >>>>
> >><http://redir.aspx?REF=rCyK29_HIPXb4AHqBP038K2B4o2SxURzlIzwumhDH6vOmAsr1FvTCAFodHRwczovL2dpdGh1Yi5jb20vZ251cmFkaW8vZ251cmFkaW8vYmxvYi9tYXN0ZXIvZ3ItZGlnaXRhbC9saWIvbW9kdWxhdGVfdmVjdG9yLmNjI0w1OQ..>
> >>>>
> >>>> Hopefully, there isn't some sort of weird race condition with the
> >>Thrift
> >>>> port being open for the modulate_vector block's mini-flowgraph and
> >>the
> >>>> main flowgraph.
> >>>>
> >>>>
> >>>> Hi Tom,
> >>>>
> >>>> Do you have any comments or insights on Arik's Thrift error message
> >>>> and/or the possibility of modulate_vector's mini-flowgraph causing
> >>>> problems with Thrift's network port and the main flowgraph?
> >>>>
> >>>> Regards,
> >>>> Andy
> >>>>
> >>>
> >>> Andy,
> >>>
> >>> I think you've got it on the Thrift issue. However, running the
> >>modulate
> >>> vector mini flowgraph first should kill itself immediately and free
> >>up the
> >>> port, so I don't think that's the problem. On the other hand, I've
> >>been
> >>> seeing issues recently where every time I launch GRC, it seems to
> >>cause a
> >>> conflict with whatever port I'm setting. Almost as though GRC is
> >>launching
> >>> something and holding on to the port specified in the thrift.conf
> >>file. I
> >>> thought this could just be a problem on my laptop (I've been
> >>traveling a
> >>> lot -- more than usual even -- and developing on my laptop can lead
> >>to me
> >>> doing a lot of crazy things), but I think I've since seen it on my
> >>office
> >>> workstation as well. That could be a problem here, too. We'll have
> >to
> >>track
> >>> that down.
> >>>
> >>> Using the GR_CONF_THRIFT_PORT variable is a good idea to set the
> >port
> >>per
> >>> flowgraph you're running. Another option is to set the port to 0 in
> >>> thrift.conf, which will cause Thrift to open an ephemeral port for
> >>each
> >>> server is creates. You'll just have to pay attention to the message
> >>output
> >>> that announces the ControlPort endpoint to find out what port it's
> >>running
> >>> on.
> >>>
> >>> Tom

OK, some I'm getting this runtime error at flowgraph startup
consistently:

"rpcmanager: Aggregator not in use, and a rpc booter is already registered"

strace shows that when the assetion stops the program, some block
threads have been spawned, but they are all waiting.  It's the main
program thread which emits this.

Running the flowgraph in gdb, and checking some of the relevant
variables in the stack frame:

(gdb) frame 6
#6  0x00007fffeacc3376 in rpcmanager::register_booter (booter=<optimized out>)
    at 
/home/andy/work/pybombs/src/gnuradio/gnuradio-runtime/lib/controlport/rpcmanager.cc:66
66          throw std::runtime_error("rpcmanager: Aggregator not in use, and a 
rpc booter is already registered\n");
(gdb) info locals
No locals.
(gdb) print rpcmanager::booter_registered
$1 = true
(gdb) print rpcmanager::aggregator_registered
$2 = false
(gdb) print rpcmanager::make_aggregator
$3 = false
(gdb)  print rpcmanager::aggregator
$4 = {_M_ptr = 0x0}
(gdb) print rpcmanager::boot
$5 = {_M_ptr = 0x0}


So two things strike me as a problem here:
1. booter_registered == true, but rpcmanager::boot == NULL
2. make_aggregator == false, and nowhere in the code can I find where it
is ever set to true.

I think this very recent commit might fix #1:
https://github.com/gnuradio/gnuradio/commit/fde2c26aa9da54cfdb5cb18b6e722de031be65ae

But can someone clue me in on how #2 is not a problem for this code:
https://github.com/gnuradio/gnuradio/blob/master/gnuradio-runtime/lib/controlport/rpcmanager.cc#L44

Is there some C++ magic I'm missing?

Regards,
Andy