[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Having trouble with C++ OOT block in restricting output to those inp

From: Martin Luelf
Subject: Re: Having trouble with C++ OOT block in restricting output to those input values I wish to pass
Date: Tue, 15 Sep 2020 07:28:26 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0

Dear George,

please keep the list in CC so other people can read this discussion in the future.

general_work can be called with any number of input parameters. This could be just 5, or 500 because it can buffer the repeated result from your repeating source. You can somewhat steer this with the forecast method by telling GNURadio that you need 100 inputs. In that case GNURadio will not call general_work with less than 100 inputs, but it might still call it with more than 100. Also for unittests that do not repeat keep in mind that GNURadio will silently drop any excess inputs that are less than the requested size.

I suggest you put a print statement both in forecast and general_work to and then run your code multiple times (also under different loads) to get a better idea of what GNURadio is doing under the hood.


On 15.09.20 06:28, George Edwards wrote:
Hi Martin,

I was out of town for the weekend, so I am just getting at my email.

Thanks for your detailed reply as usual. I really appreciate your help and I know it is a lot of effort and time on your part so I really appreciate it.
 The param cnt works well and is used to synch up the indices for the in and out arrays. For example, suppose the input data was {1,2,3,0,1,3,2,5,6} and the preamble pattern is {3,0,1,3,2}. Then after initialization, cnt = 7 and i = 7 and of course the number of sync bytes = buf_size = 5. Before leaving the initialization loop I memcpy the 5 preamble bytes to the out array, which means the next index in "out" to write on the next input data sample is out[5], however, this sample comes from in[i] = in[7]. Immediately after initialization, hence the relationship: out[i-cnt+buf_size] = in[i] is correct because out[5] = in[7] and as i increases, we get out[6] = in[8], etc.

However, I see one big problem that will cause my logic to fail and it is based on your explanation that each time the general_work method is called it brings in a new set of data and i starts over in the "for loop". I will need to change my logic, but please allow me to ask another question. Let's say, I have a source which generates 100 bytes and it Repeats. In Gnuradio, does this mean that each time the general_work method is called it receives a block of100 bytes of data? Thus, in the C logic test "for (int i = 0; i < ninput_items[0]; i++)", can I assume the Gnuradio parameter ninput_items[0] is equal to 100 and each function call?

Thanks for the help.

Best Regards,

On Sun, Sep 13, 2020 at 2:51 AM Martin Luelf <mail@mluelf.de <mailto:mail@mluelf.de>> wrote:

    Hi George,

    noutput_items is a number given to you by GNURadio. I would not
    overwrite this variable, otherwise you no longer know how many items
    can write into the output buffer. Create a new variable nitems_written
    or similar to track how many items you have written.

    Also have a look at what each variable is tracking exactly. cnt
    seems to
    be the number of input bytes you have checked for the preamble. But you
    also use it to compute the position in the out array, which does not
    make sense to me. cnt would increase with more spurious bytes in front
    of the preamble, but the output position should not depend on the
    of spurious bytes. From the code you showed me cnt also seems to
    between multiple calls to general_work, but the out array is always
    clean when general_work is called again (meaning that on every
    general_work call you should always start writing to element 0).

    You loop over your entire input array and if the preamble was found you
    copy the entire preamble and then you copy every following byte one by
    one. That means you copy the entire preamble again (without the first
    byte) and if your input buffer is longer than your message size you
    copy more bytes to the output than your message is long which could
    result in missing the next preamble. Also you are potentially writing
    more items into the out array than GNURadio has space for, which is why
    I recommend you not to overwrite noutput_items and while you are still
    learning check that your computed indices for out and in are within the
    allowed bounds before every single write and read (i.e. by using

    What messages are you using in your unittest and what kind of spurious
    bytes? Is is all zeros, or all ones? I would recommend you use
    bytes, e.g. an increasing counter for your message and a decreasing
    counter starting from 255 for your spurious bytes so you can quickly
    spot if your messages are cropped, or off in some other way. Because
    way you describe your results it seems like one message is copied
    I could not find any obvious bug in your code that would output a
    message exactly twice, so I suppose it copies some data that look
    like a
    message on the first glance.


    On 11.09.20 03:06, George Edwards wrote:
     > Hi Martin,
     > Thanks for your detailed answer. I really appreciate the great
     > you put into explaining how things work. I am still on the
    learning curve.
     > I used your suggestions to the best of my understanding and it
     > beautifully for the one sync pattern test vector in the QA test.
     > Then, I took your suggestion for repeated sync patterns using an
     > flag which I reset to 0 to restart the process. For the QA test, I
     > repeated the original input data twice (now I have 2
    sync patterns), so
     > the expected QA output should be the original output repeated
    twice. I
     > modified the C code by adding an if statement at the end to check if
     > noutput_items == Buf_size+message_size (buf_size is the length of
     > pattern, which I call preamble) and if it is, I reset the init
    flag to
     > zero as well as other params used in the initialization section
    of the
     > code. The QA test failed by producing an output with 3 repeated
     > of the original output rather than the expected 2 copies. I do not
     > expect you to send too much time looking at my code below,
    however, I
     > would appreciate it very much, if you would glance at it to see
    if you
     > can spot what I am doing wrong. The test to un-initialize
    (setting init
     > to 0) was done towards the end of the code block after the
    consume method.
     >        int kk = 0;____
     >        for (int i = 0; i < ninput_items[0]; i++)____
     >        {____
     >           if(!init){__
     >              cnt += 1;                   // cnt number of passing
     >              kk = initialize(in[i]);____
     >              if (kk == 0){____
     >                  noutput_items =  cnt;____
     >              }else{____
     >                  memcpy((void*)out, (const void*)preamble,
     >                  noutput_items = cnt;____
     >              }
     >           } else {
     >             out[i-cnt+buf_size] = in[i];____
     >             noutput_items = buf_size + message_size;____
     >           }____
     >        } ____
     >        consume_each (noutput_items);____
     >        if (noutput_items == buf_size + message_size){____
     >           init = 0;      // re-initialize all____
     >           cnt = 0;____
     >           kk = 0;____
     >        }____
     >        return noutput_items;____
     >      }
     > Thanks very much for the help.
     > Regards,
     > George
     > On Thu, Sep 10, 2020 at 12:06 AM Martin Luelf <mail@mluelf.de
     > <mailto:mail@mluelf.de <mailto:mail@mluelf.de>>> wrote:
     >     Dear George,
     >     this also caused me a lot of headache when I started with
    GNURadio, so
     >     here is what I learned about it.
     >     Let's start with the forecast method. This tells GNURadio how
     >     input
     >     samples you need on each input stream to generate the
    requested number
     >     of output items. Usually GNURadio will run your forecast
    function a
     >     couple of times with different output numbers to find a good
    data chunk
     >     size to give to the general work function. Keep in mind that
    the number
     >     of required input items you give here is a minimum number and
     >     might decide to give you more input data than you requested.
    It is also
     >     important to know that the number of samples you request here is
     >     just an
     >     estimate. You are not forced to actually process that much data.
     >     Now to the general_work function. noutput_items tells you how
     >     samples GNURadio would like you to produce (this is a maximum
     >     you
     >     may produce less). It also tells you how much memory is
    allocated in
     >     every array in the output_items vector. If you have only one
    output and
     >     you used the default <out type> *out = (<out type> *)
     >     definition this tells you how many items you can place (at
    most) into
     >     the out array.
     >     The ninput_items[] array tells you how many input items are
     >     available in
     >     the input arrays. Again if you just have one input and you
    use const
     >     <in
     >     type> *in = (const <in type> *) input_items[0];
    ninput_items[0] is the
     >     number of inputs available in the in array. You may not read
    more items
     >     than that from the array.
     >     Within the given input symbols you can start looking for your
     >     pattern. If you generate output you have to write (in your case
     >     copy) it
     >     to the out array. At the end of general_work you call
    consume(0, K)
     >     with
     >     the number of input items K that you have consumed on input
    0. That is
     >     how many of the input items you have used and do not need to
    see again.
     >     If you consume 0 symbols the next call to general_work will
    show you
     >     the
     >     exact same input samples again. If you consume
    ninput_items[0] you will
     >     see completely new input samples on the first input the next time
     >     general_work is called. And then you return the number of
    samples you
     >     generated (i.e. how much samples you put into the out array).
     >     number must be smaller or equal noutput_items, because
    otherwise you
     >     would have written out of the allocated memory which might
    result in a
     >     segfault/core dump. Note that you don't have to call consume
    at the
     >     very
     >     end of work and there is also another way of telling GNURadio
    how many
     >     samples you have produced, but let's leave that for another day.
     >     So a very easy (but not the most efficient) setup for your
     >     could be:
     >     Add a boolean flag to your _impl class that both forecast and
     >     general_work can read/write to. This flag will indicate
    whether or not
     >     you have found the sync pattern or not. You initialize this flag
     >     with false.
     >     Assume you have a sync pattern of length L and a message with
    M data
     >     symbols afterwards.
     >     In forecast if the flag is set (meaning you have found the sync
     >     pattern)
     >     you need L+M symbols of input. If the flag is not set you
    need L input
     >     samples, regardless of how many output samples GNURadio wants
    you to
     >     generate.
     >     In general work if the flag is false you search the input for
    the sync
     >     pattern. If you found it at position i (counting from 0) you
    set the
     >     flag to true, consume i samples (i.e. everything before the sync
     >     marker). If the sync marker is not found you keep the flag to
    false and
     >     consume the inputs that you have searched so far. In both
    cases you
     >     return 0 since you have not generated any output yet.
     >     If the flag is true you copy the first L+M samples from the
    input to
     >     the
     >     output, you set the flag to false (because after the data you
    have to
     >     start searching for the sync marker again) you consume L+M
    samples and
     >     return L+M samples.
     >     Note: This is a very easy to understand scheme, but
    unfortunately not
     >     very efficient. You only process a single block of either
     >     spurious symbols, or one sync marker and data at a time. So
    once you
     >     have a good understanding of how this works you should tweak
    that block
     >     to be able to process multiple blocks of spurious symbols and
     >     patterns/data within once call to general_work. It uses the same
     >     kind of
     >     logic, but requires more housekeeping of counters and indices.
     >     If your input is symbols rather than bits/bytes you should
    also look at
     >     the paper from J. Massey "Optimum Frame Synchronization" from
    1972 on
     >     how to perform optimum sync marker detection, which performs
     >     than
     >     the intuitive correlation search.
     >     Hope that gets you started.
     >     Yours
     >     Martin
     >     On 10.09.20 04:34, George Edwards wrote:
     >      > Hello,
     >      >
     >      > I am writing an OOT block in C++ that receives a sequence of
     >     numbers and
     >      > searches through for a sync pattern and passes to its
    output the
     >     sync
     >      > pattern and the  bytes of data which follows it. The QA test
     >     shows the
     >      > block recognizes the pattern and will pass the pattern
    along with
     >     the
     >      > data that follow it, but there is a problem. The block
    does not
     >     know a
     >      > priori the number of spurious bytes preceding the sync
    pattern of
     >     bytes,
     >      > so I cannot set up the true relationship between the
     >     and
     >      > noutput_items. However, the block can count the number of
    bytes that
     >      > came in before the pattern. This is my problem:
     >      >
     >      > 1) In the OOT general_work method: If I set noutput_items =
     >      > ninput_items[0], then in addition to passing the correct
    data to the
     >      > output, it passes trailing zeros equal to the number of
     >     bytes
     >      > that entered before the pattern.
     >      >
     >      > 2) If I set the return noutput_items = ninput_items - cnt
     >     cnt is
     >      > the number of spurious bytes before the pattern) depending on
     >     where I
     >      > put noutput_items in the code, it either throws a core dump or
     >     cuts off
     >      > the number of true data.
     >      >
     >      > Also, within the forecast method, I simply use a _for_
    loop and set
     >      > ninput_items_required[i]=noutput_items;
     >      >
     >      > I will appreciate any help to point me in the right
    direction in
     >     dealing
     >      > with this non-regular output/ inputs relationship.
     >      >
     >      > Thank you!
     >      >
     >      > George

reply via email to

[Prev in Thread] Current Thread [Next in Thread]