discuss-gnuradio
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Discuss-gnuradio] fast parallel filtering


From: Andy Walls
Subject: Re: [Discuss-gnuradio] fast parallel filtering
Date: Sat, 18 Mar 2017 16:45:18 -0400

Dirk,

Here is a revised flowgraph, slightly tweaked to reduce the
computational cost of the rotator and the filters.

I wasn't intending to tweak the flowgraph, but today I decided to try
with a correlation filter with -5 kHz to +5 kHz chirp taps.  That
method turned out to be inferior to the PLL, so I didn't leave it in
the flowgraph.

Regards,
Andy

On Fri, Mar 17, 2017 at 11:46 AM, Andy Walls
<address@hidden> wrote:
> On Thu, 2017-03-16 at 21:02 -0400, Andy Walls wrote:
>> Hi Dirk,
>>
>> Thanks.
>>
>> Try the attached flowgraph.  The pulse indicator output is a little crude
>> at the moment (it needs some latching persistence for a human to read),
>> but it does the job for a proof of concept.
>>
>> Note that in the future, you really do want to tune such the the SDR's center
>> freq spike is not in the band of interest.  That will prevent false lock on 
>> the
>> DC spike by the PLL.
>>
>> Regards,
>> Andy
>
> I should also mention a few things:
>
> 1. You could handle multiple channels in one of at least two ways:
>
> a. A polyphase channelizer with each channel having a PLL and Moving
> average filter instance.  (This can get CPU intensive.) Or...
>
> b. Scanning by stepping through and dwelling for a bit on your 10 kHz
> channels of interest.
>
>
> 2. You can get better sensitivity the the PLL+Moving average correlation
> solution by going to a 2 stage process:
>
> a. Use the PLL+Moving Average filter to initially acquire some pulses.
>
> b. Once you find some pulses, extract what frequency they were at from
> the PLL's loop filter state at the time it was locked on the pulse.  You
> can then use this exact frequency information for a dedicated
> correlation filter to pull even weaker pulses out of the noise, and
> maintain track.  You'll have to use the magnitude of the correlation and
> not just the real part (because you'll have an arbitrary phase shift,
> since a straight correlation filter is a phase offset detector and not
> phase tracker like a PLL).
>
> Regards,
> Andy
>
>
>> On Thu, Mar 16, 2017 at 7:12 PM, Dirk Gorissen <address@hidden> wrote:
>> > Hi Andy,
>> >
>> > Very quickly collected some data off a different Tx. First at very
>> > close range, moving away, and then back again. It was quick and
>> > antenna was close to computer so perhaps quite noisy but should
>> > definitely have some strong pulses:
>> >
>> > https://drive.google.com/open?id=0B5dKo9igl8W4WkRkUGQzcVJsQnc
>> >
>> > Havent managed to look at the pll stuff yet. Aim to do so this weekend.
>> > Cheers
>> > Dirk
>> >
>> > On 16 March 2017 at 07:22, Dirk Gorissen <address@hidden> wrote:
>> >> Mmm, thats odd. Those settings should be correct, maybe something went
>> >> wrong with the recording :(  I'll try to check in between things at
>> >> work today else will do so tonight.
>> >> Thanks for taking the time to look though.
>> >>
>> >> On 16 March 2017 at 00:39, Andy Walls <address@hidden> wrote:
>> >>> Hi Dirk:
>> >>>
>> >>> In the IQ data file you provided I can seem to find any pulses of the
>> >>> nominal 10 msec length, no matter how I filter and rotate the
>> >>> spectrum.
>> >>> There is a lot of EMI, which looks like the intermodulation products
>> >>> of a continuously on guy who is drifting/chirping down in frequency.
>> >>>
>> >>> So could you please confirm or clarify the following:
>> >>>
>> >>> 1. The format of the binary data file.  I am assuming 32 bit float I
>> >>> and 32 bit float Q pairs, as that is what the GNURadio file sink would
>> >>> normally create.
>> >>> 2. The sample rate.  I am assuming 1 Msps.
>> >>> 3. The center freq.  I am assuming 150.22 MHz.
>> >>> 4. The pulse duration. I am assuming 10 milliseconds.
>> >>> 5. At what frequency offset, from the center frequency, should the
>> >>> pulse be at?  I'm assuming somewhere withing +/- 5 kHz of the center
>> >>> spike, but there are at least two EMI spikes in that range.
>> >>>
>> >>> Thanks.
>> >>>
>> >>> -Andy
>> >>>
>> >>> On Wed, Mar 15, 2017 at 5:23 AM, Dirk Gorissen <address@hidden> wrote:
>> >>>> Hi Andy, Marcus,
>> >>>>
>> >>>> Thanks very much for taking the time to think about this.
>> >>>>
>> >>>> Just to answer your questions:
>> >>>>
>> >>>>>Dirk's problem was the low SNR in his recording, so he'd like to take
>> >>>>>the shape of his pulse into consideration, but he doesn't know the
>> >>>>>spectral position of the same – Dirk, can you confirm?
>> >>>>
>> >>>> Yes, Im dealing with a very weak signal against quite a bit of
>> >>>> background noise so the more prior knowledge I can leverage the
>> >>>> better.
>> >>>>
>> >>>>>Dirk, thinking about this, the relation of the spectral uncertainty to
>> >>>>>the bandwidth of the pulse would be interesting  – can you refresh our
>> >>>>>memory? I think the pulse was 10ms long (so, pretty long) but I forgot
>> >>>>>how it looked like.
>> >>>>
>> >>>> Yes, 10ms long (this is pretty exact), occurring every 1.5 seconds, in
>> >>>> the 150MHz range. My input stream is coming from an SDR.
>> >>>> As an aside I actually have multiple transmitters each pulsing at
>> >>>> slightly different frequencies (e.g., 150.10, 150.22, ...) but Im
>> >>>> happy to treat them independently so you can ignore that and assume
>> >>>> there is just one.
>> >>>>
>> >>>> You can see pictures of the pulse (taken a while back, for a specific
>> >>>> tx frequency) here: https://goo.gl/photos/Y3Ea6kJo1ewrcDYM8
>> >>>>
>> >>>> Note though that this is taken with the transmitter very close to the
>> >>>> antenna so the signal is unrealistically strong. So its just to
>> >>>> illustrate.
>> >>>>
>> >>>> I also quickly captured a file with raw IQ samples (File sink) in case
>> >>>> anybody is interested. It starts with the transmitter very close to
>> >>>> antenna, moves progressively further until out of range and then back.
>> >>>> Its only about a minute or two tops but at 1msps it ended
>> >>>> up as 813 MB though.
>> >>>>
>> >>>> https://drive.google.com/drive/folders/0B5dKo9igl8W4dWpiMmlEYWx3b0k?usp=sharing
>> >>>>
>> >>>>>What you really care about is
>> >>>>>that there's a *tone* for about 8ms < t < 12ms where there is none else
>> >>>>>(to rule out detection of spurs and other interference), is that right?
>> >>>>
>> >>>> Yes. The "tone" happens every 1.5 seconds and I want to catch as many
>> >>>> of those occurrences as possible as the receiver moves through space.
>> >>>> So for every 1.5 seconds I need to make a decision: was there one, or
>> >>>> not.
>> >>>>
>> >>>> Note that the receiver is moving. So perhaps initially I may be well
>> >>>> out of range of the signal and get nothing. But as I move closer as
>> >>>> some point I will start picking up those pulses. The earlier and more
>> >>>> reliably I can pick them up the better.
>> >>>>
>> >>>> I actually did spend some time looking at PLLs & the gnu radio block
>> >>>> at some point. However I readily admit to being a software person not
>> >>>> a DSP/Radio person. I have the day job to deal with today but I will
>> >>>> have a study of the PLL block given Andy's tips and report back.
>> >>>>
>> >>>> Cheers
>> >>>> Dirk
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On 14 March 2017 at 16:37, Andy Walls <address@hidden> wrote:
>> >>>>> On Tue, 2017-03-14 at 12:00 -0400, address@hidden
>> >>>>> wrote:
>> >>>>>> Date: Tue, 14 Mar 2017 15:49:44 +0100
>> >>>>>> From: Marcus M?ller <address@hidden>
>> >>>>>> To: address@hidden
>> >>>>>> Subject: Re: [Discuss-gnuradio] fast parallel filtering
>> >>>>>
>> >>>>>> Hi Andy,
>> >>>>>>
>> >>>>>> Dirk's problem was the low SNR in his recording, so he'd like to take
>> >>>>>> the shape of his pulse into consideration, but he doesn't know the
>> >>>>>> spectral position of the same ? Dirk, can you confirm?
>> >>>>>
>> >>>>> Ah, OK.
>> >>>>>
>> >>>>>
>> >>>>>> Dirk, thinking about this, the relation of the spectral uncertainty to
>> >>>>>> the bandwidth of the pulse would be interesting  ? can you refresh our
>> >>>>>> memory? I think the pulse was 10ms long (so, pretty long) but I forgot
>> >>>>>> how it looked like.
>> >>>>>>
>> >>>>>> Having slept about this a couple of times: What you really care about
>> >>>>>> is
>> >>>>>> that there's a *tone* for about 8ms < t < 12ms where there is none
>> >>>>>> else
>> >>>>>> (to rule out detection of spurs and other interference), is that
>> >>>>>> right?
>> >>>>>>
>> >>>>>> Maybe we've been focussing too much on filter-based detection. What if
>> >>>>>> we just *use* that feature of the signal being periodic for a duration
>> >>>>>> of t, and not else? A PLL would be able to "lock" on the tone (much
>> >>>>>> like
>> >>>>>> an FM Radio with a PLL will lock on the station's carrier), and if we
>> >>>>>> observe the phase error being limited for t, we can derive there was a
>> >>>>>> pulse.
>> >>>>>>
>> >>>>>> Andy, does that sound like a feasible approach?
>> >>>>>
>> >>>>> Yes.  With the added benefit of GNURadio's carrier tracking PLL 
>> >>>>> actually
>> >>>>> performing part of the correlation operation for you, if it works out.
>> >>>>>
>> >>>>> So use the carrier_tracking_pll block.  If it locks onto a good signal,
>> >>>>> it will mix it down to DC.  Then all you need is an averaging filter,
>> >>>>> like the single pole IIR filter block or the moving average filter
>> >>>>> block, operating on the real part of that output.  That will act as a
>> >>>>> lock detector, and, I think, also completes a correlation operation, if
>> >>>>> you use a non-normalized moving average filter (since the carrier
>> >>>>> tracking PLL block gives you a point by point complex multiply with the
>> >>>>> conjugate of the complex carrier tone).
>> >>>>>
>> >>>>> You'll want to set the PLL allowed min and max frequency bounds
>> >>>>> properly; be careful since the input arguments have tricky units.
>> >>>>>
>> >>>>> You'll also want the loop bandwidth set pretty wide, since you want to
>> >>>>> lock-in rapidly.
>> >>>>>
>> >>>>> Also, if I'm thinking about this properly:
>> >>>>> To get faster lock in, you may want your frequency range of interest to
>> >>>>> be somewhere in between Fs/4 and Fs/2; and not near DC.  If you're
>> >>>>> within the lock-in range of the PLL, you'll lock within 1 cycle.  If
>> >>>>> you're in the pull-in range of the PLL, it can take many cycles to get
>> >>>>> into lock.
>> >>>>>
>> >>>>> Regards,
>> >>>>> Andy
>> >>>>>
>> >>>>>> Cheers,
>> >>>>>>
>> >>>>>> Marcus
>> >>>>>>
>> >>>>>>
>> >>>>>> On 14.03.2017 14:18, Andy Walls wrote:
>> >>>>>> > If there is no information on the signal, why not just do a straight
>> >>>>>> > energy detector:
>> >>>>>> >
>> >>>>>> > source -> complex bandpass filter -> complex to mag -> burst/energy
>> >>>>>> detector -> QT Time sink (triggering on start of burst tag)
>> >>>>>> >
>> >>>>>> > The complex bandpass filter just has to catch all the frequencies of
>> >>>>>> > interest.  (A complex bandpass filter does not have a symmetrical
>> >>>>>> image
>> >>>>>> > around DC.)
>> >>>>>> >
>> >>>>>> > The burst/energy detector has to detect some minimum number of time
>> >>>>>> > domain samples contiguously above some noise/signal threshold that
>> >>>>>> you
>> >>>>>> > set, and tag the pulse.  It also has to detect when the time domain
>> >>>>>> > samples fall below the threshold.  The burst/energy detector can
>> >>>>>> work
>> >>>>>> > with a preset noise/signal threshold, or you could periodically
>> >>>>>> > re-estimate that threshold.
>> >>>>>> >
>> >>>>>> > GNURadio does not have a stock block that does burst/energy
>> >>>>>> detection,
>> >>>>>> > so you'll have to find one on the 'Net or write one yourself.
>> >>>>>> >
>> >>>>>> > -Andy
>> >>>>>> >
>> >>>>>> >
>> >>>>>> > Dirk Gorissen wrote:
>> >>>>>> >> Hi Martin,
>> >>>>>> >>
>> >>>>>> >> The aim is to check for the existence of a 10ms pulse in the
>> >>>>>> incoming
>> >>>>>> >> radio signal (from an sdr). The thing is I only know the frequency
>> >>>>>> of
>> >>>>>> >> the pulse to within ~1khz.
>> >>>>>> >>
>> >>>>>> >> So a single filter in my case is: generate a synthetic pulse
>> >>>>>> (complex
>> >>>>>> >> sin wave) of the same length and with a frequency f. Then take the
>> >>>>>> >> reverse of the complex conjugate of this pulse to give me the taps
>> >>>>>> for
>> >>>>>> >> a FIR filter.
>> >>>>>> >>
>> >>>>>> >> Repeat the above for each frequency I want to check. E.g., 10
>> >>>>>> filters
>> >>>>>> >> each 100Hz apart.
>> >>>>>> >> Then I just take the magnitude of each filter output and push
>> >>>>>> through
>> >>>>>> >> a Max to get my final correlations.
>> >>>>>> >>
>> >>>>>> >> I can come up with an algorithm that tries to be clever and with
>> >>>>>> some
>> >>>>>> >> accounting tries to find what frequency matters most but I wouldnt
>> >>>>>> be
>> >>>>>> >> surprised if there is some theory you can use to do this more
>> >>>>>> >> efficiently or even in a single shot. But this is where Im unsure.
>> >>>>>> >>
>> >>>>>> >> Cheers
>> >>>>>> >> Dirk
>> >>>>>> >>
>> >>>>>> >>
>> >>>>>> >> On 14 March 2017 at 01:09, Martin Braun <address@hidden> wrote:
>> >>>>>> >>> How related are those filters? Is this a candidate for polyphase
>> >>>>>> DSP?
>> >>>>>> >>>
>> >>>>>> >>> -- M
>> >>>>>> >>>
>> >>>>>> >>> On 03/11/2017 02:01 PM, Dirk Gorissen wrote:
>> >>>>>> >>>>  Hi Marcus,
>> >>>>>> >>>>
>> >>>>>> >>>>  Sorry, I should have clarified. You may recall an earlier thread
>> >>>>>> from
>> >>>>>> >>>>  mine where Im looking to pick out a short pulse from background
>> >>>>>> noise
>> >>>>>> >>>>  but I dont know the exact frequency of the pulse. Thought I
>> >>>>>> would
>> >>>>>> >>>>  start a new thread with a clear, specific question.
>> >>>>>> >>>>
>> >>>>>> >>>>  There is an uncertainty of +/- 1khz. So I can define multiple
>> >>>>>> filters,
>> >>>>>> >>>>  correlating for different pulse frequencies across the 1khz
>> >>>>>> range. I
>> >>>>>> >>>>  can implement this with different filters running in parallel
>> >>>>>> but I
>> >>>>>> >>>>  was looking for a more flexible / efficient way.
>> >>>>>> >>>>
>> >>>>>> >>>>  If you think this is out of scope for this list no problem at
>> >>>>>> all,
>> >>>>>> >>>>  just let me know.
>> >>>>>> >>>>
>> >>>>>> >>>>  Cheers
>> >>>>>> >>>>  Dirk
>> >>>>>> >>>>
>> >>>>>> >>>>> On 11 March 2017 at 20:02, Marcus M?ller <address@hidden> wrote:
>> >>>>>> >>>>>> Hi Dirk,
>> >>>>>> >>>>>>
>> >>>>>> >>>>>> this is more of a general DSP question than a GNU Radio
>> >>>>>> question:
>> >>>>>> >>>>>>
>> >>>>>> >>>>>> How do these filters relate to each other?
>> >>>>>> >>>>>>
>> >>>>>> >>>>>> My gut feeling is that this gets a lot easier as soon as you
>> >>>>>> tell us why
>> >>>>>> >>>>>> you're doing this, i.e. for what purpose :)
>> >>>>>> >>>>>>
>> >>>>>> >>>>>> Best regards,
>> >>>>>> >>>>>> Marcus
>> >>>>>> >>>>>>
>> >>>>>> >>>>>> On 11.03.2017 19:28, Dirk Gorissen wrote:
>> >>>>>> >>>>>>> Hello all,
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> Given a stream of samples I would like to apply n slightly
>> >>>>>> different
>> >>>>>> >>>>>>> filters to it with n being able to be chosen at runtime. Then
>> >>>>>> combine
>> >>>>>> >>>>>>> the results back to a single stream.
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> As a test I built a flowgraph with the following chains in
>> >>>>>> parallel for n
>> >>>>>> >>>>>>> = 6
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>>                 | ->  decimating fir filter 1 -> complex to
>> >>>>>> mag ->    |
>> >>>>>> >>>>>>> stream -> | ->  decimating fir filter 2 -> complex to mag ->
>> >>>>>> |  -> Max
>> >>>>>> >>>>>>> -> ...
>> >>>>>> >>>>>>>                 |                                  ....
>> >>>>>> >>>>>>>                         |
>> >>>>>> >>>>>>>                 | ->  decimating fir filter n -> complex to
>> >>>>>> mag ->    |
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> So the same stream is sent to each chain (decimation is 1) and
>> >>>>>> the
>> >>>>>> >>>>>>> output of each chain is pushed through a big Max block with 6
>> >>>>>> inputs.
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> This works but not particularly elegant and a bit annoying to
>> >>>>>> change
>> >>>>>> >>>>>>> if I suddenly decide I want to change n. In particular it also
>> >>>>>> does
>> >>>>>> >>>>>>> not seem computationally efficient.
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> What I would like is to replace the above by a single block
>> >>>>>> that
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> - replicates the input n times
>> >>>>>> >>>>>>> - applies each filter on each replica
>> >>>>>> >>>>>>> - combines the output again to a single stream
>> >>>>>> >>>>>>> - have a tunable n parameter
>> >>>>>> >>>>>>> - is fast
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> I did this with an Embedded python block doing essentially
>> >>>>>> this:
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> for i in range(n):
>> >>>>>> >>>>>>>      out[i] = scipy.signal.lfilter(taps[i], 1, input)
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> This is using exactly the same taps as in the chain case. This
>> >>>>>> works
>> >>>>>> >>>>>>> but the output is different and worse than what I get with the
>> >>>>>> >>>>>>> separate chains. As a test, instead of lfilter I tried:
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>>
>> >>>>>> gnuradio.filter.fir_filter_ccc(1,taps[i]).work(input[0],output)
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> Thinking perhaps that is a closer replica. But couldnt get it
>> >>>>>> to work..
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> I suspect there should be an easy / natural way of doing this
>> >>>>>> in
>> >>>>>> >>>>>>> gnuradio. Looked at the filter bank / channelliser blocks but
>> >>>>>> failed
>> >>>>>> >>>>>>> to get anywhere.
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> So what is the best way forward to do this?
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> Many thanks
>> >>>>>> >>>>>>> Dirk
>> >>>>>> >>>>>>>
>> >>>>>> >>>>>>> _______________________________________________
>> >>>>>> >>>>>>> Discuss-gnuradio mailing list
>> >>>>>> >>>>>>> address@hidden
>> >>>>>> >>>>>>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>> >>>>>> >>>>>>
>> >>>>>> >>>>>> _______________________________________________
>> >>>>>> >>>>>> Discuss-gnuradio mailing list
>> >>>>>> >>>>>> address@hidden
>> >>>>>> >>>>>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>> >>>>>> >>>>>
>> >>>>>> >>>>>
>> >>>>>> >>>>> --
>> >>>>>> >>>>> _________________________________________
>> >>>>>> >>>>> Dr. Dirk Gorissen
>> >>>>>> >>>>> Mob: +44-7763-806-809
>> >>>>>> >>>>> Web: dirkgorissen.com
>> >>>>>> >>>>> Skype: dirk.gorissen
>> >>>>>> >>>>> Twitter  : twitter.com/dirkgor
>> >>>>>> >>>>> LinkedIn: linkedin.com/in/dirkgorissen
>> >>>>>> >>>>
>> >>>>>> >>>>
>> >>>>>> >>>
>> >>>>>> >>> _______________________________________________
>> >>>>>> >>> Discuss-gnuradio mailing list
>> >>>>>> >>> address@hidden
>> >>>>>> >>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>> >>>>>> >>
>> >>>>>> >>
>> >>>>>> >> --
>> >>>>>> >> _________________________________________
>> >>>>>> >> Dr. Dirk Gorissen
>> >>>>>> >> Mob: +44-7763-806-809
>> >>>>>> >> Web: dirkgorissen.com
>> >>>>>> >> Skype: dirk.gorissen
>> >>>>>> >> Twitter  : twitter.com/dirkgor
>> >>>>>> >> LinkedIn: linkedin.com/in/dirkgorissen
>> >>>>>> >
>> >>>>>> >
>> >>>>>> > _______________________________________________
>> >>>>>> > Discuss-gnuradio mailing list
>> >>>>>> > address@hidden
>> >>>>>> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> _________________________________________
>> >>>> Dr. Dirk Gorissen
>> >>>> Mob: +44-7763-806-809
>> >>>> Web: dirkgorissen.com
>> >>>> Skype: dirk.gorissen
>> >>>> Twitter  : twitter.com/dirkgor
>> >>>> LinkedIn: linkedin.com/in/dirkgorissen
>> >>
>> >>
>> >>
>> >> --
>> >> _________________________________________
>> >> Dr. Dirk Gorissen
>> >> Mob: +44-7763-806-809
>> >> Web: dirkgorissen.com
>> >> Skype: dirk.gorissen
>> >> Twitter  : twitter.com/dirkgor
>> >> LinkedIn: linkedin.com/in/dirkgorissen
>> >
>> >
>> >
>> > --
>> > _________________________________________
>> > Dr. Dirk Gorissen
>> > Mob: +44-7763-806-809
>> > Web: dirkgorissen.com
>> > Skype: dirk.gorissen
>> > Twitter  : twitter.com/dirkgor
>> > LinkedIn: linkedin.com/in/dirkgorissen
>
>

Attachment: implant_pulse_detect.grc
Description: Binary data


reply via email to

[Prev in Thread] Current Thread [Next in Thread]