Re: [Discuss-gnuradio] intermittent pulse detection

[Dirk Gorissen]

Hi Marcus,

Thanks again. And yes, Id be happy to do a writeup if I get things
working with GnuRadio. I did a writeup of the first version I did of
my project (*) and happy to do a part two of the improved version when
finished (asap). Replies inline.

>Well, if you have a let's say 2 kHz uncertainty in the frequency of your 
>pulse, I'd really start very simple. Looking at your plots, I think we can 
>sufficiently suppress the noise simply by low-pass filtering:

Just note that the plots are posted are with the transmitter pretty
much sitting next to the antenna so this is a *very* strong signal, in
reality its much weaker.

>* Use a Low Pass filter (real taps) with a cutoff of 1 kHz – that will let 
>everything from -1 kHz to +1 kHz around your center frequency through. You can 
>use the "freq xlating FIR" block to first shift by a given frequency and >then 
>apply the filter; might make things easier.
>* Detect power by putting your complex samples through a complex mag^2 block.

Thanks. I did this and the following observations
- I went SDR -> Freq Xlating FIR filter with 4x decimation -> low pass
filter block with 128x decimation and 1khz cutoff -> complex to mag2

- The Freq Xlating block was giving me an error unless I filled in
something for the Tap, some googleing led me to fill in
firdes.low_pass_2 and I just used the same filter settings as the
filter block. That made it happy but to be honest Im not sure why and
if that was the right thing to do.

- I did the decimation somewhat arbitrarily in two steps as I read
somewhere that is better computationally, perhaps it should be more
even though

- With the above I could see the peaks (yay!) but it gets harder as
the signal gets weaker. I plugged in the peak detector block (peak
detector 2 seems broken btw) but found its usage very unintuitive and
I didnt get it to mark the peaks in the way I wanted. Even though they
were pretty big it wouldn't catch many of them, but perhaps its a
plotting refresh rate thing. Its not clear to me over what window the
average is taken (I was expecting that as an option).

- Also, Im not 100% sure what sample rate I should be setting (or how
to count number of samples (e.g., peak look ahead)) in blocks that are
downstream from decimation. If the source sample rate is 1msps, and I
have decimated by 4x, say, that means I have 4x less samples coming
through per second so I should be setting 0.250msps as the sample rate
in downstream blocks (filters, display, fft, ...) right?

>If the output of that looks somewhat OK, then you could e.g. low-pass filter 
>the result to get rid of noise pulses that are too short, for example :)

Ok, so low-pass filter the real valued mag^2 values. I tried this but
was not obvious how to choose the right cutoff frequency & transition
width other than just trial and error. Also, as the signal gets weaker
the peaks get narrower.

I still feel like I should be cross correlating with the known signal,
or some kind of matched filter setup, or exploiting the regularity of
the pulse somehow to give me more sensitivity, but unsure how best to
proceed there. I stumbled across a post about the correlate_and_sync
block. That seemed somewhat similar to my problem (though I guess I
have no preamble) but raised more questions than it solved.

>By the way, it might be cool to put up a short recording (ie. a file generated 
>using the GNU Radio file sink) somewhere (zipped, e.g. on Google Drive). I 
>haven't worked with that myself, but maybe putting it on
>SigIDwiki[1] might be beneficial for people encountering the wildlife tracker 
>signal later on.

Good idea. I quickly captured a file with raw IQ samples (File sink)
in case anybody is interested. It starts with the transmitter very
close to antenna, moves progressively further until out of range and
then back. Its only about a minute or two tops but at 1msps it ended
up as 813 MB though.

https://drive.google.com/drive/folders/0B5dKo9igl8W4dWpiMmlEYWx3b0k?usp=sharing

>We surely could use a few more "oh, GNU Radio is awesome because it works for 
>everyone" articles (like [2]), but what we far more severely lack is stories 
>of people that actually made progress, but also struggled,
>because they didn't know GR for years before writing a blog post. One of the 
>problems that we face (and that was a very prominent theme at the Panel at 
>FOSDEM [3]) is that we don't really know how to help people
>that are just approaching GNU Radio, and aren't already familiar with a lot of 
>the stuff that we've gotten used to.

Happy to do that. Generally I have found companion very robust and
works well. I love the python export/integration. Im just slightly
concerned about performance on a low spec machine but Ill cross that
bridge when I get to it. Main thing for me is that I find many
explanations for blocks very concise and without enough radio
background you feel a bit lost as to what exactly to use when or what
the options mean. Also, silly question but I am still confused as to
what a "tap" is. I googled around the docs hoping to find a page "what
is a tap" but failed.

Thanks for the links.

Many thanks again,

Dirk


(*) 
https://dirkgorissen.com/2016/04/19/wheres-susi-airborne-orangutan-tracking-with-python-and-react-js/

On 7 February 2017 at 14:22, Marcus Müller <address@hidden> wrote:
> Hi Dirk,
>
> Well, if you have a let's say 2 kHz uncertainty in the frequency of your
> pulse, I'd really start very simple. Looking at your plots, I think we can
> sufficiently suppress the noise simply by low-pass filtering:
>
> * Use a Low Pass filter (real taps) with a cutoff of 1 kHz – that will let
> everything from -1 kHz to +1 kHz around your center frequency through. You
> can use the "freq xlating FIR" block to first shift by a given frequency and
> then apply the filter; might make things easier.
> * Detect power by putting your complex samples through a complex mag^2
> block.
>
> If the output of that looks somewhat OK, then you could e.g. low-pass filter
> the result to get rid of noise pulses that are too short, for example :)
>
> By the way, it might be cool to put up a short recording (ie. a file
> generated using the GNU Radio file sink) somewhere (zipped, e.g. on Google
> Drive). I haven't worked with that myself, but maybe putting it on
> SigIDwiki[1] might be beneficial for people encountering the wildlife
> tracker signal later on.
>
> I actually think your project is pretty cool – short warning: if this works
> out, I'll try to convince you to write a short article for the GNU Radio
> blog!
> I'd imagine it being about how you approached the problem and what you've
> found, with emphasis on all the problems you had because of GNU Radio and
> maybe also DSP. We surely could use a few more "oh, GNU Radio is awesome
> because it works for everyone" articles (like [2]), but what we far more
> severely lack is stories of people that actually made progress, but also
> struggled, because they didn't know GR for years before writing a blog post.
> One of the problems that we face (and that was a very prominent theme at the
> Panel at FOSDEM [3]) is that we don't really know how to help people that
> are just approaching GNU Radio, and aren't already familiar with a lot of
> the stuff that we've gotten used to.
>
> Happy tracking,
>
> Marcus
>
> [1] http://www.sigidwiki.com/wiki/Adding_a_Signal_Entry
> [2]
> http://gnuradio.org/blog/filtering-time-series-data__elemental-building-blocks/
> [3] https://fosdem.org/2017/schedule/event/sdr_panel/ ; don't know – maybe
> we'll have recordings of that.
>
>
> On 02/06/2017 09:39 PM, Dirk Gorissen wrote:
>
> Thanks Marcus & Martin for the responses.
>
> To clarify, Im working on a wildlife tracking problem but from the air
> (drone). Im purely interested in finding out if the pulse (which gets
> transmitted at a fixed interval of 1500ms) occurred or not. If it did, I
> know Im within some range of the animal of interest. I have no control over
> the transmitter and no further technical details (unfortunately). I did take
> some screenshots to give you an idea what it looks like:
>
> https://goo.gl/photos/Y3Ea6kJo1ewrcDYM8  (taken with tx at close range to
> static antenna)
>
> I expect quite a bit of (fairly uniform) background noise from other
> electronics to deal with and there are hardware things that can be improved.
> But for the purposes of this thread I want to ensure Im doing the right
> things on the DSP side.
>
> And yes, Im pretty much a DSP novice but happy to learn. So be gentle :)
>
>>So, my understanding is that your SDR device first downconverts your 150.22
>>MHz signal to complex baseband, is that right?
>
> Yes, so think something like an rtlsdr or airspy.
>
> About the FFT question, thanks I got a bit confused. I now understand that
> if I want FFT over a longer timeframe I should just take a larger size and I
> can vary the sampling rate (e.g., via decimation) to get the resolution (Hz
> per bin) I require.
>
> Im now wondering if there is a founded way to pick an optimal FFT size given
> my pulse is only 10ms long. Im guessing it should not be much longer than
> the 10ms equivalent but maybe there is a noise tradeoff (?).
>
> In any case I agree this is a time based phenomenon so a time domain
> approach should likely be the main one.
>
> So it seems I should at least be trying an FIR filter to do cross
> correlation. Not quite clear how I would actually do that in practise with
> the gr block though. How would I provide the second input (the synthetic
> pulse)?
>
> I have come across matched filters before but so far failed to bridge the
> gap from theory to code. This is partly the reason I came to gnuradio.
>
>>You can implement a limited-length autocorrelation/crosscorrelation
>> relatively
>>easily; we can talk about how that would look like, but my gut feeling is
>> that this
>>might be something that might not make too much sense in your specific use
>>case.
>
> Not too sure myself tbh, but happy to take your lead.
>
>>it's possible cyclostationary estimation methods might be helpful here.
>
> So I googled this and poked around a bit on https://cyclostationary.blog.
> Looks really interesting but out of my depth here..
>
> Thanks for the tip on gr-fosphor.
>
> Cheers
> Dirk
>
> On 5 February 2017 at 12:06, Marcus Müller <address@hidden> wrote:
>>
>> Hi Dirk,
>>
>> nice to have you around, welcome to GNU Radio! I don't know your level of
>> DSP knowledge, so please excuse if I either throw too many high-level
>> concepts at you or assume you could want to read up on something that you
>> already know. If something in my reply is unclear, please don't hesitate to
>> ask for clarification.
>>
>> So, my understanding is that your SDR device first downconverts your
>> 150.22 MHz signal to complex baseband, is that right?
>>
>> So, what is the objective here? You say you need to /detect/ the pulse,
>> but for what purpose? Is it just about detecting the presence of the pulse,
>> or is it used for eg. time detection?
>>
>> Your filter->decimate approach sounds very reasonable; I'm not 100%
>> convinced by using the FFT for something that is concentrated in *time*
>> domain, but that might depend on the purpose mentioned above.
>>
>> 2) Im somewhat confused about the FFT block if I just pipe the SDR
>> straight into it. The FFT size is set to 1024 and the window is set to
>> "window.blackmanharris(1024)". So Im assuming the FFT just applies one
>> window (?) and outputs 1024 bins. However, how many samples are
>> accumulated before the FFT is run? I would have assumed I can control
>> that too. And if so, should I best be doing this every 50ms, 500ms,
>> 2000ms, ..?
>>
>> I'm not sure I understand the question. An FFT is simply an DFT. It's a
>> mapping of N-dimensional vector to N-dimensional vector, . The window is
>> multiplied point-wise with the input vector prior to the DFT to avoid
>> spectral leakage.
>>
>> There's no accumulation involved anywhere.
>>
>> 3) I can use the rational resampling block to bring the sample rate
>> down to 48khz so I can use the audio sink. From that I can still hear
>> the pulse even if it is not visible in the spectrum (gui sink). Im
>> assuming this is just because the plotting cannot keep up?
>>
>> Maybe, or maybe the spectrum sink really isn't the right tool to visualize
>> a pulse! Again, we might want to discuss what this pulse is and what it's
>> used for.
>>
>> 4) In the time domain I guess I can generate a synthetic pulse of the
>> same length / frequency and then cross correlate.
>>
>> Hm, but correlating a signal with a known fixed sequence is,
>> mathematically, a convolution.
>>
>> That is identical to doing FIR filtering – in fact, if we consider the
>> shape of your pulse as what is often referred to as pulse shape filter, then
>> that filter in the receiver would simply be the matched filter to that.
>>
>> Not obvious to me
>> how to generate the required pulse in gnuradio though (would a
>> continuous signal work?).
>>
>> "Continuous" would mean you'd do an infinite-length correlation, so that's
>> not 100% possible.
>>
>> I also notice there are no built in
>> (auto)correlation blocks?
>>
>> Hm, but an autocorrelation would take a complete signal, shift it by all
>> possible shifts and calculating the dot product between the shifted version
>> and the unshifted, right? That would require to have the complete signal at
>> once.
>>
>> But GNU Radio is a streaming architecture, so that can't work.
>>
>> You can implement a limited-length autocorrelation/crosscorrelation
>> relatively easily; we can talk about how that would look like, but my gut
>> feeling is that this might be something that might not make too much sense
>> in your specific use case.
>>
>> I found the "correlation estimator" but not
>> clear how to use it. As for dealing with the frequency uncertainty
>> problem. Does one just try correlating with different freuencies and
>> pick the best one? Or what is the good thing to do here given I may
>> also have to deal with quite a bit of noise.
>>
>> As a gut feeling: you don't really care about whether the pulse is exactly
>> at a certain frequency (it's absolutely not normal for wireless receivers to
>> know the exact frequency a priori), but when it happens. So we might want to
>> discuss the kind of pulse, and kind of noise we're talking about.
>>
>> As a further gut feeling: I think your autocorrelation question indicates
>> you might be on a very good track – it's possible cyclostationary estimation
>> methods might be helpful here.
>>
>>
>> Best regards,
>>
>> Marcus
>>
>>
>>
>> On 02/04/2017 11:39 PM, Dirk Gorissen wrote:
>>
>> Fist of all, while Im a newbie to (gnu)radio, congrats to the dev team
>> for a great piece of software.
>>
>> My question is about the need to detect a weak, noisy, short (10ms)
>> pulse that occurs every 1.5 seconds. It is transmitted at a particular
>> frequency (e.g., 150.22 MHz) but in practise I have found this can
>> vary by as much as +/- 500Hz. There is no modulation, it is simply an
>> on/off keyed pulse.
>>
>> Say I have an SDR generating data at 2.5 MSPS. I have so far been
>> experimenting with standard scipy/numpy routines to collect a batch of
>> samples, perform a FFT (freq domain) and do a cross correlation (time
>> domain). However, Im by no means a dsp guy and would like to leverage
>> gnuradio as much as I can. I have been poking a bit but have some
>> basic questions.
>>
>> 1) 2.5 Msps gives me way more bandwidth than I neeed. Assuming, for
>> now, I only care about a single pulse frequency I really only need
>> ~1khz bandwidth. In the frequency domain I can directly decimate down
>> (with a big factor) to the 1-2 khz range using the low pass filter
>> block, do an fft, and look for peaks. Is that the right approach?
>>
>> 2) Im somewhat confused about the FFT block if I just pipe the SDR
>> straight into it. The FFT size is set to 1024 and the window is set to
>> "window.blackmanharris(1024)". So Im assuming the FFT just applies one
>> window (?) and outputs 1024 bins. However, how many samples are
>> accumulated before the FFT is run? I would have assumed I can control
>> that too. And if so, should I best be doing this every 50ms, 500ms,
>> 2000ms, ..?
>>
>> 3) I can use the rational resampling block to bring the sample rate
>> down to 48khz so I can use the audio sink. From that I can still hear
>> the pulse even if it is not visible in the spectrum (gui sink). Im
>> assuming this is just because the plotting cannot keep up?
>>
>> 4) In the time domain I guess I can generate a synthetic pulse of the
>> same length / frequency and then cross correlate. Not obvious to me
>> how to generate the required pulse in gnuradio though (would a
>> continuous signal work?). I also notice there are no built in
>> (auto)correlation blocks? I found the "correlation estimator" but not
>> clear how to use it. As for dealing with the frequency uncertainty
>> problem. Does one just try correlating with different freuencies and
>> pick the best one? Or what is the good thing to do here given I may
>> also have to deal with quite a bit of noise.
>>
>> Any guidance appreciated.
>>
>> Many thanks,
>>
>> Dirk
>>
>> _______________________________________________
>> Discuss-gnuradio mailing list
>> address@hidden
>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>
>> _______________________________________________ Discuss-gnuradio mailing
>> list address@hidden
>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>
> --
> _________________________________________ Dr. Dirk Gorissen Mob:
> +44-7763-806-809 Web: dirkgorissen.com Skype: dirk.gorissen Twitter  :
> twitter.com/dirkgor LinkedIn: linkedin.com/in/dirkgorissen



-- 
_________________________________________
Dr. Dirk Gorissen
Mob: +44-7763-806-809
Web: dirkgorissen.com
Skype: dirk.gorissen
Twitter  : twitter.com/dirkgor
LinkedIn: linkedin.com/in/dirkgorissen