Re: [Discuss-gnuradio] costas ambiguity and correlate-and-sync block in qpsk

[Andy Walls]

On Sun, 2016-04-03 at 20:24 +0000, Landsman, Arik wrote:
> Hi Andy, 
> 
> So now that the Thrift issue is resolved (separate thread, thank you!
> ), had a few follow up questions on the actual corr_est implementation
> and placement of tags: 
> 
> >> ./Rx_syncd.py | grep -v volk | sed -e 's/(//g' -e 's/)//g' >
> foo.txt
> 
> this doesn't want to port the samples; it does however add a header
> with something to the likes of "Volk machine: sse4_1_64". Tried as
> sudo as well, still no go..


After you have GRC generate the Rx_syncd_(n).py script, you have to hand
edit it near the bottom to print out the samples of the preamble
generated my the modulate_vector block.  The edit looks something like
this:

    ...
    tb.start()
    tb.show()

    # Hand added statement:
    print tb.get_modulated_preamble_untrimmed()

    def quitting():
        tb.stop()
        tb.wait()
    ...

So when you run that edited script, it will emit, to stdout, the sample
vector created by the modulate_vector block in one of the flowgraphs
that I gave you.  (In the above specific example, the Rx_syncd_2.grc has
a modulate vector block that generates "modulate_preamble_untrimmed".) 


> 
> I also couldn't find setup_env.sh, searched the whole folder
> structure. so didn't run >>source <etc>. maybe that's part of the
> problem? 

Nope.  If you're running gnuradio-companion just fine without that
script and you don't know what it is, then you don't need that
environment script.

Here are the contents of mine, just so you feel better about ignoring
it:
        $ cat ../target/setup_env.sh 
        # WARNING: This file is auto-generated by pybombs, any manual changes 
to it may be overwritten!
        export 
PATH="/home/andy/work/target/bin/:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/home/andy/.local/bin:/home/andy/bin:/usr/lib64/qt4/bin/"
        export 
PYTHONPATH="/home/andy/work/target/python/:/home/andy/work/target/lib/python2.6/site-packages/:/home/andy/work/target/lib64/python2.6/site-packages/:/home/andy/work/target/lib/python2.6/dist-packages/:/home/andy/work/target/lib64/python2.6/dist-packages/:/home/andy/work/target/lib/python2.7/site-packages/:/home/andy/work/target/lib64/python2.7/site-packages/:/home/andy/work/target/lib/python2.7/dist-packages/:/home/andy/work/target/lib64/python2.7/dist-packages/:/home/andy/work/target/python/:/home/andy/work/target/lib/python2.6/site-packages/:/home/andy/work/target/lib64/python2.6/site-packages/:/home/andy/work/target/lib/python2.6/dist-packages/:/home/andy/work/target/lib64/python2.6/dist-packages/:/home/andy/work/target/lib/python2.7/site-packages/:/home/andy/work/target/lib64/python2.7/site-packages/:/home/andy/work/target/lib/python2.7/dist-packages/:/home/andy/work/target/lib64/python2.7/dist-packages/"
        export 
LD_LIBRARY_PATH="/home/andy/work/target/lib/:/home/andy/work/target/lib64/:"
        export 
PKG_CONFIG_PATH="/home/andy/work/target/lib/pkgconfig/:/home/andy/work/target/lib64/pkgconfig/:"

Super boring stuff that is probably already set for you, since you can
run GNURadio applications.  My GNURadio custom build is
in /home/andy/work/target/ .




> Any other way to send the raw samples into a file from GRC? otherwise
> going the python route.. )

Sure, just build a simple flowgraph with the modulator modulating six
consecutive instances of your preamble, streaming the complex I&Q floats
to a file sink.  You can then use MatLab, octave, Python, whatever to
parse the 32 bit float, native-endian, I,Q,I,Q,I,Q,... samples from the
file. 


> btw, played with tag delay manually, this has an immense effect on
> costas rotation once fading model is added to channel.

Well there are a number of things going on here, so let me state how
things are set up:

1. The reference sample sequence given to the corr_est block is your 8
symbol preamble, modulated at 10 samples/symbol, minus the first 5
samples of the first symbol, and minus the last 5 samples of the last
symbol.  So 150 samples total, spanning from preamble symbol 0.5 to
preamble symbol 7.5.

2. The corr_est block is performing a complex cross-correlation of this
reference sample sequence with the input signal.  The corr_est block
outputs that cross correlation, undelayed, on its second output.  The
corr_est block outputs the original input signal, *delayed* by the
length of your reference sample sequence, on its primary output.

3. This paper goes over some properties of a particular complex
cross-correlation that one can generalize somewhat:

Marple, Jr., S. L., "Estimating Group Delay and Phase Delay
via Discrete-Time 'Analytic' Cross-Correlation, _IEEE_Transcations_
_on_Signal_Processing_, Volume 47, No. 9, September 1999

But my  comment in the corr_est code says it all:

        // The analytic cross-correlation is:
        //
        // 2A*e_bb(t-t_d)*exp(-j*2*pi*f*(t-t_d) - j*phi_bb(t-t_d) - j*theta_c)
        //

        // The analytic auto-correlation's envelope, e_bb(), has its
        // peak at the "group delay" time, t = t_d.  The analytic
        // cross-correlation's center frequency phase shift, theta_c,
        // is determined from the argument of the analytic
        // cross-correlation at the "group delay" time, t = t_d.
        //
        // Taking the argument of the analytic cross-correlation at
        // any other time will include the baseband auto-correlation's
        // phase term, phi_bb(t-t_d), and a frequency dependent term
        // of the cross-correlation, which I don't believe maps simply
        // to expected symbol phase differences.

So the peak of the cross-correlation happens exactly at the delay time
when the reference sequence and the signal line up, which is at the end
of the preamble (or preamble symbol 7.5 as I have chosen things).  This
is the *only* instant, when we can learn learn the phase of the incoming
signal preamble's relative to the phase of the reference preamble at
that final sample.  The time dependent phase and frequency terms obscure
the value everywhere else, but that's OK, *if* we are frequency locked
and the phase offset introduced by the channel is relatively constant.

4. The polyphase clock sync block looks at the "time_est" tag emitted by
the corr_est block and expects that tag to be on the sample at the
center of a symbol - the optimal sampling time for clock recovery.  This
means, given the 150 sample reference sample sequence I used and the
tx_sps value of 10 and me starting and ending the reference sequence
mid-symbol, the sensible tag marking delays are either: 149, 139,
129, ...., 19, 9, or 0.  Other delay values will not resync clock
recovery properly, and make clock recovery take longer.  If you a not
locked in frequency, then really the best place to mark is at the
correlation peak, using a corr_est tag delay of 149, in our working
example.

5. The costas loop block looks for the "phase_est" tag emitted by the
corr_est block, and expects the tag at the place where the phase
tracking should be reset.  In PSK, it makes sense to mark this in the
middle of a symbol where the phase should take on a well defined value
very near a constellation point.  So again, the sensible tag marking
delays are either: 149, 139, 129, ...., 19, 9, or 0 in our working
example.  If you a not locked in frequency, then really the best place
to mark is at the correlation peak, the only place that phase offset
estimate is really valid, using a corr_est tag delay of 149, in our
working example.


If you have it noticed it so far, frequency lock to the carrier is
important if dealing with absolute phase values.  The costas loop will
do that fine frequency locking for you, as long as you are close with
your coarse frequency lock.

Your initial phase offset estimate, determined by cross-correlation with
a preamble, is happening before fine frequency locking and relies on the
matched filtering equivalence of the cross correlation to remove noise.
Without doing anything differently, you are probably stuck marking the
phase_est at the peak of the correlation (end of the preamble) and
nowhere else.

You ought to experiment with channel impairments one at a time to see
which one has the most effect, while still using a _realistic_ channel
model.  (Anyone can break a system with a drastically severe,
unrealistic channel.):

- Carrier frequency offset: constant, slow time-varying, fast
time-varying, etc.
- Carrier phase offset: constant, slow time-varying, fast time-varying,
etc.
- Rayleigh fading channel (no dominant LOS signal)
- Rician fading channel (one strong LOS)
- Narrow frequency selective fades in your signal bandwidth.

I suppose for extracting absolute phase offset, the Rayleigh fading,
fast varying CFO, and fast varying Phase Offset will be the worst.

Regards,
Andy

> Thanks,
> Arik 
> 
> 
> ______________________________________________________________________
> From: Andy Walls address@hidden
> Sent: Saturday, April 02, 2016 8:11 PM
> To: Landsman, Arik
> Cc: address@hidden
> Subject: Re: [Discuss-gnuradio] costas ambiguity and
> correlate-and-sync block in qpsk
> 
> 
> 
> Hi Arik:
> 
> On Sat, Apr 2, 2016 at 7:41 PM, Landsman, Arik
> <address@hidden> wrote:
>         Hi Andy,
>         
>         Still looking it over, but one thing jumps out right away -
>         for some reason I couldn't execute either Rx_syncd_2.grc or
>         the Rx_syncd.py. getting the same error in both cases:
>         
>         ""
>         Generating:
>         '/home/ubuntu/Desktop/MSProject/alternateVersions/Rx_syncd_2.py'
>         
>         Executing: '/usr/bin/python2
>         -u /home/ubuntu/Desktop/MSProject/alternateVersions/Rx_syncd_2.py'
>         
>         Using Volk machine: sse4_1_64
>         Thrift: Sat Apr  2 19:05:54 2016 TServerSocket::listen() BIND
>         9090
>         terminate called after throwing an instance of
>         'apache::thrift::transport::TTransportException'
>           what():  Could not bind: Transport endpoint is not connected
>         ""
>         
>         does this look familiar by any chance?
> 
> 
> It looks like a GNURadio ControlPort / Apache Thrift error.
> 
> 
> I took your flowgraph and modified it slightly, only adding a few
> blocks:
> 
> -modulate_vector
> 
> -skip_head
> 
> -keep_1_in_n
> 
> -qt_time_sink
> 
> -feedforward_agc
> 
> -multiply const
> 
> -qt gui range
> 
> 
> 
> One of those might be doing it.  Disable skip_head, keep_1_in_n, and
> the constellation sink they connect to.  Hopefully it's one of them.
> 
> Otherwise, you didn't drop a control port related block on the
> flowgraph did you?
> 
> 
> 
> FWIW, I also modified the Polyphase clock sync filter taps, the
> Corr_Est symbols, and changed a few 3.14's to math.pi.
> 
>  
>         
>         Also could you elaborate a bit on what you did with
>         
>         ""
>                 $ source <path_to_target>/setup_env.sh
>                 $ ./Rx_syncd.py | grep -v volk | sed -e 's/(//g' -e
>         's/)//g' > foo.txt
>                 $ octave
>                 octave:1> x = csvread('foo.txt');
>                 octave:2> t = [1:length(x)];
>                 octave:3> plot(t,abs(x))
>                 octave:4> plot(t,arg(x)/pi)
>         ""
>         
>         at the very least where can I find setup_env.sh?.
> 
> 
> If you build GNURadio with the PyBOMBS build system and install in a
> target directory somewhere, PyBOMBS generates this environment file
> which is essential to run GNURadio Companion and gnuradio scripts
> properly.  If you are using the GNURadio supplied by your
> distribution, then sourcing a setup_env.sh file is not necessary.
> 
> 
> 
>  
>         . wondering what <path_to_target> may be.
>         I suppose you then pipe a portion of the stream in Rx_syncd.py
>         into foo.txt to parse with octave?
> 
> 
> Yes, I just have it emit the complex samples vector that the
> modulate_vector block creates, and then inspect the samples in octave.
> You can do it in MatLab or Python or Excel, or whatever.
> 
>  
>         going to try it if I find setup_env.sh on my end. and once I
>         can run Rx_syncd.py... )
>         
>         Thanks,
>         Arik
> 
> 
> And another, "Oh by the way" for when things are working.  In the
> flowgraph I sent, I mark the phase correction on the last symbol of
> the preamble - where the phase correction was measured.  This means,
> that the previous symbols in the preamble, if they suffered a phase
> rotation, may not get corrected by the costas loop.  You can "fix"
> that by setting the corr_est tag marking delay to 0, to mark the first
> symbol in the preamble with the phase correction.  This is OK as long
> as you are very close to the correct center frequency.  The measured
> phase rotation will become less correct the farther you move from the
> measurement point at the peak of the correlation, if you don't have
> good frequency lock when performing the correlation. 
> 
> 
> 
> 
> Regards,
> 
> Andy
> 
>  
>         
>         
>         
>         ________________________________________
>         From: Andy Walls address@hidden
>         Sent: Friday, April 01, 2016 9:18 AM
>         To: Landsman, Arik
>         Cc: address@hidden
>         Subject: Re: [Discuss-gnuradio] costas ambiguity and
>         correlate-and-sync block in qpsk
>         
>         Hi Arik,
>         
>         I have attached a slightly improved flowgraph and a hand
>         edited .py
>         file.
>         
>         The only modifications to the flowgraph from before, is that I
>         set the
>         reference samples for the corr_est block to start at the
>         _middle_ of the
>         first symbol of the preamble and stop at the _middle_ of the
>         last symbol
>         in the preamble; I also ensured the phase_est and time_est
>         tags stayed
>         marked at the center of the last symbol of the preamble.  This
>         has two
>         benefits:
>         
>         1. The phase_est value will not be thrown off by the end of
>         symbol
>         transition to a different unknown symbol after the preamble.
>         2. The correlation won't be sensitive to the unknown symbols
>         before and
>         after the preamble.
>         
>         In other words, it was a mistake to have the correlation
>         reference
>         sequence start and end at the symbol transition times. :P
>         
>         In the generated *.py script I added one line near the end:
>         
>             print tb.get_modulated_preamble_untrimmed()
>         
>         So you can so this:
>         
>                 $ source <path_to_target>/setup_env.sh
>                 $ ./Rx_syncd.py | grep -v volk | sed -e 's/(//g' -e
>         's/)//g' > foo.txt
>                 $ octave
>                 octave:1> x = csvread('foo.txt');
>                 octave:2> t = [1:length(x)];
>                 octave:3> plot(t,abs(x))
>                 octave:4> plot(t,arg(x)/pi)
>         
>         and see why I picked the values I did to trim the modulated
>         preamble to
>         the correct set of samples.
>         
>         The modulator that generates the reference sequence has a
>         pretty long
>         delay, and the first copy of the preamble it outputs is a
>         little
>         distorted at the first symbol.  That's why I snipped out the
>         second copy
>         of the preamble.
>         
>         Regards,
>         Andy
>         
>         
>         On Thu, 2016-03-31 at 22:04 -0400, Andy Walls wrote:
>         > Hi Arik,
>         >
>         > Try the attached flowgraph.  I didn't get to put everything
>         back in, but
>         > it demonstrates the principal.
>         >
>         > 1. Observe the corr_est block correlates to your preamble
>         bytes.
>         > 2. Slide the phase offset around:
>         > a. observe the phase changing/wrapping in the time plot of
>         the output of
>         > the correlator
>         > b. observe the phase_est tag value approximately matches the
>         value of
>         > the phase offset slider.
>         > c. Observe the constellation rotate as you change the phase
>         offset.
>         > d. Observe your decoded bits stay the same.
>         >
>         > Regards,
>         > Andy
>         >
>         >
>         >
>         
>