[Discuss-gnuradio] viterbi.c, integer overflow at cumulative metric

[Vaskelainen Vesa]

Hi all,

I noticed that in viterbi.c when input symbol pair quality is
good, branch metrics 'mets[ ]' maximum value is 512. In case of
errorless input symbols, this causes cumulative metric increase
by 512 for the best path for each decoded bit. In BUTTERFLY
macro, m0 and m1 are type of int, i.e. they can hold integers
less than 2^31, but since metric increases by 8x2^9=2^12 for
each output byte, this implies that after 2^31 / 2^12 = 2^19:th
byte occurs an integer overflow and decoder output gets messy.

This is simple to check, if you connect constant source of -1's
to ccsds27decoder and write output to a file and then run
hexdump of the file.

I corrected this by changing the type of m0, m1 and metric to
long long, when the integer overflow should not exist before
2^51:th byte. However, with periodic initialization of metric
the integer overflow of cumulative metric would be fully avoided.

Best regards,
Vesa Vaskelainen