Re: [Discuss-gnuradio] Streaming IQ File Compression

[Dave NotTelling]

Marcus,

     Thanks for the idea anyway :)  It sounded like a really neat outside the box idea.

I ran some quick tests with FLAC and found that it does a really good job compressing a constant filtered gaussian noise output.  I just created a flow graph with:

     fast noise source [float] -> low pass filter [float] -> float to short -> file sink

The filter was just there so that the spectrum wasn't completely flat.  Running the signal through FLAC with little endian, 16 bit, single channel resulted in a 2GB file ending up at ~ 400 MB.  But, that was just a constant signal.  I need to throw some 900 MHz captures through to see how that works.

I also found someone who modified FLAC to be multithreaded [1] as well as a CUDA implementation (windows only) [2]

[1] https://www.akalin.com/parallelizing-flac-encoding

[2] http://www.cuetools.net/wiki/FLACCL

On Sun, Jul 17, 2016 at 7:49 AM, Marcus Müller <address@hidden> wrote:

Well, having wasted a couple of hours of sleep time on this, maybe you shouldn't do the logarithmic storage... at its core its the same idea as storing floating point, but with a fixed mantissa.

The mathematical effects of the "rounding to the nearest exponential step" on superimposed sinusoids of different amplitude are … funky.

Best regards,

Marcus

On 16.07.2016 21:57, Juha Vierinen wrote:

Can you reduce the number of bits that you are using?

With radar signals, the receiver noise most of the time excites only about 8 bits out of 16. Ground clutter or meteor echoes excite nearly all of the bits occasionally, so I can't just truncate to 8 bits. In this case, bzip2 actually does a pretty good job of getting rid of the 8/16 most significant bits that are zero most of the time. Thus, I get a compression ratio close to 50% when using sc16. pbzip2 is a good tool for doing parallel compression on files. 

juha

On Sat, Jul 16, 2016 at 3:59 AM, Dave NotTelling <address@hidden> wrote:

Anyone have experience with streaming 100+ MSPS of 16 bit IQ data through a compression tool and then to disk?  I'd like to be able to take really wide band snaps for several minutes.  Currently that would take up 16 * 2 * SAMP_RATE bits per second.  So, for 200 MSPS that would end up being 800 MBytes/s.  That rate eats up a hard drive pretty quickly.  Running it through gzip by way of pigz was my first thought, but even on an 8 core Intel machine pigz just can't keep up.  I can sustain maybe 300 MBytes/s but that's it.  And I know it's not a hard disk limitation as the M.2 drive I am using can easily sustain 800 MBytes/s of uncompressed data.

Thoughts?

Thank you!

-Dave

_______________________________________________
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

_______________________________________________
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

_______________________________________________
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio