Re: [Discuss-gnuradio] state of offline signal analysis tools in 2014?

[Peter A. Bigot]

Thanks for the recommendations.

I should clarify that I am a software engineer, not a signals engineer,
and my recurring need to visualize time-series data is often satisfied
without having to invoke DSP.  An example of the sort of thing I
frequently want to do is to interactively explore data collected from
multiple wireless sensors (e.g., six months of temperature and humidity
data collected at 1-minute intervals).  Technically signals, but simple
shifts, scales, and basic windowed statistics are more useful than FFTs
and complex filters.

Sometimes I do need DSP techniques to extract the data from third-party
wireless transmissions, hence my current dabbling with GNU Radio, but
it's the data not the extraction process that's the primary focus.  For
example, the reason I'm using GNU Radio is my need to demodulate packets
from a WS-2080 Weather Station and some other 433 MHz OOK sensors.  Yes,
I've tried rtl433 and rtlsdr-433m-sensor; the signal I'm capturing is
too noisy or I don't have frequency/bandwidth/filter settings right.
The specific use case that introduced my question about analysis tools
is my desire to interactively identify regions of interest from wideband
captures and then re-play that data repeatedly through various
processing chains, tweaking parameters until I get something that
reliably produces the underlying data.

@mossman: The GSoC project is very close to what I'd want for
DSP-oriented analysis.  If it existed it'd probably handle the
motivating example above.  It's too domain-specific for generalized
time-series visualization, though, so I'd still have an unmet need.

@marcus.mueller: Thanks for the details.  My experience is that a
metered stream/dataflow-oriented architecture is simply unsuited to the
sort of offline analysis I'm trying to do.  The ability to jump forwards
and backwards in time is crucial, as is the ability to decouple the
signal rate from the data processing rate.  Example: Qt Time Raster
schedules its updates based on wall clock not sample time, so speeding
up or slowing down the rate of data through the system changes the
displayed images, and running unthrottled drops all the information.

@dan.cajacob: pandas reminds me a lot of R.  The intro video showed it'd
be good for CLI-based data manipulation, but my initial need is to
explore data graphically.

@mdammer: kst-plot's web site shows some promising graphs, and it built
cleanly (though it doesn't obey CMAKE_INSTALL_PREFIX), but I've been
unable to locate examples and documentation that would allow me to
evaluate its capabilities quickly.

@madengr: Lack of source code for baudline and its focus on DSP knocks
it out of contention for my general needs.

On 07/16/2014 09:52 AM, Peter A. Bigot wrote:
> The sort of capabilities I'm looking for include: Read time-series 
> data from files of different formats (some too large to fit in 
> physical memory).  Display the data, optionally applying linear 
> transformations. Interactively pan and zoom.  Jump forwards and 
> backwards among time-registered events.  Enable/disable/time-shift 
> data overlays. Export selected data to new files.  Calculate and 
> display statistics and other non-linear transformations of selected data.

A rough course towards the tool I imagine would be:

* Collect/develop a suite of Qt/C++ widgets for graphical data display
  and manipulation that are not sensitive to the processing rate, don't
  have a unidirectional concept of time, can be accessed from Python,
  and can be combined to build something with the capabilities listed
  above.

* Use a command-line interface like pandas/R to display original files,
  extract regions of interest, apply transformations, and repeat until
  satori.

* Use GNU Radio Companion to glue components together to form
  domain-specific analysis applications.

That's a lot of yak shaving just to get a reliable OOK packet extractor,
so it probably won't happen.

Thanks again for the suggestions.

Peter