[announcement] master: SWIG -> Pybind11 (breaks OOT interface)

[Marcus Müller]

Hello, most prolific SDR community to wander this wide globe,

TL;DR: brace yourself. If you're on master, we'll be breaking your OOT. 
You don't have to worry about anything if you're using GNU Radio 3.8.

We're close to merging the pybind branch into master – and that means 
we're improving things a lot.

Formerly, we used SWIG to generate Python bindings for GNU Radio's C++ 
classes – a decision made much more than a decade ago[1], when it still 
wasn't clear that Python would be the only language we support (SWIG can 
generate wrappers for *a lot* of languages).

In what it does, SWIG is pretty impressive: give it some C++ header, and 
it will generate the correct binding for python, including generating of 
some mapper types, transparent handling of pointers etc. And, we've 
really been using it a lot and in a lot of complex usage scenarios. I 
honestly think that, together with OpenCV and LLDB, we really might be 
the healthiest and largest project to use SWIG in such a wide way. I'm 
very thankful SWIG exists!

Now, having said that, SWIG was always kind of a pain to work into our 
source tree. The promise of "throwing in unmodified C++ headers, getting 
out perfect Python bindings", far too often, turned out to be "after 
massaging these headers quite a bit, then adding a bit of SWIG 
directives here and there in swig .i files that we have to keep updated 
whenever we change header structure"; and every time something broke our 
SWIG bindings, be it a bug on our side or an unexpected change in SWIG 
semantics, or be it that SWIG stubbornly refuses to use the include 
paths we specify on some platforms, we sunk literal working days into 
getting things fixed.

So, as awesome as SWIG's automatism are, using it on such a type-diverse 
code base, with such a platform-diverse user community, was kind of 
liable for a lot of time not spent on developing a better SDR framework, 
but fixing tooling.

Thus, Josh took it upon himself to get this sorted out – and replace 
SWIG with Pybind11. Now, Pybind11 is pretty different from SWIG. It's 
actually pure C++11 code, and just compiles with standard compilers. No 
extra tooling required.

It does come at the expense of you having to declare your interface 
yourself. That sounds like a regression, but honestly, there's been so 
much work put into making this automateable by Josh, for most blocks 
that don't do anything "special", you can just let the automatism take 
over. If that doesn't work, you'll typically get a compiler error, 
telling you exactly what's wrong. To illustrate, I'll attach the code 
generated to wrap a simple block below[2].

However, what that means is that we're also changing the way OOTs need 
to wrap their C++ to Python so that they can still can work with GNU 
Radio C++ objects wrapped to Python. gr_modtool has been updated to have 
new stubs, and the aforementioned automatism is available, too. Small 
problem: unlike compilation, the code generation does need an extra bit 
of tooling, pygccxml to be specific. This is really only necessary when 
you wrap a block that's not been wrapped before, not when compiling any 
existing module or the GNU Radio source tree.

In terms of compile time: according to my current level of overview, for 
proper multicore machines, building the many smaller C++ files tends to 
be a bit faster than first running SWIG to generate 100kLOC-files and 
then compile these monsters. For single- or dualcore machines, the 
opposite tends to be true. It's my duty to remind you that compiling on 
a RaspberryPi itself might get even more cumbersome than it currently 
is, and that you should prefer the cross-building toolchains of the 
operating system you're running on the embedded platform rather than 
building directly on it. Especially debian makes that pretty easy.

Best regards,
and the happiest of hacking,

Marcus

[1] 
https://lists.gnu.org/archive/html/discuss-gnuradio/2003-03/msg00135.html

[2]

py::class_<pub_sink,
        gr::sync_block,
        gr::block,
        gr::basic_block,
        std::shared_ptr<pub_sink>>(m, "pub_sink", D(pub_sink))
.def(py::init(&pub_sink::make),
        py::arg("itemsize"),
        py::arg("vlen"),
        py::arg("address"),
        py::arg("timeout") = 100,
        py::arg("pass_tags") = false,
        py::arg("hwm") = -1,
        py::arg("key") = "",
        D(pub_sink, make))
.def("last_endpoint", &pub_sink::last_endpoint, D(pub_sink, last_endpoint))
;