Re: [Discuss-gnuradio] comments on stream tags and metadata storage

[Tom Rondeau]

Alright, guys, I think we're officially done with this thread!

Seriously, while there's been good conversation here, there's too much going on to make any sane judgement on technical arguments. Let's move any discussion over to the issues tracker where Peter set them up and discuss there.

Thanks!

Tom

On Sat, Jul 26, 2014 at 8:22 AM, Marcus Müller <address@hidden> wrote:

Hi Peter,

On 26.07.2014 13:25, Peter A. Bigot wrote:
> I'm confused: are we to discuss these issues on the mailing list, or as
> comments on the wiki issues I created?  I thought the latter was the
> right location.  Putting detailed discussion in two places that do not
> link to each other is not a good approach.

While I agree, I think the bug tracker might in the first place not have
been the right choice for "architectural ideas", which fit better here.

>> That's a good idea, and actually, standarizing the way stream tags and
>> messages should be named and structured was one of the points of the
>> last dev call, and is thus work to be done.
>
> I've updated the issue with my proposed approach.  If this is already
> being worked and people have an alternative architecture in mind, then
> please comment on that issue so I stop thinking about it.

Um, that's being worked on, but please don't take it personal when I say
that I think the guys that have been using tags in the GNU Radio project
should be the ones to agree on a way to standarize things, and do that
at their own pace; I'm pretty sure your input will be valued, though!
There have been things in discussion, and I guess there will be some
communication regarding this soon, so I think it's safe to say it
doesn't hurt if you don't spent too much time on this.

>
> I do not see how this allows me to guarantee association of a tag with a
> specific absolute sample.

That's always guaranteed - a Tag is a (offset, PMT) tuple.

> I don't think we disagree on this behavior, only on how to ensure that
> the original insertion doesn't get lost or misordered (and whether
> that's a problem for which the infrastructure should take some
> responsibility).

I agree on disagreeing on whether this needs to be dealt with by the
infrastructure enforcing this. Especially on the "introducing a complex
restrictionary functionality just to stop people from doing logically
wrong things".

Basically, we're giving people vectors of void* to write samples to -- I
think our users do pretty well at not writing to arbitrary memory
locations, so we should trust ourselves not to produce tags for samples
that happened in the past. If there is reason to do so, let them do so.
I'm pretty confident that programmers that use tags are aware of the
fact that the samples flow through their flow graph and that downstream
blocks don't care about items that they already processed, and that they
can't magically take into account tags you add while they are processing
the samples that these tags are related to.

> If there is information that is conveyed by receiving tags in a
> non-monotonic offset order, I'd like to have more details.

I don't think there is. But I can't rule it out, either. So it's a
undecided, but applications might be relying on current behaviour.

>
> If the concern is that having the infrastructure maintain the tag order
> I propose at http://gnuradio.org/redmine/issues/701 will introduce a
> performance bottleneck, I'd like to see an argument. Preferably as a
> comment on that issue.

sorry to break your preference, but sorted inserting means that instead
of constant time insertion you get time dependent on amount of tags, no
arguing that, unless you find a better insertion algorithm for a sorted
list ;)

>
> Otherwise, since I feel that a deterministic non-decreasing order of
> arrival of tags simplifies implementing blocks that process those tags,

agree on the simplifying aspect.

> if you feel that every block that might take advantage of ordered
> arrival should instead enforce it in their work() functions then we do
> indeed disagree.

So, indeed, we disagree.

>> Also, I don't understand the bullet points at the end of your ticket
>> #701.
>> * preserve order of insertion: GR does this, so how is this an issue?
> You missed the "within the same offset".  Yes, I really do want
> downstream blocks to be guaranteed that the tags they receive are sorted
> by non-decreasing offset.  No, I don't want to make every block that
> inserts or receives tags have to do the sorting.

GR does this, it's just appending to a deque. How would that mix up the
order later on? If you do a std::stable_sort, the original order will
also be kept. One more reason to let the user do this himself: He can
decide whether stable_sorted, sorted or unsorted tags suffice.

>> * never discard a tag at the block that inserted it: Why would someone
>> discard a tag he inserted it? GR doesn't discard tags, so how is this an
>> issue?
>
> The implementation of file_metadata_sink discards tags, as described.

file_metadata_sink is not infrastructure, so I think we're mixing up
subjects again. I was hoping you'd pack all metadata sink-related issues
into one bug report and keep the discussion purely on the GNU Radio
framework as is.

> This suggests that GNU Radio as a holistic system does not have an
> architectural requirement that metadata be preserved. I'd like to see
> that resolution adopted.

Metadata is preserved within the running framework. Ok, storage seems to
break this in some minor aspects, but I fail to see how that's affecting
the way GNU Radio applications need to be designed.

>
> From the rest of your mail I get the impression that our expectations of
> what the framework should provide versus what individual component
> developers are responsible for handling are at odds.

Me too :)

> If the consensus
> of the GNU Radio development team is that the solution to all this is to
> clarify and document the existing behavior with no framework changes at
> all, then most of my issues can be closed once that's done.

Wow, no, this is not a consensus. You got replies of four people so far,
and I think Martin, Tom and I made it clear how we feel, as persons,
about this, and that we all hope this discussion yields approaches that
can be discussed :)

> Ten years
> experience with systems named X-Midas and Midas 2k, which do much of
> what GNU Radio does, leads me to believe that would have significant
> long term costs in development effort and system reliability.

Peter, please don't feel like your input isn't valued! This discussion
has shown that at least you and I disagree on what should be enforced by
a framework. GNU Radio is constantly changing, and thus, we should
definitely look at the way tags are handled today for the GNU Radio of
tomorrow.

I think much of our time went into analysis of each other's mails;
that's a bad thing, and I'd love to blame you for mixing in the metadata
filesink so casually that half of the infrastructure points are mixed up
with filesink issues, but that wouldn't honor the fact that I just went
along with it and didn't as you to consider the filesink and the tags
infrastructure as separate things.

I got the feeling that this is exactly the discussion you were hoping
for: Less on a "this is a bug, needs fixin'" level, more on a "what are
the pros and cons of (not) doing A and B" level. Now you have that
discussion, and I disagree with you, shouldn't that encourage you?

I really do think the "unordered tag retrieval" thing is something that
deserves discussion. For how things are now, my opinion is "let the user
deal with that". For how things can be with 3.8 or even later versions,
this is something we should put attention too, and that's why everyone
is encouraging you to partake in the hangout.

>From my side, I really strongly dislike the way get_tags_in_range works:
iterating through *all* tags, comparing each offset, filling the result
vector with the matching tags, each time the function is called.
Corollary, I'd like to use internally ordered tag storage, assuming that
insertion happens less often than extraction, this should yield an
enormous speed gain, but it would break GNU Radio as it is now.

However, many times in this discussion, I've got the feeling that you
liked to mix up things that are *wrong* now (which is the filesink
stuff, mainly) with things that you architecturally disagree with (if
tags should be ordered on retrieval) with things that would need
attention if things were different than they are, with assumptions you
made based on Sean's email (which wasn't wrong, by the way, but all the
"there's potential for a race condition" discussion really lacked all
background, if you read the code; I don't see the pruning Sean mentions,
though, so maybe I should ask Sean where that happens -- most likely
while tag propagation over buffers). Maybe that's a reason for a slight
frustration on my side when you mention your 10 years of experience -- I
don't have that, but I feel like we were trying to engage in a fruitful
discussion, and yet you pull the "I've seen things"-Joker. That doesn't
work well when you're actively discussing architecture with people of
years of experience with that exact framework you are looking at, and
these people nevertheless encourage you to specify your concerns, file
bugs, take part in community meetings etc.

Greetings,
Marcus

_______________________________________________
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio