Re: Proposed interface changes for "parse.error custom"

[Akim Demaille]

Adrian,

> Le 26 févr. 2020 à 23:58, Adrian Vogelsgesang <address@hidden> a écrit :
> 
> regarding our discussion on memory management for yyexpected_tokens
> --------------------------------------------------------------------------------------------------
> 
> 
>> If your use case in incrementally proposing possible keywords, then you want 
>> them all, and a max of the possible number of tokens is available 
>> statically: YYNTOKENS.
> You are right, I would want to retrieve the whole list of expected tokens.
> 
>> I don't see where this would be useful.
> 
> The code I am having in mind to do so would be
>    int onstack_tokens[10];
>    std::vector<int> heap_tokens;
>    int cnt_tokens = context.yyexpected_tokens(onstack_tokens, 10);
>    if (cnt_tokens > 10) {
>       heap_tokens.resize(cnt_tokens);
>       context.yyexpected_tokens(heap_tokens.data(), cnt_tokens);
>    }
>    int* tokens = (cnt_tokens <= 10) ? onstack_tokens : heap_tokens.data();

Yes, that definitely makes sense when no (reasonable) bound is known.

Actually if memory is really an issue, we could also use a bitset.  But a list 
of indexes seems the right approach here.  An iterator over the set of expected 
tokens would also work and meet all these use cases.  But maybe that's 
over-engineering.

Hum...

We should really think about that.

> That way, we can use the stack allocation in the common case (<= 10 expected 
> tokens) and only fallback to the heap-allocation if we have too many expected 
> tokens. At least in our grammar, it is common that only very few keywords are 
> expected, except in a few places, where you can write almost everything. Now 
> that I think about this, I am not actually sure how well this observation 
> generalizes to other grammars, though. Preallocating YYNTOKENS (480 in our 
> case)

Wow!!!  I believe this is the largest set of tokens I heard of!  
Congratulations :)

How many rules, how many states?

>  on the stack would cost us ~2KB of stack space.

Right.

> Probably that would be fine...
> 
>> The point here is efficiency
> It's funny how both of us are having performance/efficiency as a goal but are 
> coming to a different conclusion :) 

Yep :)

> I was not seeing your point about "not spend time counting how many more 
> there are" on my own. Thanks for pointing that out! With that reasoning, I 
> can completely agree that the existing implementation is fine.

Great, thanks!