Re: Parsing a language with optional spaces

[Daniele Nicolodi]

On 08/07/2020 15:30, Christian Schoenebeck wrote:
> On Mittwoch, 8. Juli 2020 22:14:23 CEST Akim Demaille wrote:
>>> So yes, you could certainly address this to work correctly with Flex with
>>> additional measures,
>>
>> The case you are citing above is straightforward to handle the same
>> way I did with the BASIC.  That's fairly simple given that this is
>> plain rational languages, fully under the perimeter of Flex.
>>
>> The general case though will not work just as well.  John reported that
>> the grammar is
>>
>> DO <number> <variable> = <expression> , <expression> [, <expression>]
>>
>> and here the problem is that the <expression>s can have parenthesized
>> expressions with embedded commas.  So on this case, Flex is screwed.
>>
>> That reminds me of a paper I read long ago, someone proposing
>> "heisentokens": having the scanner return multiple tokens concurrently,
>> for the different scanning options, branched into GLR to follow the
>> various options and let dead ends die (that would be a second time?).
> 
> "Dying branches" is exactly what I always had in mind for a potential scanner-
> parser marriage in first place.
> 
> But it's probably not only about tokens in parallel; you would probably also 
> need to maintain several parser states in parallel, as it might need several 
> reductions to get to the point where all ambiguities are resolved and only 
> one 
> progression in the grammar remains.
> 
> But the profits would be stunning: you could write parsers in a very straight-
> forward way, even for very complex languages. And on the other side it would 
> eventually allow to design languages in a much more user friendly and 
> flexible 
> way as well, as it would be straight-forward e.g. to restrict keywords (or 
> other patterns) to certain grammar contexts only.

I may be missing something, but isn't this what a PEG parser does?

Cheers,
Dan