Re: SMIE implementation for the C-like languages

[Stefan Monnier]

> Language itself is close to the family of C-like languages
> with some differences to the language constructions.  I'm looking for
> some advice about applying smie to the languages that use braces as a
> terminators for the code blocks.

Indeed, SMIE is not great for that currently.

I have an "smc-mode" (i.e. SMIE-based c-mode) here which I wrote as an
exercise to try and see what it takes to get SMIE working acceptably for
the C language syntax.  It's not usable (it was really meant as an
experimental prototype/proof-of-concept), but if you're interested to
look at it, I could make it available somewhere.

> 1. As stated in documentation tokens that are defined in syntax table
> don't have to be tokenised in lexer. I tried to go this way, but
> encountered situations where defined grammars are not respected.

Not sure which situations you're referring to.

> It seems that smie only tries to indent closer token with respect to
> the opener, rather than parent token defined by grammar.

By default, yes.  Of course, the smie-rule-function is there to tweak
that as/when needed.

> cases, but I encountered issues with paren blinking: in some
> situations blinking fails with "Mismatched parenthesis".

Same as before: if you don't give an example, it's hard to know what
might be the cause.

> During some tests I decided to change lexer rules for braces to return
> begin/end tokens instead of braces. I noticed that smie still tries to
> indent "}" token in some situations, specifically `:close-all . "}"`.

At this point, I'm very confused, because I don't know what your code
does when, nor when you see which behavior.

> So my question is what will be the semantically correct way of
> handling braces for the C-like languages?

Where?  In the lexer, the grammar, or the indentation rules?
Note that even if you answer this question, there's no single right
answer: you largely get to decide and pick between different consequences.

> And secondary question is it expected that smie tries to indent tokens
> that are not returned by lexer?

If your tokenizer returns nil (or "") and you're in front of a paren,
then SMIE will take this paren to be the next token, yes.

> 2. As a sort of continuation of the previous problem, we are having
> problem understanding what will be semantically correct way of defining
> `sexp` for the smie based mode. At the moment we see a different
> behavior between non-smie c++ mode (which is close to the Swift)
> and something like ruby-mode. One of the contributers summarised
> differences in this post
> https://github.com/chrisbarrett/swift-mode/pull/117#issuecomment-154753070.
> I personally think grammar based sexp provided by smie are extremely
> useful, but they yield confusing results when it comes to blinking
> parens. For example grammar for "if" from here:
> https://github.com/chrisbarrett/swift-mode/blob/simplify_smie/swift-mode.el#L74-L129

Does Swift allow a "{ ... }" block to appear on its own (rather than
as part of a while/if/...), like in C?

If it does (and you want swift-mode to support those blocks), then
I think your approach to use rules like

   ("while" exp "{" insts "}")

will be very problematic because SMIE's parser when parsing backward
(which is the more usual direction) won't know whether to stop after
skipping a {...} or whether to keep going on the off-chance that this
{...} is really part of a for/while/if...
[ The specific complaint you should get is that "{" will appear both as
  an "opener" and as a "neither" (aka "inner") token.  ]

> works well for indentation and movements, but blinking on the close
> ("}") returns "if" token.

Indeed, if these two notions of "sexp" need to be different, then you should
probably disable SMIE's builtin paren-blinking support.


        Stefan