[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Dynamic tokens

From: Ervin Hegedüs
Subject: Dynamic tokens
Date: Sun, 2 Feb 2020 20:29:28 +0100


is there any way to make a parser with "dynamic" tokens?

I mean in compiling time I don't know the available tokens. Now I describe
the necessary token with regex, but I bumped into a problem.

The language syntax is some like this:


where (as you can see) the LANG_OP is optional. If there isn't LANG_OP,
that means that is the most usable operator (namely "@rx" in my case). The
syntax of the operator (with regex): "@[a-z][a-zA-Z0-9]+".

Example from the language:
@eq 1
@lt 2
@streq foo

The problem is that the LANG_OP_ARGUMENT could be anything - for example,
that could be also starts with "@". So, the next expression is valid:

@streq @streq

Now I'm using this rules:
@[a-z][a-zA-Z0-9]+     { BEGIN(ST_LANG_OP); return LANG_OP; }

but now the operator isn't optional. If I write in the language:

"@rx" that means that's an operator argument, without operator.

I resolved this with put all available operators into one pattern, eg:
@(eq|lt|streq|...)    { BEGIN(ST_LANG_OP); return LANG_OP; }

and I can play with tokens and language rules... so it works as well.

I'ld like to modify the parser that it can be capable to extend to
recognize the operators (and other keywords of language) without modify and
rebuild the whole parser.

My idea is that if I can substitute the token pattern in the declaration
with a variable, and if I set up that variable before I call the parser. In
the example above, with a pseudo mode I could:

strcpy(VAR_LANG_OP_LIST, "@(eq|lt|streq|myop)");

and in the declaration of token should be:


Is there any way to solve this? If not, what do you think, how can I do

Thank you,


reply via email to

[Prev in Thread] Current Thread [Next in Thread]