[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Overlay mechanic improvements
From: |
Stephen Leake |
Subject: |
Re: Overlay mechanic improvements |
Date: |
Sat, 20 Sep 2014 11:28:12 -0500 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/24.3 (windows-nt) |
Richard Stallman <address@hidden> writes:
> [[[ To any NSA and FBI agents reading my email: please consider ]]]
> [[[ whether defending the US Constitution against all enemies, ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> The problem here is that tokens are not just lexemes(token text). To
> be general and granular enough one also has to remember dependencies
> between tokens. So, a token is: ["lexeme", lookback, lookahead,
> position].
>
> Ok, but why is it a "problem"? It seems easy enough to construct a
> list (LEXEME LOOKBACK LOOKAHEAD) and put it as the value of the
> `token' property for all the characters in the token.
>
> There is no need to include POSITION in this list, since you can
> find the start of the token by calling
> `previous-char-property-change'.
No need to store LEXME, either; just call buffer-substring-no-properties
when you need the text.
--
-- Stephe
- Re: Overlay mechanic improvements, (continued)
- Re: Overlay mechanic improvements, Vladimir Kazanov, 2014/09/20
- Tokenizing, Richard Stallman, 2014/09/20
- Re: Tokenizing, Stephen Leake, 2014/09/20
- Re: Tokenizing, Vladimir Kazanov, 2014/09/20
- Re: Tokenizing, Eric Ludlam, 2014/09/20
- Re: Tokenizing, Vladimir Kazanov, 2014/09/20
- Re: Tokenizing, Vladimir Kazanov, 2014/09/20
- Re: Tokenizing, Stefan Monnier, 2014/09/20
- Re: Tokenizing, Richard Stallman, 2014/09/21
- Re: Overlay mechanic improvements, Richard Stallman, 2014/09/20
- Re: Overlay mechanic improvements,
Stephen Leake <=