Re: How to add pseudo vector types

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Of course those parameters would be configurable on Emacs, but disabling
>> TS on a 2MB file because it uses 20MB is way too conservative, IMHO.
>
> Why would we limit ourselves to 20MB?  uint32_t supports upto 4GB.

I didn't suggest that we should limit ourselves to 20MB, I observed that
current machines have enough resources for handling large files ("large"
meaning "big enough to keep me busy reading for some years.")

>> Guys, you are speculating too much about minutia and worst-case
>> scenarios. (Do we really care about TS not supporting files larger than
>> 4GB? I mean, REALLY?)
>
> Yes, we do.  For at least 2 reasons: (a) source code files produced by
> programs can be very large;

I know, I work with machine-generated (read: code-dense) 20+MB C++ files
on a regular basis.

However, I wouldn't agree on renouncing to useful features because they
could be problematic when dealing with large files. That is, it would be
a mistake to discard TS as inadequate for Emacs just because it doesn't
benefit (and I say "not benefit", not "penalise") certain use cases.

> (b) having a feature that fails before you
> reach the max size of a buffer Emacs supports is a problem, because it
> will cause hard-to-deal-with problems.

We can put reasonable limits on when to use TS once we have some
experience with it. What matters right now is if TS would be usable for
the typical use case, and I guess the answer is positive. Also, it is
not as if we had other options to consider.

>> I'll rather focus on implementing the thing and optimize later. My bet
>> is that a crude implementation would work fine for the 99% of the users
>> and be an improvement over what we have now on practically all cases.
>
> This is not a prototype project.  (Or at least I hope it won't end up
> being that.)  This is supposed to be the industry-strength code that
> core Emacs will use for the years to come to support features which
> need language-dependent parsing.  It cannot work correctly only in 99%
> of use cases.  So we must assess the limitations seriously and plan
> ahead for them.

I said "would work *fine* for the 99% of users", this does not imply
that it would work incorrectly for the rest.

On the "planning ahead" part, TS support would be an optional,
quasi-external feature for some time, it is not as if it comes out with
some critical bug Emacs would become unusable. TS support can be
fine-tuned without disrupting the rest of Emacs development. If, on the
other hand, we start making changes on Emacs' internals for allowing
some TS-related optimizations (even when we don't know if they are
neccessary at all) that could be a destabilizing move for Emacs as a
whole. Apart from delaying TS support.

>> BTW, a 10x AST/source-code size ratio is quite reasonable.
>
> It could be, but please don't forget that this is _in_addition_to_ the
> "normal" Emacs memory footprint, and that could easily be 1GB and
> sometimes several times that.

Yes, but if you want something you need to pay something, and you can
hardly get TS' features with less than that. At least for complex
languages like C++.

Talking about scenarios of heavy memory usage, I'll comment in passing
that in my recent experience, once Emacs exceeds 2GB the gc pauses start
to be so annoying that I don't care anymore about how much memory an
external tool would use if it works fast enough.