Re: LLM Experiments, Part 1: Corrections

[Andrew Hyatt]

Thanks for this super useful response. BTW, I'm going to try again to
use gnus to respond, after making some changes, so apologies if the
formatting goes awry. If it does, I'll re-respond in gmail.

On Tue, Jan 23, 2024 at 05:26 PM contact@karthinks.com wrote:

> Hi Andrew,
>
> Having worked on similar problems in gptel for about nine months now
> (without much success), here are some thoughts.
>
>> Question 1: Does the llm-flows.el file really belong in the llm 
>> package?  It does help people code against llms, but it expands 
>> the scope of the llm package from being just about connecting to 
>> different LLMs to offering a higher level layer necessary for 
>> these more complicated flows.  I think this probably does make 
>> sense, there's no need to have a separate package just for this 
>> one part.
>
> I think llm-flows works better as an independent package for now,
> since it's not clear yet what the "primitive" operations of working
> with LLMs look like.  I suspect these usage patterns will also be a
> matter of preference for users, as I've realized from comparing how I
> use gptel to the feature requests I get.

I agree that the more we assume specific patterns (which indeed seems to
be where things are going), the more it should go into its own package
(or maybe be part of ellama or something).  But there may be some
commonality that are just useful regardless of the usage patterns.  I
think we'll have to see how this plays out.

>> Question 2: What's the best way to write these flows with multiple 
>> stages, in which some stages sometimes need to be repeated? It's 
>> kind of a state machine when you think about it, and there's a 
>> state machine GNU ELPA library already (fsm). I opted to not model 
>> it explicitly as a state machine, optimizing instead to just use 
>> the most straightforward code possible.
>
> An FSM might be overkill here.  At the same time, I'm not sure that
> all possible interactions will fit this multi-step paradigm like
> rewriting text does.
>
>> Question 3: How should we deal with context? The code that has the 
>> text corrector doesn't include surrounding context (the text 
>> before and after the text to rewrite), but it usually is helpful. 
>> How much context should we add? The llm package does know about 
>> model token limits, but more tokens add more cost in terms of 
>> actual money (per/token billing for services, or just the CPU 
>> energy costs for local models). Having it be customizable makes 
>> sense to some extent, but users are not expected to have a good 
>> sense of how much context to include. My guess is that we should 
>> just have a small amount of context that won't be a problem for 
>> most models. But there's other questions as well when you think 
>> about context generally: How would context work in different 
>> modes? What about when context may spread in multiple files? It's 
>> a problem that I don't have any good insight into yet.
>
> I see different questions here:
>
> 1.  What should be the default amount of context included with
> requests?
> 2.  How should this context be determined? (single buffer, across
> files etc)
> 3.  How should this be different between modes of usage, and how
> should this be communicated unambiguously?
> 4.  Should we track token costs (when applicable) and communicate them
> to the user?
>
> Some lessons from gptel, which focuses mostly on a chat interface:
>
> 1.  Users seem to understand gptel's model intuitively since they
> think of it like a chat application, where the context is expected to
> be everything in the buffer up to the cursor position.  The only
> addition is to use the region contents instead when the region is is
> active.  This default works well for more than chat, actually.  It's
> good enough when rewriting-in-place or for continuing your prose/code.

I think it may be still useful to use the context even when modifying
the region, though.

>
> 2.  This is tricky, I don't have any workable ideas yet.  In gptel
> I've experimented with providing the options "full buffer" and "open
> project buffers" in addition to the default, but these are both
> overkill, expensive and rarely useful -- they often confuse the LLM
> more than they help.  Additionally, in Org mode documents I've
> experimented with using sparse trees as the context -- this is
> inexpensive and can work very well but the document has to be
> (semantically) structured a certain way.  This becomes obvious after a
> couple of sessions, but the behavior has to be learned nevertheless.
>
> 3a.  For coding projects I think it might be possible to construct a
> "sparse tree" with LSP or via treesitter, and send (essentially) an
> "API reference" along with smaller chunks of code.  This should make
> basic copilot-style usage viable.  I don't use LSP or treesitter
> seriously, so I don't know how to do this.
>
> 3b. Communicating this unambiguously to users is a UI design question,
> and I can imagine many ways to do it.

Thanks, this is very useful.  My next demo is with org-mode, and there
I'm currently sending the tree (just the structure) as context.

There's a meta-question here is that we don't know how what the right
thing to do is, because we don't have a quality process. If we were
doing this seriously, we'd want a corpus of examples to test with, and a
way to judge quality (LLMs can actually do this as well). The fact that
people could be running any model is a significant complicating factor.
But as it is, we just have our own anecdotal evidences, and reasonable
hunches.  As these things get better, I think it would converge to what
we would think is reasonable context anyway.

>
> 4.  I think optionally showing the cumulative token count for a
> "session" (however defined) makes sense.

The token count should only be applicable to the current LLM count. My
understanding is that the machinery around the LLMs is adding the
previous conversation, in whole, or in summary, as context. So it's
really hard to understand the actual token usage once you start having
conversations. That's fine, though, since most tokens are used in the
first message where context appears, and subsequent rounds are fairly
light.

>
>> Question 5: Should there be a standard set of user behaviors about 
>> editing the prompt? In another demo (one I'll send as a followup), 
>> with a universal argument, the user can edit the prompt, minus 
>> context and content (in this case the content is the text to 
>> correct). Maybe that should always be the case. However, that 
>> prompt can be long, perhaps a bit long for the minibuffer. Using a 
>> buffer instead seems like it would complicate the flow. Also, if 
>> the context and content is embedded in that prompt, they would 
>> have to be replaced with some placeholder. I think the prompt 
>> should always be editable, we should have some templating system. 
>> Perhaps emacs already has some templating system, and one that can 
>> pass arguments for number of tokens from context would be nice.
>
> Another unsolved problem in gptel right now.  Here's what it uses
> currently:
>
> - prompt: from the minibuffer
> - context and content: selected region only
>
> The main problem with including context separate from the content here
> is actually not the UI, it's convincing the LLM to consistently
> rewrite only the content and use the context as context.  Using the
> prompt+context as the "system message" works, but not all LLM APIs
> provide a "system message" field.

Yes, the LLM library already separates the "system message" (which we
call context, which I now realize is a bit confusing), and each provider
just deals with it in the best way possible.  Hopefully as LLM
instruction following gets better, it will stop treating context as
anything other than context.  In the end, IIUC, it all ends up in the
same place when feeding text to the LLM anyway.

>
>> Question 6: How do we avoid having a ton of very specific 
>> functions for all the various ways that LLMs can be used? Besides 
>> correcting text, I could have had it expand it, summarize it, 
>> translate it, etc. Ellama offers all these things (but without the 
>> diff and other workflow-y aspects). I think these are too much for 
>> the user to remember.
>
> Yes, this was the original reason I wrote gptel -- the first few
> packages for LLM interaction (only GPT-3.5 back then) wanted to
> prescribe the means of interaction via dedicated commands, which I
> thought overwhelmed the user while also missing what makes LLMs
> different from (say) language checkers like proselint and vale, and
> from code refactoring tools.
>
>> It'd be nice to have one function when the 
>> user wants to do something, and we work out what to do in the 
>> workflow. But the user shouldn't be developing the prompt 
>> themselves; at least at this point, it's kind of hard to just 
>> think of everything you need to think of in a good prompt.  They 
>> need to be developed, updated, etc. What might be good is a system 
>> in which the user chooses what they want to do to a region as a 
>> secondary input, kind of like another kind of 
>> execute-extended-command.
>
> I think having users type out their intention in natural language into
> a prompt is fine -- the prompt can then be saved and added to a
> persistent collection.  We will never be able to cover
> (programmatically) even a reasonable fraction of the things the user
> might want to do.

Agreed. Increasingly, it seems like a really advanced prompt management
system might be necessary.  How to do that is it's own separate set of
questions.  How should they be stored?  Are these variables?  Files?
Data in sqlite?

>
> The things the user might need help with is what I'd call "prompt
> decoration".  There are standard things you can specify in a prompt to
> change the brevity and tone of a response.  LLMs tend to generate
> purple prose, summarize their responses, apologize or warn
> excessively, etc.  We can offer a mechanism to quick-add templates to
> the prompt to stem these behaviors, or encourage other ones.

Agreed.  I have a prompting system in my ekg project, and it allows
transcluding other prompts, which is quite useful.  So you might want to
just include information about your health, or your project, or even
more dynamic things like the date, org agenda for the day, or whatever.
This is powerful and a good thing to include in this future prompt
management system.

>
> Karthik