Re: LLM Experiments, Part 1: Corrections

[Andrew Hyatt]

On 22 January 2024 14:06, "T.V Raman" <raman@google.com> wrote: 

   Some more related thoughts below, mostly thinking aloud: 1. 
   From using gptel and ellama against the same model, I see 
   different 
      style responses, and that kind of inconsistency would be 
      good to get a handle on; LLMs are difficult enough to 
      figure out re what they're doing without this additional 
      variation.

Is this keeping the prompt and temperature constant?  There's 
inconsistency, though, even keeping everything constant due to the 
randomness of the LLM.  I often get very different results, for 
example, to make the demo I shared, I had to run it like 5 times 
because it would either do things too well (no need to demo 
corrections), or not well enough (for example, it wouldn't follow 
my orders to put everything in one paragraph).

   2. Package LLM has the laudible goal of bridgeing between 
   models and 
      front-ends, and this is going to be vital. 
   3. (1,2) above lead  to the following question: 4. Can we 
   write down  a list of common configuration vars --- here 
     common across the model axis. Make  it a union of all such 
     params.

I think the list of common model-and-prompt configuration should 
already be already in the llm package already, but we probably 
will need to keep expanding this.


   5. Next, write down a list of all configurable params on the 
   UI side.

This will change quite a bit depending on the task. It's unclear 
how much should be configurable - for example, in the demo, I have 
ediff so the user can see and evaluate the diff. But maybe that 
should be configurable, so if the user wants to see just a diff 
output instead, perhaps that should be allowed? When I was 
thinking about a state machine, I was thinking that parts of the 
state machine might be overridable by the user, such as a "have 
the user check the results of the operation" is a state in the 
state machine that the user can just define their own function 
for.  I suspect we'll have a better idea of this after a few more 
demos.

   6. When stable, define a single data-structure in elisp that 
   acts as 
      the bridge between the front-end emacs UI and the LLM 
      module.

If I understand you correctly, this would be the configuration you 
listed in your point (4) and (5)?


   7. Finally factor out  the settings of that structure and make 
   it 
      possible to create "profiles" so that one can predictably 
      experiment across front-ends and models. 

I like this idea, thanks!