How do i make the model use information that has been previously entered?

[bennybweng]

See this example code:

output = llm("My favorite food is Bread.", max_tokens=40, echo=True)
print(output)
output = llm("My favorite food is", max_tokens=40, echo=True)
print(output)

I would expect the second output to be something like: "My favorite food is bread" but instead it returns: "My favorite food is pizza."
How do i make the model remember information? Do i need to use the low-level-API?
If my question is a little unclear, i am looking for something interactive like in the main.cpp from llama.cpp but instead of entering the prompts in the terminal, i want to enter them through the python code. Sorry if the answer is obvious, I'm pretty new to this stuff.

[vmajor]

In python you can manually feed the output back as input. You will need to feed both the LLM output and your new inputs back as a prompt. Please be aware that you will run out of the tiny 2048 token context pretty quickly.

[AnOpenSauceDev]

i used something similar to this for my project:

def chat(message):
    global chat_memory

    USER_NAME = "Human"
    AI_NAME = "llama-cpp-python"
    out = model.create_completion(prompt=f""" 
    [put your prompt here]
    ---An example conversation---
	[PUT YOUR EXAMPLE HERE]
    --- end of example conversation----
    here is your current chat history, use this to remember context from earlier. (if {AI_NAME} said this, you said this. Otherwise, that was a user.).
    this is for you to refrence as memory, not to use in chat. i.e. "oh yes, i remember you saying this some time ago." if it isn't acutally in history, dont say it.
    ---beginning of your chat history, use this as memory.---
    {chat_memory}
    ---end of your chat history---
    --- current chat session, use this in your response ---
    {USER_NAME}, message:{message}
    {AI_NAME}:""",temperature=0.7,repeat_penalty=1.1,max_tokens=512,stop=[f"message:",f"{USER_NAME}:",f"{AI_NAME}:"],top_p=0.95,top_k=20)
    
    dat = json.dumps(out)
    final = json.loads(dat)
    list = final.get("choices")
    nxt = next(iter(list), 'fail')

    txt = nxt.get("text")
    chat_memory += f"{AI_NAME}, message: {txt} \n"
    return txt

what this method does is:

1. feed the model a prompt with a chat history segment
2. get the output of the completion as a single string
3. add the completion as a history entry to use in further prompts

do note you almost certainly want to set n_ctx to something higher like 4000+ so you can feed it more tokens. (this will use more VRAM though). 7B models sometimes hallucinate with this, but with vicuna-33b and Wizard-Vicuna-13B-Uncensored, i got pretty good results.


EDIT: you WILL need to also add user messages to chat_memory too, this example method doesn't handle that.