microsoft guidance ( A guidance language for controlling large language models. )

[jediknight813]

https://github.com/microsoft/guidance

Has anyone figured out how to make this work with llama-cpp-python?

[abetlen]

Yup, just got this running with the drop-in OpenAI server, you may need to upgrade llama-cpp-python because there was a small compatibility bug I had to fix.

The notebook is here
https://github.com/abetlen/llama-cpp-python/blob/main/examples/notebooks/Guidance.ipynb

[abetlen]

Note this has the same limitations as the OpenAI models for guidance, the next step is to implement a LlamaCpp class that extends guidance.LLM with all the token healing and other guidance features.

[vmajor]

I am getting:

INFO: 127.0.0.1:55114 - "POST /completions HTTP/1.1" 404 Not Found on server console. I am using the latest available llama-cpp-python

EDIT: now it works. Nothing changed except that I restarted the notebook. Now I have another error but that is for a separate post.

[vmajor]

It works! This is with gpt4-alpaca-lora_mlp-65B.ggmlv3.q5_1.bin:

Tweak this proverb to apply to model instructions instead.

Where there is no guidance, a people falls,
but in an abundance of counselors there is safety.
- Proverbs 11:14

UPDATED
Where there is no guidance from the instruction manual, a build falls apart,
but with an abundance of instructions, there is success.
- GPT 3:20