How does the response_format parameter in create_chat_completions get incorporated into the model's input?

[mjspeck]

If I pass in a json schema, is that just converted to a string and then incorporated into the prompt? How so? I'm just looking to understand more of the backend of this and LLMs in general.

[abetlen]

@mjspeck it isn't included in the prompt, it just overrides the grammar to the generic json grammar or the specific schema grammar if you provided one.

[mjspeck]

Right, but at the end of the day is an LLM not a function that takes in some text and outputs some text? How is the base LLM incorporating the schema information?

[vriesdemichael]

The llm input is unchanged, but the sampler is configured to only choose from valid tokens according to the grammar, instead of all possible tokens

[YiandLi]

Hello, I have a question about response_format parameter , when I use create_chat_completion method, there is a response_format parameter, but it may not work if there is no "schema" key.

In the method chat_completion_handler:

        if response_format is not None and response_format["type"] == "json_object":
            try:
                # create grammar from json schema
                if "schema" in response_format:
                    grammar = llama_grammar.LlamaGrammar.from_json_schema(
                        json.dumps(response_format["schema"])
                    )
            except Exception as e:
                grammar = llama_grammar.LlamaGrammar.from_string(llama_grammar.JSON_GBNF)

In my opinion, if there is no schema in response_format passed ， it will just skip this parameter. Is that right ?

BTW ; I have another question, is every model support this parameter ? Or where I can find which model could support this parameter ?

[vriesdemichael]

When you do not provide a json schema it will default to a JSON grammar. This means you will have the guarantee of a valid json output, but no control over the content of the json output.

It should work for any model btw. It is part of how the output of the llm is dealt with.

The output of an llm is a probability distribution with a probability for every token known to the model.

Normally the sampler will choose a token based on the probability (and a random factor determined by the temperature parameter).

The schema or grammar you specify limits this choice to only valid tokens.

[YiandLi]

Thank you for your reply !! Helps a lot !

[lauraminkova]

Hi, I have a question regarding "response_format"={"type": "json_object"}. I expected it to constrain my output to solely JSONs, but unfortunately the model continues to provide explanations outside of the JSON. I'm currently testing mistral-7b-instruct-v0.2 on GPU.

LLM_MISTRAL = Llama(model_path="mistral-7b-instruct-v0.2.Q4_K_M.gguf", n_ctx=4000, n_gpu_layers=-1, verbose=True, use_mlock=True)
output = LLM_MISTRAL.create_chat_completion(
            messages=[
                {"role": "user", "content": "<s>You are a helpful assistant that only outputs valid JSON structures. You do not generate any other information.</s>"},
                {"role": "user", "content": input_prompt}
            ],
            temperature=0,
            response_format={"type": "json_object"})`

I'm wondering if this has to do with llama-cpp-python or with the Mistral model itself? Any help would be really appreciated!

[mitar]

How does this interact with temperature? If I set temperature to 0, can I still my own JSON schema for expected output? Or will then sampling not have enough tokens to pick from?