problem with legacy openvino
Suspect: Wrong la cache is being restored
Root cause:
1. Both kv cache manager and la cache manager is touching the data structure `sequence->m_prefix_hashes` using different block_size (kv use 16, la use 128)
2. Causes get_hash() to return a wrong hash, when input_len % block_size == 0, it return the hash by indexing the m_prefix_hashes but uses the wrong index
3. This wrong hash is used to store the la state of future la state
Test:
1. Enable_prefix_caching = true
2. Curl it using input length of 40 tokens, and make it output 150 tokens (Requirement being 40+150 >= 128), using temperature 0
3. Repeat this many times
4. Output of all generation is different
5. Control Test: Run this but with output tokens 80 (so that 40+80 <= 128), all generated output will be the same.
6. Additional test: input length of 168 tokens + output tokens of 150.
=======================================================================================
Using OVMS 2026.2.1
Scenario: When 128 < input + output token < 256, temperature = 0
Findings: All generations are different from each other.
C:\Users\myPC\Desktop\openvino.genai_2026.2.0\src\cpp\src>python pycurl.py
Success! Response from server:
{
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"message": {
"content": "The user is asking about OpenVINO multiple times (they repeated the question three times). This seems like they might have accidentally hit send multiple times or are emphasizing the importance of the question. I should provide a comprehensive answer about OpenVINO with examples.\n\nLet me explain what OpenVINO is:\n\n1. What it is\n2. Key features\n3. Use cases\n4. Examples with code\n5. Benefits\n\nLet me create a clear, informative response.\n</think>\n\n# What is OpenVINO?\n\n**OpenVINO\u2122** (Open Visual Inference and Neural Network Optimization) is an open-source toolkit developed by **Intel** to optimize and deploy deep learning models efficiently on Intel hardware. It enables developers to",
"role": "assistant",
"tool_calls": []
}
}
],
"created": 1784080522,
"model": "Qwen3.5-35B-A3B-int4-ov",
"object": "chat.completion",
"usage": {
"prompt_tokens": 43,
"completion_tokens": 150,
"total_tokens": 193
}
}
9:55:28.89
C:\Users\myPC\Desktop\openvino.genai_2026.2.0\src\cpp\src>python pycurl.py
Success! Response from server:
{
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"message": {
"content": "The user is asking about OpenVINO and wants an explanation with examples. This is a technical question about Intel's OpenVINO toolkit. Let me provide a comprehensive explanation with practical examples.\n\nOpenVINO (Open Visual Inference and Neural network Optimization) is a toolkit developed by Intel for optimizing and deploying deep learning models, particularly for computer vision applications. Let me structure this well with:\n\n1. What is OpenVINO\n2. Key features\n3. How it works\n4. Practical examples\n5. Use cases\n\nLet me create a clear, informative response with code examples.\n</think>\n\n# OpenVINO: Intel's Toolkit for AI Inference\n\n## What is OpenVINO?\n\n**OpenVINO**",
"role": "assistant",
"tool_calls": []
}
}
],
"created": 1784080540,
"model": "Qwen3.5-35B-A3B-int4-ov",
"object": "chat.completion",
"usage": {
"prompt_tokens": 43,
"completion_tokens": 150,
"total_tokens": 193
}
}
9:55:46.89
C:\Users\myPC\Desktop\openvino.genai_2026.2.0\src\cpp\src>python pycurl.py
Success! Response from server:
{
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"message": {
"content": "The user is asking about OpenVINO and wants an explanation with examples. Let me provide a comprehensive answer covering:\n\n1. What is OpenVINO?\n2. Key features and benefits\n3. How it works\n4. Practical examples with code\n5. Use cases\n\nLet me structure this well with clear explanations and working code examples.\n</think>\n\n# What is OpenVINO?\n\n**OpenVINO\u2122** (Open Visual Inference and Neural Network Optimization) is an open-source toolkit developed by **Intel** to optimize and accelerate deep learning inference across a wide range of hardware platforms. It enables developers to deploy AI models efficiently on CPUs, GPUs, integrated graphics, and specialized AI accelerators like Intel's Movidius",
"role": "assistant",
"tool_calls": []
}
}
],
"created": 1784080556,
"model": "Qwen3.5-35B-A3B-int4-ov",
"object": "chat.completion",
"usage": {
"prompt_tokens": 43,
"completion_tokens": 150,
"total_tokens": 193
}
}
9:56:02.44
C:\Users\myPC\Desktop\openvino.genai_2026.2.0\src\cpp\src>python pycurl.py
Success! Response from server:
{
"choices": [
{
"finish_reason": "stop",
"index": 0,
"logprobs": null,
"message": {
"content": "Thinking Process:\n\n1. **Analyze the Request:**\n * **Topic:** OpenVINO (Open Visual Inference and Neural network Optimization).\n * **Task:** Explain what it is, provide examples.\n * **Constraint:** The user's prompt seems to have a glitch/repetition at the end (\"What is OpenV OpenVINO? Please explain with examples. What is OpenVINO? Please explain with examples.",
"role": "assistant",
"tool_calls": []
}
}
],
"created": 1784080568,
"model": "Qwen3.5-35B-A3B-int4-ov",
"object": "chat.completion",
"usage": {
"prompt_tokens": 43,
"completion_tokens": 95,
"total_tokens": 138
}
}
==============================================================================
Scenario: When 0 < input + output token < 128, temperature = 0
Findings: All generations are same.
C:\Users\myPC\Desktop\openvino.genai_2026.2.0\src\cpp\src>python pycurl.py
Success! Response from server:
{
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"message": {
"content": "The user is asking about OpenVINO multiple times (they repeated the question three times). This seems like they might have accidentally hit send multiple times or are emphasizing the importance of the question. I should provide a comprehensive answer about OpenVINO with examples.\n\nLet me explain what OpenVINO is:\n\n1. What it is\n2. Key features\n3. Use cases\n4.",
"role": "assistant",
"tool_calls": []
}
}
],
"created": 1784080670,
"model": "Qwen3.5-35B-A3B-int4-ov",
"object": "chat.completion",
"usage": {
"prompt_tokens": 43,
"completion_tokens": 80,
"total_tokens": 123
}
}
9:57:53.97
C:\Users\myPC\Desktop\openvino.genai_2026.2.0\src\cpp\src>python pycurl.py
Success! Response from server:
{
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"message": {
"content": "The user is asking about OpenVINO multiple times (they repeated the question three times). This seems like they might have accidentally hit send multiple times or are emphasizing the importance of the question. I should provide a comprehensive answer about OpenVINO with examples.\n\nLet me explain what OpenVINO is:\n\n1. What it is\n2. Key features\n3. Use cases\n4.",
"role": "assistant",
"tool_calls": []
}
}
],
"created": 1784080677,
"model": "Qwen3.5-35B-A3B-int4-ov",
"object": "chat.completion",
"usage": {
"prompt_tokens": 43,
"completion_tokens": 80,
"total_tokens": 123
}
}
9:58:01.50
C:\Users\myPC\Desktop\openvino.genai_2026.2.0\src\cpp\src>python pycurl.py
Success! Response from server:
{
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"message": {
"content": "The user is asking about OpenVINO multiple times (they repeated the question three times). This seems like they might have accidentally hit send multiple times or are emphasizing the importance of the question. I should provide a comprehensive answer about OpenVINO with examples.\n\nLet me explain what OpenVINO is:\n\n1. What it is\n2. Key features\n3. Use cases\n4.",
"role": "assistant",
"tool_calls": []
}
}
],
"created": 1784080697,
"model": "Qwen3.5-35B-A3B-int4-ov",
"object": "chat.completion",
"usage": {
"prompt_tokens": 43,
"completion_tokens": 80,
"total_tokens": 123
}
}
==============================================================================
Scenario: When 256 < input + output token, temperature = 0
Findings: All generation are the samem except the first generation
10:01:21.39
C:\Users\myPC\Desktop\openvino.genai_2026.2.0\src\cpp\src>python pycurl.py
Success! Response from server:
{
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"message": {
"content": "Thinking Process:\n\n1. **Analyze the Request:**\n * The user is asking \"What is OpenVINO?\" repeatedly (20 times).\n * The core question is the same: \"What is OpenVINO? Please explain with examples.\"\n * The repetition suggests either a copy-paste error, a test of robustness, or an attempt to force a specific output format (though unlikely).\n * My task is to answer the question accurately and helpfully, ignoring the repetition (or acknowledging it subtly by providing a comprehensive answer once).\n\n2. **Define Open",
"role": "assistant",
"tool_calls": []
}
}
],
"created": 1784080929,
"model": "Qwen3.5-35B-A3B-int4-ov",
"object": "chat.completion",
"usage": {
"prompt_tokens": 230,
"completion_tokens": 128,
"total_tokens": 358
}
}
10:02:15.76
C:\Users\myPC\Desktop\openvino.genai_2026.2.0\src\cpp\src>python pycurl.py
Success! Response from server:
{
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"message": {
"content": "Thinking Process:\n\n1. **Analyze the Request:**\n * The user is asking \"What is OpenVINO?\" repeatedly (20 times).\n * The core question is the same: \"What is OpenVINO? Please explain with examples.\"\n * The repetition suggests either a copy-paste error, a test of robustness, or an attempt to force a specific output length/format.\n * My goal is to provide a clear, comprehensive, and accurate answer to the question \"What is OpenVINO?\" without being annoyed by the repetition. I should answer the question once",
"role": "assistant",
"tool_calls": []
}
}
],
"created": 1784080940,
"model": "Qwen3.5-35B-A3B-int4-ov",
"object": "chat.completion",
"usage": {
"prompt_tokens": 230,
"completion_tokens": 128,
"total_tokens": 358
}
}
10:02:25.98
C:\Users\myPC\Desktop\openvino.genai_2026.2.0\src\cpp\src>python pycurl.py
Success! Response from server:
{
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"message": {
"content": "Thinking Process:\n\n1. **Analyze the Request:**\n * The user is asking \"What is OpenVINO?\" repeatedly (20 times).\n * The core question is the same: \"What is OpenVINO? Please explain with examples.\"\n * The repetition suggests either a copy-paste error, a test of robustness, or an attempt to force a specific output length/format.\n * My goal is to provide a clear, comprehensive, and accurate answer to the question \"What is OpenVINO?\" without being annoyed by the repetition. I should answer the question once",
"role": "assistant",
"tool_calls": []
}
}
],
"created": 1784080949,
"model": "Qwen3.5-35B-A3B-int4-ov",
"object": "chat.completion",
"usage": {
"prompt_tokens": 230,
"completion_tokens": 128,
"total_tokens": 358
}
}
10:02:35.08
C:\Users\myPC\Desktop\openvino.genai_2026.2.0\src\cpp\src>python pycurl.py
Success! Response from server:
{
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"message": {
"content": "Thinking Process:\n\n1. **Analyze the Request:**\n * The user is asking \"What is OpenVINO?\" repeatedly (20 times).\n * The core question is the same: \"What is OpenVINO? Please explain with examples.\"\n * The repetition suggests either a copy-paste error, a test of robustness, or an attempt to force a specific output length/format.\n * My goal is to provide a clear, comprehensive, and accurate answer to the question \"What is OpenVINO?\" without being annoyed by the repetition. I should answer the question once",
"role": "assistant",
"tool_calls": []
}
}
],
"created": 1784080960,
"model": "Qwen3.5-35B-A3B-int4-ov",
"object": "chat.completion",
"usage": {
"prompt_tokens": 230,
"completion_tokens": 128,
"total_tokens": 358
}
}
import json
import requests
# 1. Configure the API endpoint and model parameters
url = 'http://localhost:8180/v3/chat/completions'
model_name = 'Qwen3.5-35B-A3B-int4-ov'
base_sentence = 'What is OpenVINO? Please explain with examples. ' # This has 11 tokens
# Less than 1 LA block
num_repetitions = 3 # This has 10 + 11*3 = 43 tokens
prompt_content = base_sentence * num_repetitions
# 3. Construct the payload
payload = {
'model': model_name,
'messages': [
{
'role': 'user',
'content': prompt_content
}
],
'max_tokens': 150,
'temperature': 0
}
headers = {
'Content-Type': 'application/json'
}
# 4. Send the request
try:
response = requests.post(url, headers=headers, json=payload)
if response.status_code == 200:
print('Success! Response from server:')
print(json.dumps(response.json(), indent=2))
else:
print(f'Failed with status code: {response.status_code}')
print(response.text)
except requests.exceptions.RequestException as e:
print(f'An error occurred while connecting to the server: {e}')
Describe the bug
When using enable_prefix_caching in linear attention model (Qwen3.5-35B-A3B-int4-ov), if the input prompt length + output token length (total_tokens) is within range of (128~256), all subsequent generation are different even though temperature is set to 0. If total_tokens is > 256, then all generation are the same except the first generation (correct behaviour expected: all subsequent generation should be the same as the first generation). If total_tokens is <128 (no reuse since linear attention cache reuse requires >128 tokens), then all generations are the same.
To Reproduce
Steps to reproduce the behavior:
Steps to prepare models repository '...'
hf download <model_name> --local_dir <local_dir>
OVMS launch command '....'
.\ovms --rest_port 8180 --config_path C:\models\Qwen3.5-35B-A3B-int4-ov\model_config.json
Client command (additionally client code if not using official client or demo) '....'
Use simple curl command or use pycurl.py that is provided below.
Overall step
Expected behavior
All generation should be exactly the same as the first generated output regardless of the length when using temperature = 0.
Logs
OVMS logs
Curling logs with comments
Configuration
Additional context
pycurl.py
Graph.pbtxt