Qwen3.5-35B-A3B-int4-ov gibberish output during high memory pressure

[TayQiXiang]

Describe the bug
When inferencing the model with an input token count of 16389 and an output token count of 512, the returned response is gibberish (All Exclamation Marks).

To Reproduce
Steps to reproduce the behavior:

1. Download ovms_windows_2026.2.0_python_off.zip
2. Download models from HuggingFace
3. CD into downloaded ovms folder, run setupvars.bat
4. OVMS launch command: .\ovms --rest_port 8180 --config_path "C:\models\Qwen3.5-35B-A3B-int4-ov\model_config.json"
5. Client command: python pycurl.py

Expected behavior
A proper response regarding the question.

Logs

Configuration

1. OVMS version: 2026.2.0
2. OVMS config.json file

{
  "mediapipe_config_list": [
    {
      "name": "Qwen3.5-35B-A3B-int4-ov",
      "base_path": "C:\\models\\Qwen3.5-35B-A3B-int4-ov"
    }
  ],
  "model_config_list": []
}

3. CPU, accelerator's versions if applicable

Intel(R) Core(TM) Ultra X7 358H, 1900 Mhz, 16 Core(s), 16 Logical Processor(s)
Intel(R) Arc(TM) B390 GPU
32 GB 8533Mt/s
Driver version `32.0.101.8801`

4. Model repository directory structure

C:.
|   .gitattributes
|   chat_template.jinja
|   config.json
|   generation_config.json
|   graph.pbtxt
|   model_config.json
|   openvino_config.json
|   openvino_detokenizer.bin
|   openvino_detokenizer.xml
|   openvino_language_model.bin
|   openvino_language_model.xml
|   openvino_text_embeddings_model.bin
|   openvino_text_embeddings_model.xml
|   openvino_tokenizer.bin
|   openvino_tokenizer.xml
|   openvino_vision_embeddings_merger_model.bin
|   openvino_vision_embeddings_merger_model.xml
|   openvino_vision_embeddings_model.bin
|   openvino_vision_embeddings_model.xml
|   openvino_vision_embeddings_pos_model.bin
|   openvino_vision_embeddings_pos_model.xml
|   preprocessor_config.json
|   processor_config.json
|   README.md
|   tokenizer.json
|   tokenizer_config.json
|   
\---.cache
    \---huggingface
        |   .gitignore
        |   
        \---download
                .gitattributes.metadata
                chat_template.jinja.metadata
                config.json.metadata
                generation_config.json.metadata
                openvino_config.json.metadata
                openvino_detokenizer.bin.metadata
                openvino_detokenizer.xml.metadata
                openvino_language_model.bin.metadata
                openvino_language_model.xml.metadata
                openvino_text_embeddings_model.bin.metadata
                openvino_text_embeddings_model.xml.metadata
                openvino_tokenizer.bin.metadata
                openvino_tokenizer.xml.metadata
                openvino_vision_embeddings_merger_model.bin.metadata
                openvino_vision_embeddings_merger_model.xml.metadata
                openvino_vision_embeddings_model.bin.metadata
                openvino_vision_embeddings_model.xml.metadata
                openvino_vision_embeddings_pos_model.bin.metadata
                openvino_vision_embeddings_pos_model.xml.metadata
                preprocessor_config.json.metadata
                processor_config.json.metadata
                README.md.metadata
                tokenizer.json.metadata
                tokenizer_config.json.metadata

5. Model or publicly available similar model that reproduces the issue

OpenVINO/Qwen3.5-35B-A3B-int4-ov

Additional context
graph.pbtxt:

input_stream: "HTTP_REQUEST_PAYLOAD:input"
output_stream: "HTTP_RESPONSE_PAYLOAD:output"

node: {
  name: "LLMExecutor"
  calculator: "HttpLLMCalculator"
  input_stream: "LOOPBACK:loopback"
  input_stream: "HTTP_REQUEST_PAYLOAD:input"
  input_side_packet: "LLM_NODE_RESOURCES:llm"
  output_stream: "LOOPBACK:loopback"
  output_stream: "HTTP_RESPONSE_PAYLOAD:output"
  input_stream_info: {
    tag_index: 'LOOPBACK:0'
    back_edge: true
  }
  node_options: {
    [type.googleapis.com/mediapipe.LLMCalculatorOptions]: {
      models_path: "./"
      plugin_config: '{}'
      enable_prefix_caching: true
      dynamic_split_fuse: false
      max_num_seqs: 15
      max_num_batched_tokens: 131072
      device: "GPU"
    }
  }
  input_stream_handler {
    input_stream_handler: "SyncSetInputStreamHandler"
    options {
      [mediapipe.SyncSetInputStreamHandlerOptions.ext] {
        sync_set {
          tag_index: "LOOPBACK:0"
        }
      }
    }
  }
}

pycurl.py

import json
import requests
 
# 1. Configure the API endpoint and model parameters
url = "http://127.0.0.1:8180/v3/chat/completions"
model_name = "Qwen3.5-35B-A3B-int4-ov"
base_sentence = "What is OpenVINO? Please explain with examples. " # This has 11 tokens
 

num_repetitions = 1489
prompt_content = base_sentence * num_repetitions
 
 
 
# 3. Construct the payload
payload = {
    "model": model_name,
    "messages": [
        {
            "role": "user",
            "content": prompt_content
        }
    ],
    "max_tokens": 512,
    "temperature": 0
}
 
headers = {
    "Content-Type": "application/json"
}
 
# 4. Send the request
try:
    response = requests.post(url, headers=headers, json=payload)
    if response.status_code == 200:
        print("Success! Response from server:")
        print(json.dumps(response.json(), indent=2))
    else:
        print(f"Failed with status code: {response.status_code}")
        print(response.text)
except requests.exceptions.RequestException as e:
    print(f"An error occurred while connecting to the server: {e}")

[mzegla]

This model support is actively worked on and improved. Could you try a package from newer release?
https://github.com/openvinotoolkit/model_server/releases/tag/v2026.2.1

[TayQiXiang]

Hi @mzegla , I tested the latest release and still encountered the same issue. Here are the logs:

[exzile]

@TayQiXiang @mzegla
Tested this locally against the actual OpenVINO/Qwen3.5-35B-A3B-int4-ov checkpoint on GPU (Intel Arc Pro B70), and I believe this is the same root cause as a bug I've been tracking upstream for hybrid SSM/linear-attention models (the Qwen3.5/3.6 family): openvino.genai#4050, merged today.

Root cause: under continuous batching with enable_prefix_caching: true (as in your graph.pbtxt), the full recurrent linear-attention state gets checkpointed every cache_interval tokens. The old fixed interval made that checkpoint cache dwarf the KV cache on long prompts, exhausting the memory budget — which lines up with your title ("high memory pressure") and 16k+ token repro.

What I confirmed (before/after, same model/prompt/GPU, only the genai commit differs):

• Built genai at the commit right before the fix (9a1fbf0) and at the fix (2e9631f8), both from source.
• Same 16,380-token prompt (your pycurl.py sentence repeated), same GPU, cache budget deliberately constrained to reproduce the memory-pressure condition.
• Before the fix: request silently dropped, 0 tokens generated, no error.
• After the fix: normal, coherent ~500-token response, matching the unconstrained baseline.

Caveat: what I reproduced is empty output, not the exact !!!! gibberish you're seeing — both are documented failure modes of the same LA-cache-exhaustion path, but I haven't yet nailed the precise gibberish variant (may need a narrower memory-pressure band, or your exact GPU/VRAM size to hit it precisely — Arc B390 has meaningfully less VRAM than what I tested on).

This fix is not in any release yet (merged today, after 2026.2.1) and isn't in model_server's current vendored genai pin either. If you're able to build model_server/genai from source, or want to try genai directly from master (commit 2e9631f8 or later) against this same model/prompt, that would help confirm whether it resolves your exact symptom too.