OVMS 2026.2.1 cannot load Qwen3.6-35B-A3B to 2xA770 with HETERO pipeline parallel

[notsyncing]

Describe the bug
Hello, I'm using OVMS to load the Qwen3.6-35B-A3B model (https://huggingface.co/OpenVINO/Qwen3.6-35B-A3B-int4-ov) to 2x A770 16GB with HETERO: GPU.0, GPU.1 and MODEL_DISTRIBUTION_POLICY: PIPELINE_PARALLEL, but it failed:

With pipeline_type: VLM:

[2026-07-02 07:00:10.990][1][modelmanager][info][modelmanager.cpp:180] Available devices for Open VINO: CPU, GPU.0, GPU.1
[2026-07-02 07:00:10.991][1][serving][info][modelmanager.cpp:171] Running inside Docker container
[2026-07-02 07:00:10.991][1][serving][info][modelmanager.cpp:172] cpu quota: 0, cpu affinity: 28, max_open_files: 1024
[2026-07-02 07:00:10.991][1][serving][info][capimodule.cpp:40] C-APIModule starting
...
[2026-07-02 07:00:11.043][1][serving][info][httpservermodule.cpp:53] Started REST server at 0.0.0.0:9000
[2026-07-02 07:00:11.043][1][serving][info][servablemanagermodule.cpp:51] ServableManagerModule starting
[2026-07-02 07:00:11.045][1][serving][info][mediapipegraphdefinition.cpp:101] Graph queue globally disabled via OVMS_GRAPH_QUEUE_OFF=1 for mediapipe: Qwen/Qwen3.6-35B-A3B
[2026-07-02 07:00:11.045][1][serving][info][mediapipegraphdefinition.cpp:479] MediapipeGraphDefinition initializing graph nodes
[2026-07-02 07:00:11.045][1][modelmanager][info][servable_initializer.cpp:467] Initializing Visual Language Model Legacy servable
[2026-07-02 07:01:23.890][1][serving][error][servable_initializer.cpp:88] Error during llm node initialization for models_path: /models/OpenVINO/Qwen3.6-35B-A3B-int4-ov/./ exception: Exception from src/inference/src/cpp/core.cpp:117:
Exception from src/inference/src/dev/plugin.cpp:54:
Exception from src/plugins/hetero/src/compiled_model.cpp:36:
Standard exception from compilation library: Exception from src/inference/src/dev/plugin.cpp:54:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:168:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:385:
[GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES
[2026-07-02 07:01:23.890][1][modelmanager][error][servable_initializer.cpp:471] Error during LLM node resources initialization: The LLM Node resource initialization failed
[2026-07-02 07:01:23.890][1][serving][error][llm_node_initializer.cpp:60] Failed to process LLM node graph Qwen/Qwen3.6-35B-A3B
[2026-07-02 07:01:23.890][1][modelmanager][info][pipelinedefinitionstatus.hpp:59] Mediapipe: Qwen/Qwen3.6-35B-A3B state changed to: LOADING_PRECONDITION_FAILED after handling: ValidationFailedEvent: 

In this situation, one GPU is full (100% VRAM), another is empty.

With pipeline_type VLM_CB:

[2026-07-02 07:20:39.108][1][modelmanager][info][servable_initializer.cpp:450] Initializing Visual Language Model Continuous Batching servable
[2026-07-02 07:21:21.639][1][serving][error][servable_initializer.cpp:240] Error during llm node initialization for models_path: /models/OpenVINO/Qwen3.6-35B-A3B-int4-ov/./ exception: Exception from src/inference/src/cpp/core.cpp:117:
Exception from src/inference/src/dev/plugin.cpp:54:
Exception from src/plugins/hetero/src/compiled_model.cpp:36:
Standard exception from compilation library: Exception from src/inference/src/dev/plugin.cpp:54:
Exception from src/plugins/intel_gpu/src/plugin/ops/moe.cpp:130:
[GPU] MOECompressed (GEMM3_SWIGLU) reached the GPU backend without being fused: FuseMOE3GemmCompressed transformation did not match the routing subgraph for op '__module.model.model.language_model.layers.19.mlp/aten::sum/ReduceSum_Reshape/MOECompressed'. Please check the routing pattern.
[2026-07-02 07:21:21.639][1][modelmanager][error][servable_initializer.cpp:455] Error during LLM node resources initialization: The LLM Node resource initialization failed
[2026-07-02 07:21:21.639][1][serving][error][llm_node_initializer.cpp:60] Failed to process LLM node graph Qwen/Qwen3.6-35B-A3B
[2026-07-02 07:21:21.639][1][modelmanager][info][pipelinedefinitionstatus.hpp:59] Mediapipe: Qwen/Qwen3.6-35B-A3B state changed to: LOADING_PRECONDITION_FAILED after handling: ValidationFailedEvent: 

To Reproduce
Steps to reproduce the behavior:

1. Download model from https://huggingface.co/OpenVINO/Qwen3.6-35B-A3B-int4-ov
2. OVMS launch command --config_path /configs/model_config.json --rest_port 9000
3. See error

Expected behavior
Model loads and works.

Logs
debug log with VLM_CB: ovms_log_vlm_cb.txt
debug log with VLM: ovms_log_vlm.txt

Configuration

1. OVMS 2026.2.1 (openvino/model_server:2026.2.1-gpu)
2. CPU i9-10940X, 2x A770 16GB
3. Model: https://huggingface.co/OpenVINO/Qwen3.6-35B-A3B-int4-ov
4. graph.pbtxt: graph.txt
5. OS: Ubuntu 26.04 LTS, kernel 7.0.0-27-generic

[atobiszei]

Hi @notsyncing,
What is your gpu driver version?
Did you try to reproduce issue in OpenVINO GenAI?

[notsyncing]

I'm launching OVMS in docker. The driver version on host is 26.18.38308.4.

With OpenVINO GenAI, the error is different:

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    pipe = ov_genai.LLMPipeline("/data/ovms/model_repo/OpenVINO/Qwen3.6-35B-A3B-int4-ov", "HETERO:GPU.0,GPU.1")
RuntimeError: Exception from src/inference/src/cpp/core.cpp:117:
Exception from src/inference/src/dev/plugin.cpp:54:
Exception from src/plugins/hetero/src/compiled_model.cpp:36:
Standard exception from compilation library: Exception from src/inference/src/dev/plugin.cpp:54:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:168:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_memory.cpp:569:
[GPU] clWaitForEvents, error code: -58 CL_INVALID_EVENT






Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import sys; from multiprocessing.forkserver import main; main(4, 6, ['__main__'], sys_argv=sys.argv[1:], **{'sys_path': ['/usr/lib/python314.zip', '/usr/lib/python3.14', '/usr/lib/python3.14/lib-dynload', 'venv/lib/python3.14/site-packages'], 'main_path': 'test.py', 'authkey_r': 8})
                                                             ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.14/multiprocessing/forkserver.py", line 230, in main
    spawn.import_main_path(main_path)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/lib/python3.14/multiprocessing/spawn.py", line 307, in import_main_path
    _fixup_main_from_path(main_path)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/lib/python3.14/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                                  run_name="__mp_main__")
  File "<frozen runpy>", line 287, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "test.py", line 3, in <module>
    pipe = ov_genai.LLMPipeline("/data/ovms/model_repo/OpenVINO/Qwen3.6-35B-A3B-int4-ov", "HETERO:GPU.0,GPU.1")
RuntimeError: Exception from src/inference/src/cpp/core.cpp:117:
Exception from src/inference/src/dev/plugin.cpp:54:
Exception from src/plugins/hetero/src/compiled_model.cpp:36:
Standard exception from compilation library: Exception from src/inference/src/dev/plugin.cpp:54:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:168:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_memory.cpp:569:
[GPU] clWaitForEvents, error code: -58 CL_INVALID_EVENT

test.py:

import openvino_genai as ov_genai

pipe = ov_genai.LLMPipeline("/data/ovms/model_repo/OpenVINO/Qwen3.6-35B-A3B-int4-ov", "HETERO:GPU.0,GPU.1")
print(pipe.generate("What is OpenVINO?", max_new_tokens=100))

[notsyncing]

The script above missed the MODEL_DISTRIBUTION_POLICY parameter. After added, it has the same error:

test.py:8: DeprecationWarning: 'config' parameters is deprecated, please use kwargs to pass config properties instead.
  pipe = genai.LLMPipeline(
Traceback (most recent call last):
  File "test.py", line 8, in <module>
    pipe = genai.LLMPipeline(
RuntimeError: Exception from src/inference/src/cpp/core.cpp:117:
Exception from src/inference/src/dev/plugin.cpp:54:
Exception from src/plugins/hetero/src/compiled_model.cpp:36:
Standard exception from compilation library: Exception from src/inference/src/dev/plugin.cpp:54:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:168:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:385:
[GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES

code:

import openvino.properties.hint as hints
import openvino_genai as genai

config = {
    hints.model_distribution_policy: "PIPELINE_PARALLEL"
}

pipe = genai.LLMPipeline(
    "/data/ovms/model_repo/OpenVINO/Qwen3.6-35B-A3B-int4-ov",
    device="HETERO:GPU.0,GPU.1",
    config=config
)
print(pipe.generate("What is OpenVINO?", max_new_tokens=100))

[jpongallo1]

I was able to get this to load. My issue is I didn't check exactly why. It has all my current PR's inside it, so I would expect this to be fixed on the next release or when a few of my PR's are merged in. @atobiszei @notsyncing

If needed I can track down the exact fix.

[atobiszei]

The script above missed the MODEL_DISTRIBUTION_POLICY parameter. After added, it has the same error:

test.py:8: DeprecationWarning: 'config' parameters is deprecated, please use kwargs to pass config properties instead.
  pipe = genai.LLMPipeline(
Traceback (most recent call last):
  File "test.py", line 8, in <module>
    pipe = genai.LLMPipeline(
RuntimeError: Exception from src/inference/src/cpp/core.cpp:117:
Exception from src/inference/src/dev/plugin.cpp:54:
Exception from src/plugins/hetero/src/compiled_model.cpp:36:
Standard exception from compilation library: Exception from src/inference/src/dev/plugin.cpp:54:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:168:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:385:
[GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES

code:

import openvino.properties.hint as hints
import openvino_genai as genai

config = {
hints.model_distribution_policy: "PIPELINE_PARALLEL"
}

pipe = genai.LLMPipeline(
"/data/ovms/model_repo/OpenVINO/Qwen3.6-35B-A3B-int4-ov",
device="HETERO:GPU.0,GPU.1",
config=config
)
print(pipe.generate("What is OpenVINO?", max_new_tokens=100))

@peterchen-intel Here we have GenAI reproducer

[wgzintel]

@notsyncing I ran on DG2 machine https://gtax-ril-fm.intel.com/#/clients/4817
CPU: 13th Gen Intel(R) Core(TM) i5-13400
GPU0, GPU1: Intel(R) Arc(TM) A770 Graphics
GPU driver: 32.0.101.7028
OS: windows 11
genai version: 2026.3.0.0-3251-8f328b186b7
ov runtime version: 2026.3.0-22407-a085b841635

(1) Run with HETERO:GPU.0,GPU.1 PIPELINE_PARALLEL
(py312_hetero_test) c:\wgz>python test_qwen3.6-35b.py
genai version: 2026.3.0.0-3251-8f328b186b7, ov runtime version:2026.3.0-22407-a085b841635
Device selected: HETERO:GPU.0,GPU.1
c:\wgz\test_qwen3.6-35b.py:17: DeprecationWarning: 'config' parameters is deprecated, please use kwargs to pass config properties instead.
pipe = genai.LLMPipeline(
Traceback (most recent call last):
File "c:\wgz\test_qwen3.6-35b.py", line 22, in
print(pipe.generate("What is OpenVINO?", max_new_tokens=100))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Exception from src\core\src\runtime\tensor.cpp:97:
Exception from src\plugins\intel_gpu\src\runtime\ocl\ocl_memory.cpp:149:
[GPU] clEnqueueWriteBuffer, error code: -5 CL_OUT_OF_RESOURCES

The error isn't "ProgramBuilder build failed!"

(2) Run with GPU.0
(py312_hetero_test) c:\wgz>python test_qwen3.6-35b.py
genai version: 2026.3.0.0-3251-8f328b186b7, ov runtime version:2026.3.0-22407-a085b841635
Device selected: GPU.0
Traceback (most recent call last):
File "c:\wgz\test_qwen3.6-35b.py", line 22, in
print(pipe.generate("What is OpenVINO?", max_new_tokens=100))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Exception from src\inference\src\cpp\infer_request.cpp:80:
Check '::getPort(port, name, {_impl->get_inputs(), _impl->get_outputs()})' failed at src\inference\src\cpp\infer_request.cpp:80:
Port for tensor name input_ids was not found.

Can you run Qwen3.6-35B-A3B-int4-ov successfully with GPU.0 only?

My test code:

import openvino.properties.hint as hints
import openvino_genai as genai
from openvino import get_version
print(f"genai version: {genai.version}, ov runtime version:{get_version()}")
config = {
hints.model_distribution_policy: "PIPELINE_PARALLEL"
}
device = "GPU.0"
print('Device selected:', device)
model_path = "C:\wgz\model\Qwen3.6-35B-A3B-int4-ov\"
pipe = genai.LLMPipeline(
model_path,
device,
config=config
)
print(pipe.generate("What is OpenVINO?", max_new_tokens=100))

[notsyncing]

src\core\src\runtime\tensor.cpp:97: Exception from src\plugins\intel_gpu\src\runtime\ocl\ocl_memory.cpp:149: [GPU] clEnqueueWriteBuffer, error code: -5 CL_OUT_OF_RESOURCES

The error isn't "ProgramBuilder build failed!"

The error code of mine is the same as yours CL_OUT_OF_RESOURCES, but on a different method clFinish, see my logs above.

Can you run Qwen3.6-35B-A3B-int4-ov successfully with GPU.0 only?

With device: "GPU.0" in your script:

genai version: 2026.2.1.0-3123-7dea0459b2a, ov runtime version:2026.2.1-21919-ede283a88e3-releases/2026/2
Device selected: GPU.0
genai version: 2026.2.1.0-3123-7dea0459b2a, ov runtime version:2026.2.1-21919-ede283a88e3-releases/2026/2
Device selected: GPU.0
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    pipe = genai.LLMPipeline(
        "/data/ovms/model_repo/OpenVINO/Qwen3.6-35B-A3B-int4-ov",
        device,
        config=config
    )
RuntimeError: Exception from src/inference/src/cpp/core.cpp:117:
Exception from src/inference/src/dev/plugin.cpp:54:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:168:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_memory.cpp:569:
[GPU] clWaitForEvents, error code: -58 CL_INVALID_EVENT

BTW, I'm using the xe kernel driver. After switch to i915 driver, I got the same error as you:

RuntimeError: Exception from src/inference/src/cpp/infer_request.cpp:75:
Check '::getPort(port, name, {_impl->get_inputs(), _impl->get_outputs()})' failed at src/inference/src/cpp/infer_request.cpp:77:
Port for tensor name input_ids was not found.

[wgzintel]

@notsyncing Could you try running Qwen3.6-35B-A3B-int4-ov on the CPU alone? See if it runs successfully? What machine do you test on?
From my test, CPU is failed same with GPU.0
(py312_hetero_test) c:\wgz>python test_qwen3.6-35b.py
genai version: 2026.3.0.0-3251-8f328b186b7, ov runtime version:2026.3.0-22407-a085b841635
Device selected: CPU
Traceback (most recent call last):
File "c:\wgz\test_qwen3.6-35b.py", line 22, in
print(pipe.generate("What is OpenVINO?", max_new_tokens=100))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Exception from src\inference\src\cpp\infer_request.cpp:80:
Check '::getPort(port, name, {_impl->get_inputs(), _impl->get_outputs()})' failed at src\inference\src\cpp\infer_request.cpp:80:
Port for tensor name input_ids was not found.

[notsyncing]

@notsyncing Could you try running Qwen3.6-35B-A3B-int4-ov on the CPU alone? See if it runs successfully? What machine do you test on? From my test, CPU is failed same with GPU.0 (py312_hetero_test) c:\wgz>python test_qwen3.6-35b.py genai version: 2026.3.0.0-3251-8f328b186b7, ov runtime version:2026.3.0-22407-a085b841635 Device selected: CPU Traceback (most recent call last): File "c:\wgz\test_qwen3.6-35b.py", line 22, in print(pipe.generate("What is OpenVINO?", max_new_tokens=100)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Exception from src\inference\src\cpp\infer_request.cpp:80: Check '::getPort(port, name, {_impl->get_inputs(), _impl->get_outputs()})' failed at src\inference\src\cpp\infer_request.cpp:80: Port for tensor name input_ids was not found.

Got same error on CPU only.

RuntimeError: Exception from src/inference/src/cpp/infer_request.cpp:75:
Check '::getPort(port, name, {_impl->get_inputs(), _impl->get_outputs()})' failed at src/inference/src/cpp/infer_request.cpp:77:
Port for tensor name input_ids was not found.

My machine CPU is i9-10940X, with 2x A770 16GB on two PCIE 3.0 x8 ports.