Skip to content

CLI improvements #4368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dtrawins wants to merge 13 commits into
base: main
Choose a base branch
from
Open

CLI improvements #4368

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
df1c0cd
initial version
dtrawins Jul 12, 2026
17e3825
docs
dtrawins Jul 12, 2026
2910c33
style
dtrawins Jul 12, 2026
0a6b255
Merge remote-tracking branch 'origin/main' into configure
dtrawins Jul 16, 2026
a21302a
fix tests
dtrawins Jul 16, 2026
b589fed
more tests
dtrawins Jul 16, 2026
e6162b4
fix documentation
dtrawins Jul 16, 2026
d86d18d
fix relative path in staring ovms with model_path
dtrawins Jul 17, 2026
3f42739
style
dtrawins Jul 17, 2026
162fc8f
Merge remote-tracking branch 'origin/main' into configure
dtrawins Jul 17, 2026
a58ae06
more tests
dtrawins Jul 17, 2026
e879d17
review changes
dtrawins Jul 17, 2026
a7cd8d2
build fix
dtrawins Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 40 additions & 5 deletions docs/parameters.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ Configuration options for the server are defined only via command-line options a
| `api_key_file` | `string` | Path to the text file with the API key for generative endpoints `/v3/`. The value of first line is used. If not specified, server is using environment variable API_KEY. If not set, requests will not require authorization.|
| `allowed_local_media_path` | `string` | Path to the directory containing images to include in requests. If unset, local filesystem images in requests are not supported.|
| `allowed_media_domains` | `string` | Comma separated list of media domains from which URLs can be used as input for LLMs. Set to \"all\" to disable this restrictions. If unset, URLs in requests are not supported."
| `verbose_response` | `NA` | When enabled, responses include an extra `__verbose` object with additional debug information. Applies for text generation models |

## Config management mode options

Expand All @@ -68,10 +69,29 @@ Configuration options for the config management mode, which is used to manage co
| `list_models` | `NA` | List all models paths in the model repository. |
| `model_name` | `string` | Name of the model as visible in serving. If `--model_path` is not provided, path is deduced from name. |
| `model_path` | `string` | Optional. Path to the model repository. If path is relative then it is prefixed with `--model_repository_path`. |
| `add_to_config` | `NA` | Directive to add new model to the config file. |
| `add_to_config` | `NA` | Directive to add new model to the config file. Accepts optional model parameters: `--batch_size`, `--shape`, `--layout`, `--mean`, `--scale`, `--color_format`, `--precision`, `--model_version_policy`, `--nireq`, `--target_device`, `--plugin_config`. |
| `remove_from_config` | `NA` | Directive to remove model from the config file. |
| `config_path` | `string` | Path to the configuration file. |

## Configure mode options

Configure mode creates or updates `graph.pbtxt` for a local model without starting the server. It requires `--model_path` and `--task` parameters along with task-specific options.

| Option | Value format | Description |
|-------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| `--configure` | `NA` | Runs in configure mode to create or update `graph.pbtxt` for a local model. Does not start the server. |
| `--model_path` | `string` | Path to the local model directory where `graph.pbtxt` will be created. |
| `--model_name` | `string` | Optional. Name of the model as exposed by the server. |
| `--task` | `string` | Task type for the model (`text_generation`, `embeddings`, `rerank`, `image_generation`, `text2speech`, `speech2text`). |
| `--target_device` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"NPU"/"MULTI"/"HETERO"`. |

Task-specific options (e.g., `--max_num_seqs`, `--cache_size`, `--num_streams`) are the same as documented in the [pull mode task options](#text-generation) below.

**Example:**
```text
ovms --configure --model_path /models/my_llm --task text_generation --target_device GPU --max_num_seqs 128 --cache_size 8
```

## Pull mode configuration options

Shared configuration options for the pull, and pull & start mode. In the presence of ```--pull``` parameter OVMS will only pull model without serving.
Expand All @@ -85,7 +105,7 @@ Shared configuration options for the pull, and pull & start mode. In the presenc
| `--model_repository_path` | `string` | Directory where all required model files will be saved. |
| `--model_name` | `string` | Name of the model as exposed externally by the server. |
| `--target_device` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"MULTI"/"HETERO"` |
| `--task` | `string` | Task type the model will support (`text_generation`, `embeddings`, `rerank`, `image_generation`). |
| `--task` | `string` | Task type the model will support (`text_generation`, `embeddings`, `rerank`, `image_generation`, `text2speech`, `speech2text`). |
| `--overwrite_models` | `NA` | If set, an existing model with the same name will be overwritten. If not set, the server will use existing model files if available. |
| `--gguf_filename` | `string` | Filename of the wanted quantization type from Hugging Face GGUF repository. |

Expand Down Expand Up @@ -116,10 +136,9 @@ There are also additional environment variables that may change the behavior of
|-------------------------------------|---------|------------------------------------------------------------------------------------------------------------|
| `GIT_OPT_SET_SERVER_CONNECT_TIMEOUT`| `int` | Timeout to attempt connections to a remote server. Default value 4000 ms. |
| `GIT_OPT_SET_SERVER_TIMEOUT` | `int` | Timeout for reading from and writing to a remote server. Default value 4000 ms. |
| `GIT_OPT_SET_SSL_CERT_LOCATIONS` | `string`| Path to check for ssl certificates. |
| `GIT_OPT_SET_ENABLE_SEARCH_PATHS`| `int` | When set to 1, the pull functionality reads host-level git configuration locations like ~/.gitconfig. Default value 0. |
| `GIT_OPT_SET_SSL_CERT_LOCATIONS` | `string`| Path to check for ssl certificates. |

Task specific parameters for different tasks (text generation/image generation/embeddings/rerank) are listed below:
Task specific parameters for different tasks (text generation/image generation/embeddings/rerank/text2speech/speech2text) are listed below:

### Text generation
| option | Value format | Description |
Expand All @@ -137,6 +156,7 @@ Task specific parameters for different tasks (text generation/image generation/e
| `--reasoning_parser` | `string` | Type of parser to use for reasoning content extraction from model output. Auto-detected from chat template if not specified. Use `none` to explicitly disable. Supported: [qwen3, gptoss, lfm2, gemma4] |
| `--tool_parser` | `string` | Type of parser to use for tool calls extraction from model output. Auto-detected from chat template if not specified. Use `none` to explicitly disable. Supported: [llama3, phi4, hermes3, mistral, qwen3coder, gptoss, devstral, lfm2, gemma4] |
| `--enable_tool_guided_generation` | `bool` | Enables enforcing tool schema during generation. Requires setting response parser. Default: false. |
| `--cache_interval_multiplier` | `integer` | Multiplier for the KV cache block interval. Controls the granularity of cache allocation. Default: adaptive for the model. |

### Image generation
| option | Value format | Description |
Expand Down Expand Up @@ -165,3 +185,18 @@ Task specific parameters for different tasks (text generation/image generation/e
|---------------------------|--------------|--------------------------------------------------------------------------------|
| `--num_streams` | `integer` | The number of parallel execution streams to use for the model. Use at least 2 on 2 socket CPU systems. Default: 1. |
| `--max_allowed_chunks` | `integer` | Maximum allowed chunks. Default: 10000. |

### Text to speech
| option | Value format | Description |
|---------------------------|--------------|--------------------------------------------------------------------------------|
| `--num_streams` | `integer` | The number of parallel execution streams to use for the model. Use at least 2 on 2 socket CPU systems. Default: 1. |
| `--model_type` | `string` | Type of the source TTS model: `speecht5` (default) or `kokoro`. |
| `--vocoder` | `string` | The vocoder model to use for text2speech. For example `microsoft/speecht5_hifigan`. |

### Speech to text
| option | Value format | Description |
|---------------------------|--------------|--------------------------------------------------------------------------------|
| `--num_streams` | `integer` | The number of parallel execution streams to use for the model. Use at least 2 on 2 socket CPU systems. Default: 1. |



1 change: 1 addition & 0 deletions src/capi_frontend/server_settings.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ enum OvmsServerMode : int {
LIST_MODELS_MODE,
MODIFY_CONFIG_MODE,
IN_MEMORY_GRAPH_MODE,
CONFIGURE_MODE,
UNKNOWN_MODE
};

Expand Down
Loading