RHIDP-12707: Update docs and add E2E tests for MCP APIs

docs/ARCHITECTURE.md

@@ -108,7 +108,7 @@ This section describes the major functional components that make up LCore. Each
 - **Lifecycle Management**: 
   - **Startup**: Load configuration, initialize Llama Stack client, load MCP server configuration and register all defined servers with Llama Stack to build the tools list, establish database connections
   - **Shutdown**: Clean up A2A storage resources (database connections and other resources are cleaned up automatically by Python's context managers)
-- **Router Registration**: Mount all endpoint routers (query, conversation, model info, auth, metrics, A2A, feedback, admin, mcp_auth)
+- **Router Registration**: Mount all endpoint routers (query, conversation, model info, auth, metrics, A2A, feedback, admin, mcp_auth, mcp_servers)
 
 **Note:** All configured MCP servers must be running and accessible at startup time for LCore to initialize successfully.
 
@@ -207,6 +207,9 @@ The system defines 30+ actions that can be authorized. Examples (see `docs/auth.
 **Agent-to-Agent Protocol:**
 - `A2A_JSONRPC` - A2A protocol access
 
+**MCP Server Management:**
+- `REGISTER_MCP_SERVER`, `LIST_MCP_SERVERS`, `DELETE_MCP_SERVER`
+
 **Metadata Operations:**
 - `LIST_MODELS`, `LIST_SHIELDS`, `LIST_TOOLS`, `LIST_PROVIDERS`
 
@@ -325,7 +328,7 @@ MCP servers are remote HTTP services that expose tools/capabilities to LLMs (e.g
 
 **How It Works:**
 
-1. **Configuration:** MCP servers are defined in the config file with name, URL, and authorization headers
+1. **Configuration:** MCP servers are defined in the config file with name, URL, and authorization headers. Servers can also be registered dynamically at runtime via `POST /v1/mcp-servers`.
 2. **Registration at Startup:** LCore tells Llama Stack about each MCP server by calling `toolgroups.register()` - this makes the MCP server's tools available in Llama Stack's tool registry
 3. **Query Processing:** When processing a query, LCore determines which tools to make available to the LLM and finalizes authorization headers (e.g., merging client-provided tokens with configured headers)
 4. **Tool Execution:** When the LLM calls a tool, Llama Stack routes the request to the appropriate MCP server URL with the finalized authorization headers
@@ -489,6 +492,11 @@ This section documents the REST API endpoints exposed by LCore for client intera
 - Returns MCP servers that accept client-provided authentication tokens
 - Includes header names that need to be provided via MCP-HEADERS
 
+**MCP Server Management:**
+- `POST /v1/mcp-servers` - Register a new MCP server at runtime
+- `GET /v1/mcp-servers` - List all registered MCP servers (static and dynamic)
+- `DELETE /v1/mcp-servers/{name}` - Unregister a dynamically registered MCP server
+
 **List Shields:** `GET /shields`
 - Returns available guardrails
 

docs/config.md

@@ -159,7 +159,7 @@ Global service configuration.
 | llama_stack |  | This section contains Llama Stack configuration. Lightspeed Core Stack service can call Llama Stack in library mode or in server mode. |
 | user_data_collection |  | This section contains configuration for subsystem that collects user data(transcription history and feedbacks). |
 | database |  | Configuration for database to store conversation IDs and other runtime data |
-| mcp_servers | array | MCP (Model Context Protocol) servers provide tools and capabilities to the AI agents. These are configured in this section. Only MCP servers defined in the lightspeed-stack.yaml configuration are available to the agents. Tools configured in the llama-stack run.yaml are not accessible to lightspeed-core agents. |
+| mcp_servers | array | MCP (Model Context Protocol) servers provide tools and capabilities to the AI agents. These are configured in this section. Servers can also be registered dynamically at runtime via the `POST /v1/mcp-servers` API endpoint. Only MCP servers defined in lightspeed-stack.yaml or registered via the API are available to the agents. Tools configured in the llama-stack run.yaml are not accessible to lightspeed-core agents. |
 | authentication |  | Authentication configuration |
 | authorization |  | Lightspeed Core Stack implements a modular authentication and authorization system with multiple authentication methods. Authorization is configurable through role-based access control. Authentication is handled through selectable modules configured via the module field in the authentication configuration. |
 | customization |  | It is possible to customize Lightspeed Core Stack via this section. System prompt can be customized and also different parts of the service can be replaced by custom Python modules. |
@@ -359,10 +359,10 @@ Useful resources:
 Model context protocol server configuration.
 
 MCP (Model Context Protocol) servers provide tools and capabilities to the
-AI agents. These are configured by this structure. Only MCP servers
-defined in the lightspeed-stack.yaml configuration are available to the
-agents. Tools configured in the llama-stack run.yaml are not accessible to
-lightspeed-core agents.
+AI agents. These are configured by this structure. MCP servers defined in
+lightspeed-stack.yaml and servers registered at runtime via the
+`POST /v1/mcp-servers` API are available to agents. Tools configured in
+the llama-stack run.yaml are not accessible to lightspeed-core agents.
 
 Useful resources:
 

docs/getting_started.md

@@ -240,7 +240,7 @@ mcp_servers:
     url: "http://localhost:3002"
 ```
 
-**Important**: Only MCP servers defined in the `lightspeed-stack.yaml` configuration are available to the AI agents. Tools configured in the llama-stack `run.yaml` are not accessible to LCS agents.
+**Important**: MCP servers defined in `lightspeed-stack.yaml` or registered dynamically via the API (see [Dynamic MCP Server Management](#dynamic-mcp-server-management-via-api)) are available to the AI agents. Tools configured in the llama-stack `run.yaml` are not accessible to LCS agents.
 
 #### Step 3: Pass authentication or metadata via MCP headers (optional)
 
@@ -255,4 +255,84 @@ curl -X POST "http://localhost:8080/v1/query" \
 ```
 
 #### Step 4: Verify connectivity
-After starting the MCP servers and updating `lightspeed-stack.yaml`, test by sending a prompt to the AI agent. LCS evaluates the prompt against available tools’ metadata, selects the appropriate tool, calls the corresponding MCP server, and uses the result to generate more accurate agent response.
+After starting the MCP servers and updating `lightspeed-stack.yaml`, test by sending a prompt to the AI agent. LCS evaluates the prompt against available tools' metadata, selects the appropriate tool, calls the corresponding MCP server, and uses the result to generate more accurate agent response.
+
+### Dynamic MCP Server Management via API
+
+In addition to static configuration in `lightspeed-stack.yaml`, MCP servers can be registered, listed, and removed at runtime through the REST API. This is useful for development, testing, or scenarios where MCP servers are provisioned dynamically.
+
+When authorization is enabled, callers need the following permissions:
+- `REGISTER_MCP_SERVER` for `POST /v1/mcp-servers`
+- `LIST_MCP_SERVERS` for `GET /v1/mcp-servers`
+- `DELETE_MCP_SERVER` for `DELETE /v1/mcp-servers/{name}`
+
+#### Register an MCP server
+
+```bash
+curl -X POST "http://localhost:8080/v1/mcp-servers" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "name": "my-dynamic-tools",
+    "url": "http://localhost:9000/mcp",
+    "provider_id": "model-context-protocol"
+  }'
+```
+
+Response (201 Created):
+```json
+{
+  "name": "my-dynamic-tools",
+  "url": "http://localhost:9000/mcp",
+  "provider_id": "model-context-protocol",
+  "message": "MCP server 'my-dynamic-tools' registered successfully"
+}
+```
+
+Optional fields in the request body:
+- `authorization_headers` (object): Headers to send to the MCP server (e.g., `{"Authorization": "Bearer token123"}`).
+- `headers` (array): List of HTTP header names to forward from incoming requests to this MCP server.
+- `timeout` (integer): Request timeout in seconds.
+
+#### List all MCP servers
+
+```bash
+curl "http://localhost:8080/v1/mcp-servers"
+```
+
+Response (200 OK):
+```json
+{
+  "servers": [
+    {
+      "name": "filesystem-tools",
+      "url": "http://localhost:3000",
+      "provider_id": "model-context-protocol",
+      "source": "config"
+    },
+    {
+      "name": "my-dynamic-tools",
+      "url": "http://localhost:9000/mcp",
+      "provider_id": "model-context-protocol",
+      "source": "api"
+    }
+  ]
+}
+```
+
+Each server has a `source` field indicating how it was registered: `"config"` for servers defined in `lightspeed-stack.yaml`, or `"api"` for servers registered via the REST API.
+
+#### Delete a dynamically registered MCP server
+
+```bash
+curl -X DELETE "http://localhost:8080/v1/mcp-servers/my-dynamic-tools"
+```
+
+Response (200 OK):
+```json
+{
+  "name": "my-dynamic-tools",
+  "message": "MCP server 'my-dynamic-tools' unregistered successfully"
+}
+```
+
+**Note:** Only dynamically registered servers (source `"api"`) can be deleted via the API. Attempting to delete a statically configured server returns 403 Forbidden. Dynamically registered servers do not persist across service restarts.

tests/e2e/configuration/library-mode/lightspeed-stack-mcp-auth.yaml

@@ -0,0 +1,25 @@
+name: Lightspeed Core Service (LCS)
+service:
+  host: 0.0.0.0
+  port: 8080
+  auth_enabled: false
+  workers: 1
+  color_log: true
+  access_log: true
+llama_stack:
+  # Library mode - embeds llama-stack as library
+  use_as_library_client: true
+  library_client_config_path: run.yaml
+user_data_collection:
+  feedback_enabled: true
+  feedback_storage: "/tmp/data/feedback"
+  transcripts_enabled: true
+  transcripts_storage: "/tmp/data/transcripts"
+authentication:
+  module: "noop-with-token"
+mcp_servers:
+  - name: "mcp-oauth"
+    provider_id: "model-context-protocol"
+    url: "http://mock-mcp:3001"
+    authorization_headers:
+      Authorization: "oauth"

tests/e2e/configuration/library-mode/lightspeed-stack-no-mcp.yaml

@@ -0,0 +1,19 @@
+name: Lightspeed Core Service (LCS)
+service:
+  host: 0.0.0.0
+  port: 8080
+  auth_enabled: false
+  workers: 1
+  color_log: true
+  access_log: true
+llama_stack:
+  # Library mode - embeds llama-stack as library
+  use_as_library_client: true
+  library_client_config_path: run.yaml
+user_data_collection:
+  feedback_enabled: true
+  feedback_storage: "/tmp/data/feedback"
+  transcripts_enabled: true
+  transcripts_storage: "/tmp/data/transcripts"
+authentication:
+  module: "noop"

tests/e2e/configuration/server-mode/lightspeed-stack-mcp-auth.yaml

@@ -0,0 +1,26 @@
+name: Lightspeed Core Service (LCS)
+service:
+  host: 0.0.0.0
+  port: 8080
+  auth_enabled: false
+  workers: 1
+  color_log: true
+  access_log: true
+llama_stack:
+  # Server mode - connects to separate llama-stack service
+  use_as_library_client: false
+  url: http://llama-stack:8321
+  api_key: xyzzy
+user_data_collection:
+  feedback_enabled: true
+  feedback_storage: "/tmp/data/feedback"
+  transcripts_enabled: true
+  transcripts_storage: "/tmp/data/transcripts"
+authentication:
+  module: "noop-with-token"
+mcp_servers:
+  - name: "mcp-oauth"
+    provider_id: "model-context-protocol"
+    url: "http://mock-mcp:3001"
+    authorization_headers:
+      Authorization: "oauth"

tests/e2e/configuration/server-mode/lightspeed-stack-no-mcp.yaml

@@ -0,0 +1,20 @@
+name: Lightspeed Core Service (LCS)
+service:
+  host: 0.0.0.0
+  port: 8080
+  auth_enabled: false
+  workers: 1
+  color_log: true
+  access_log: true
+llama_stack:
+  # Server mode - connects to separate llama-stack service
+  use_as_library_client: false
+  url: http://llama-stack:8321
+  api_key: xyzzy
+user_data_collection:
+  feedback_enabled: true
+  feedback_storage: "/tmp/data/feedback"
+  transcripts_enabled: true
+  transcripts_storage: "/tmp/data/transcripts"
+authentication:
+  module: "noop"

tests/e2e/features/environment.py

@@ -76,6 +76,14 @@
         "tests/e2e/configuration/{mode_dir}/lightspeed-stack-mcp-oauth-auth.yaml",
         "tests/e2e-prow/rhoai/configs/lightspeed-stack-mcp-oauth-auth.yaml",
     ),
+    "mcp-auth": (
+        "tests/e2e/configuration/{mode_dir}/lightspeed-stack-mcp-auth.yaml",
+        "tests/e2e-prow/rhoai/configs/lightspeed-stack-mcp-auth.yaml",
+    ),
+    "no-mcp": (
+        "tests/e2e/configuration/{mode_dir}/lightspeed-stack-no-mcp.yaml",
+        "tests/e2e-prow/rhoai/configs/lightspeed-stack-no-mcp.yaml",
+    ),
 }
 
 
@@ -435,6 +443,24 @@ def before_feature(context: Context, feature: Feature) -> None:
         switch_config(context.feature_config)
         restart_container("lightspeed-stack")
 
+    if "MCPFileAuth" in feature.tags:
+        context.feature_config = _get_config_path("mcp-file-auth", mode_dir)
+        context.default_config_backup = create_config_backup("lightspeed-stack.yaml")
+        switch_config(context.feature_config)
+        restart_container("lightspeed-stack")
+
+    if "MCPServerAPIAuth" in feature.tags:
+        context.feature_config = _get_config_path("mcp-auth", mode_dir)
+        context.default_config_backup = create_config_backup("lightspeed-stack.yaml")
+        switch_config(context.feature_config)
+        restart_container("lightspeed-stack")
+
+    if "MCPNoConfig" in feature.tags:
+        context.feature_config = _get_config_path("no-mcp", mode_dir)
+        context.default_config_backup = create_config_backup("lightspeed-stack.yaml")
+        switch_config(context.feature_config)
+        restart_container("lightspeed-stack")
+
 
 def after_feature(context: Context, feature: Feature) -> None:
     """Run after each feature file is exercised.
@@ -467,3 +493,18 @@ def after_feature(context: Context, feature: Feature) -> None:
         switch_config(context.default_config_backup)
         restart_container("lightspeed-stack")
         remove_config_backup(context.default_config_backup)
+
+    if "MCPFileAuth" in feature.tags:
+        switch_config(context.default_config_backup)
+        restart_container("lightspeed-stack")
+        remove_config_backup(context.default_config_backup)
+
+    if "MCPServerAPIAuth" in feature.tags:
+        switch_config(context.default_config_backup)
+        restart_container("lightspeed-stack")
+        remove_config_backup(context.default_config_backup)
+
+    if "MCPNoConfig" in feature.tags:
+        switch_config(context.default_config_backup)
+        restart_container("lightspeed-stack")
+        remove_config_backup(context.default_config_backup)