Add Meta Muse Spark computer-use provider

.agents/skills/update-models/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: update-models
-description: Discover latest OpenAI, Anthropic, Google/Gemini, Tzafon, and Yutori models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results.
+description: Discover latest OpenAI, Anthropic, Google/Gemini, Meta, Tzafon, and Yutori models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results.
 ---
 
 # Update Models
@@ -9,14 +9,14 @@ Use this workflow to keep CUA current with provider model releases and computer-
 
 ## Quick Start
 
-1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `TZAFON_API_KEY`, and `YUTORI_API_KEY`.
+1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `META_API_KEY`, `TZAFON_API_KEY`, and `YUTORI_API_KEY`.
 2. If credentials live in `~/AGENTS.md`, load them into the current shell without printing them:
 
 ```bash
 eval "$(python3 - <<'PY'
 import pathlib, re, shlex
 text = pathlib.Path('~/AGENTS.md').expanduser().read_text()
-for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'TZAFON_API_KEY', 'YUTORI_API_KEY']:
+for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'META_API_KEY', 'TZAFON_API_KEY', 'YUTORI_API_KEY']:
     m = re.search(r'export\s+' + re.escape(key) + r'=(?:"([^"]+)"|([^\s\n]+))', text)
     if m:
         print(f'export {key}={shlex.quote(m.group(1) or m.group(2))}')
@@ -68,6 +68,15 @@ When live discovery finds a new model with passing smoke tests, update `packages
 
 ## Provider Checks
 
+Meta:
+
+- Discover with the OpenAI SDK against `https://api.meta.ai/v1` using `META_API_KEY`.
+- Smoke-test the Responses API with screenshot input and explicit function tools matching CUA's canonical actions.
+- Pass condition: response output contains a `function_call` for one of the supplied browser actions.
+- Use `store: true` plus `previous_response_id` for CUA tool loops. Meta rejects `include: ["reasoning.encrypted_content"]` on requests that set `previous_response_id`.
+- Set `parallel_tool_calls: false` because browser actions mutate shared state.
+- Treat Meta computer use as custom-function-tool support, not a provider-native `{ type: "computer" }` tool.
+
 OpenAI:
 
 - Discover with `OpenAI().models.list()` and optionally `models.retrieve(modelId)`.
@@ -116,6 +125,7 @@ Yutori:
 Run action probes when updating adapters or when docs/examples show drift:
 
 ```bash
+npx tsx .agents/skills/update-models/reference/discover-models.ts --provider meta --models muse-spark-1.1
 npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5
 npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider anthropic --model claude-opus-4-7
 npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider gemini --model gemini-3-flash-preview
@@ -160,6 +170,7 @@ All CUA model and adapter support lives in `packages/ai` (`@onkernel/cua-ai`). W
   - Update the snapshot in `packages/ai/docs/supported-models.md` to match.
 
 - New provider-native action, response field, or tool version:
+  - Meta: update `packages/ai/src/providers/meta/index.ts` and `provider.ts`, including Responses threading and reasoning compatibility.
   - OpenAI: update `packages/ai/src/providers/openai/index.ts` and its action vocabulary, plus the shared canonical types in `packages/ai/src/providers/common.ts` if the action set changes.
   - Anthropic: update the `ANTHROPIC_CUA_ACTION_TYPES` set in `packages/ai/src/providers/anthropic/actions.ts` and `index.ts`. The computer tool version and `computer-use-*` beta header are selected by `pi-ai` per model, so a new dated tool version usually means bumping `@earendil-works/pi-ai`, not editing this package.
   - Gemini: update `packages/ai/src/providers/gemini/index.ts`, including coordinate handling if needed.

.agents/skills/update-models/reference/README.md

@@ -11,10 +11,11 @@ These scripts support the `update-models` skill. Run them from the repository ro
   - `OPENAI_API_KEY`
   - `ANTHROPIC_API_KEY`
   - `GOOGLE_API_KEY` or `GEMINI_API_KEY`
+  - `META_API_KEY`
   - `TZAFON_API_KEY`
   - `YUTORI_API_KEY`
 
-The scripts never print API keys. Smoke tests are non-destructive: they ask each model to emit a provider-native computer-use tool call, then inspect the response without executing the action.
+The scripts never print API keys. Smoke tests are non-destructive: they ask each model to emit a computer-use tool call, then inspect the response without executing the action. Meta uses supplied function tools; other providers may use provider-native computer tools.
 
 ## Common Commands
 

.agents/skills/update-models/reference/audit-official-examples.ts

@@ -5,7 +5,7 @@ import { basename, join, resolve } from "node:path";
 import { spawnSync } from "node:child_process";
 import process from "node:process";
 
-type Provider = "openai" | "anthropic" | "gemini" | "yutori";
+type Provider = "openai" | "anthropic" | "gemini" | "meta" | "yutori";
 
 interface ExampleRepo {
 	provider: Provider;
@@ -51,6 +51,14 @@ const EXAMPLES: ExampleRepo[] = [
 		confidence: "provider-owned",
 		patterns: ["computer_use", "ComputerUse", "function_call", "functionCall", "FunctionResponse", "safety_decision"],
 	},
+	{
+		provider: "meta",
+		name: "meta-model-cookbook",
+		repo: "https://github.com/meta-models/meta-model-cookbook.git",
+		confidence: "provider-owned",
+		pathHint: "03_use_cases/13_macos_cua",
+		patterns: ["muse-spark-1.1", "function_call", "function_call_output", "previous_response_id", "reasoning.encrypted_content", "parallel_tool_calls"],
+	},
 	{
 		provider: "yutori",
 		name: "kernel-cli-yutori-template",
@@ -65,6 +73,7 @@ const ACTION_REGEXES: Record<Provider, RegExp[]> = {
 	openai: [/\b(click|double_click|scroll|type|wait|keypress|drag|move|screenshot)\b/g],
 	anthropic: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up|cursor_position|zoom)\b/g],
 	gemini: [/\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g],
+	meta: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up)\b/g],
 	yutori: [/\b(left_click|double_click|triple_click|right_click|scroll|type|key_press|hover|drag|wait|refresh|go_back|goto_url|mouse_move|middle_click|mouse_down|mouse_up|go_forward|hold_key|extract_elements|find|set_element_value|execute_js)\b/g],
 };
 

.agents/skills/update-models/reference/discover-models.ts

@@ -1,11 +1,12 @@
 #!/usr/bin/env tsx
+import { existsSync } from "node:fs";
 import { readFile, writeFile } from "node:fs/promises";
 import { homedir } from "node:os";
 import { join } from "node:path";
 import process from "node:process";
 import { parse as parseToml } from "smol-toml";
 
-type Provider = "openai" | "anthropic" | "gemini" | "tzafon" | "yutori";
+type Provider = "openai" | "anthropic" | "gemini" | "meta" | "tzafon" | "yutori";
 
 interface Args {
 	provider: Provider | "all";
@@ -38,7 +39,7 @@ interface ModelResult {
 	cua?: Record<string, unknown>;
 }
 
-const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "tzafon", "yutori"];
+const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "meta", "tzafon", "yutori"];
 const TZAFON_KNOWN_MODELS = [
 	"tzafon.northstar-cua-fast",
 ];
@@ -109,7 +110,7 @@ function usage(): never {
   npx tsx .agents/skills/update-models/reference/discover-models.ts --provider openai --models gpt-5.5,gpt-5.4
 
 Options:
-  --provider <all|openai|anthropic|gemini|tzafon|yutori>
+  --provider <all|openai|anthropic|gemini|meta|tzafon|yutori>
   --models <comma-separated model ids>    Smoke-test explicit models instead of inferred candidates.
   --candidate-limit <n>                  Max inferred candidates per provider. Default: 20.
   --no-smoke                             Only list metadata.
@@ -137,6 +138,7 @@ async function runProvider(provider: Provider, args: Args): Promise<Record<strin
 		if (provider === "openai") return await discoverOpenAI(args);
 		if (provider === "anthropic") return await discoverAnthropic(args);
 		if (provider === "gemini") return await discoverGemini(args);
+		if (provider === "meta") return await discoverMeta(args);
 		if (provider === "tzafon") return await discoverTzafon(args);
 		if (provider === "yutori") return await discoverYutori(args);
 		throw new Error(`unknown provider ${provider satisfies never}`);
@@ -175,6 +177,79 @@ async function discoverOpenAI(args: Args): Promise<Record<string, unknown>> {
 	return { provider: "openai", metadata_source: "client.models.list()", models, candidates };
 }
 
+async function discoverMeta(args: Args): Promise<Record<string, unknown>> {
+	const OpenAI = await importDefault("openai", "OpenAI");
+	const apiKey = process.env.META_API_KEY;
+	const client = new OpenAI({ apiKey, baseURL: "https://api.meta.ai/v1" });
+	const rawModels = await collectAsync(client.models.list());
+	const models: ModelResult[] = rawModels.map((m) => ({
+		id: String(m.id),
+		display_name: String(m.id),
+		created_at: typeof m.created === "number" && m.created > 0 ? new Date(m.created * 1000).toISOString() : null,
+		raw: m,
+		supports_generation: true,
+	}));
+	const candidates = explicitOrCandidates(args, models.map((model) => model.id));
+	if (args.smoke) {
+		await Promise.all(candidates.map(async (id) => {
+			const model = models.find((candidate) => candidate.id === id) ?? { id, display_name: id, supports_generation: true };
+			model.computer_use = await smokeMeta(client, id);
+			if (!models.find((candidate) => candidate.id === id)) models.unshift(model);
+		}));
+	}
+	await annotateCuaSupport("meta", models);
+	return { provider: "meta", metadata_source: "Meta Model API models.list()", models, candidates };
+}
+
+async function smokeMeta(client: any, model: string): Promise<SmokeResult> {
+	try {
+		const screenshotPath = [
+			join(process.cwd(), "examples", "screenshot.png"),
+			join(process.cwd(), "packages", "ai", "examples", "screenshot.png"),
+		].find(existsSync);
+		if (!screenshotPath) throw new Error("could not find packages/ai/examples/screenshot.png");
+		const screenshot = await readFile(screenshotPath);
+		const response = await client.responses.create({
+			model,
+			store: false,
+			parallel_tool_calls: false,
+			max_output_tokens: 512,
+			reasoning: { effort: "low" },
+			input: [{
+				role: "user",
+				content: [
+					{ type: "input_text", text: "Call the click tool for the sign in link. Do not answer only in text." },
+					{ type: "input_image", image_url: `data:image/png;base64,${screenshot.toString("base64")}` },
+				],
+			}],
+			tools: [{
+				type: "function",
+				name: "click",
+				description: "Click at normalized 0-1000 screen coordinates.",
+				parameters: {
+					type: "object",
+					properties: { x: { type: "number" }, y: { type: "number" } },
+					required: ["x", "y"],
+					additionalProperties: false,
+				},
+			}],
+		});
+		const output: any[] = response.output ?? [];
+		const calls = output.filter((item) => item?.type === "function_call");
+		return {
+			status: calls.length > 0 ? "pass" : "inconclusive",
+			tool_name: "function_tools",
+			tool_version: null,
+			beta_header: null,
+			observed_actions: unique(calls.map((call) => call?.name).filter(Boolean)),
+			response_item_types: unique(output.map((item) => item?.type).filter(Boolean)),
+			error: null,
+		};
+	} catch (err) {
+		return smokeError(err, { tool_name: "function_tools" });
+	}
+}
+
 function likelyOpenAIGenerationModel(id: string): boolean {
 	const lower = id.toLowerCase();
 	if (OPENAI_EXCLUDE.some((needle) => lower.includes(needle))) return false;

.agents/skills/update-models/reference/provider-doc-drift.ts

@@ -2,7 +2,7 @@
 import { readFile, writeFile } from "node:fs/promises";
 import process from "node:process";
 
-type Provider = "openai" | "anthropic" | "gemini" | "yutori";
+type Provider = "openai" | "anthropic" | "gemini" | "meta" | "yutori";
 
 interface Args {
 	examples: string;
@@ -30,6 +30,11 @@ const DOCS: Record<Provider, string[]> = {
 		"https://ai.google.dev/gemini-api/docs/computer-use",
 		"https://ai.google.dev/api/models",
 	],
+	meta: [
+		"https://dev.meta.ai/docs/getting-started/cookbook/computer-use-macos",
+		"https://dev.meta.ai/docs/features/responses",
+		"https://dev.meta.ai/docs/features/tool-calling",
+	],
 	yutori: [
 		"https://docs.yutori.com/reference/navigator",
 		"https://docs.yutori.com/reference/n1",
@@ -42,13 +47,15 @@ const LOCAL_FILES: Record<Provider, string> = {
 	openai: "packages/ai/src/providers/openai/index.ts",
 	anthropic: "packages/ai/src/providers/anthropic/actions.ts",
 	gemini: "packages/ai/src/providers/gemini/index.ts",
+	meta: "packages/ai/src/providers/meta/index.ts",
 	yutori: "packages/ai/src/providers/yutori/actions.ts",
 };
 
 const ACTION_REGEXES: Record<Provider, RegExp> = {
 	openai: /\b(click|double_click|scroll|type|wait|keypress|drag|move|screenshot)\b/g,
 	anthropic: /\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up|cursor_position|zoom)\b/g,
 	gemini: /\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g,
+	meta: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position|left_click|right_click|middle_click|triple_click|left_click_drag|mouse_move|key|hold_key|left_mouse_down|left_mouse_up)\b/g,
 	yutori: /\b(left_click|double_click|triple_click|right_click|scroll|type|key_press|hover|drag|wait|refresh|go_back|go_forward|goto_url|mouse_move|middle_click|mouse_down|mouse_up|hold_key|extract_elements|find|set_element_value|execute_js)\b/g,
 };
 
@@ -123,8 +130,8 @@ async function checkProvider(provider: Provider, examples: any): Promise<Record<
 		documented_actions: sorted(documentedActions),
 		example_repo_actions: sorted(exampleActions),
 		repo_supported_actions: sorted(localActions),
-		unknown_documented_actions: sorted(difference(new Set(documentedActions), new Set(localActions))),
-		unknown_example_actions: sorted(difference(new Set(exampleActions), new Set(localActions))),
+		unknown_documented_actions: provider === "meta" ? [] : sorted(difference(new Set(documentedActions), new Set(localActions))),
+		unknown_example_actions: provider === "meta" ? [] : sorted(difference(new Set(exampleActions), new Set(localActions))),
 		response_fields_from_examples: sorted(example.response_fields ?? []),
 		notes: notesFor(provider, documentedToolVersions, exampleToolVersions, localToolVersions),
 	};
@@ -154,6 +161,9 @@ function notesFor(provider: Provider, documentedToolVersions: string[], exampleT
 	if (provider === "gemini") {
 		notes.push("Gemini official computer use emits predefined function-call names; keep this separate from CUA custom function declarations.");
 	}
+	if (provider === "meta") {
+		notes.push("Meta uses developer-defined function tools, so provider action-name drift does not apply; compare payload threading and coordinate conventions instead.");
+	}
 	if (provider === "yutori") {
 		notes.push("Yutori Navigator emits OpenAI-compatible tool_calls for built-in browser actions; local AgentTools should execute those names but outbound payloads should not duplicate the built-in browser schemas.");
 		notes.push("Track n1 vs n1.5 separately because n1.5 can add tool_set/disable_tools and expanded browser actions.");

.agents/skills/update-models/reference/report-schema.md

@@ -12,6 +12,7 @@ Use this shape for JSON reports and the same fields when writing a Markdown summ
     "openai": {},
     "anthropic": {},
     "gemini": {},
+    "meta": {},
     "tzafon": {},
     "yutori": {}
   },

.github/workflows/ci.yml

@@ -132,6 +132,7 @@ jobs:
           OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
           ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
           GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
+          META_API_KEY: ${{ secrets.META_API_KEY }}
           TZAFON_API_KEY: ${{ secrets.TZAFON_API_KEY }}
           YUTORI_API_KEY: ${{ secrets.YUTORI_API_KEY }}
         run: npm run test:integration --workspace @onkernel/cua-ai
@@ -158,6 +159,7 @@ jobs:
           OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
           ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
           GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
+          META_API_KEY: ${{ secrets.META_API_KEY }}
           TZAFON_API_KEY: ${{ secrets.TZAFON_API_KEY }}
           YUTORI_API_KEY: ${{ secrets.YUTORI_API_KEY }}
         run: npm test --workspace @onkernel/cua-agent -- test/e2e.live.test.ts

README.md

@@ -12,7 +12,7 @@ cua "go to news.ycombinator.com and tell me the top 3 story titles"
 
 ## Why this exists
 
-Every frontier model now ships its own first-party "computer use" tool:
+Frontier models expose computer use through different tool protocols:
 
 - **OpenAI gpt-5.5**: a built-in `computer` tool that emits actions like
   `{type:"click", x, y}`, `{type:"scroll", x, y, scroll_x, scroll_y}`,
@@ -23,6 +23,8 @@ Every frontier model now ships its own first-party "computer use" tool:
 - **Google gemini-2.5-pro / gemini-3.x**: a set of predefined
   computer-use functions (`click_at`, `type_text_at`, `scroll_at`,
   `navigate`, `go_back`, …) with 0-1000 normalized coordinates.
+- **Meta Muse Spark 1.1**: OpenAI-compatible Responses API function calls
+  paired with screenshot inputs and 0-1000 normalized coordinates.
 - **Yutori Navigator n1 / n1.5**: OpenAI-compatible `chat.completions`
   responses with built-in browser action `tool_calls` like `left_click`,
   `goto_url`, `type`, and `scroll` in 0-1000 normalized coordinates.
@@ -100,6 +102,7 @@ npx tsx packages/cli/src/cli.ts --help
 export OPENAI_API_KEY=sk-...                 # for gpt-5.5
 export ANTHROPIC_API_KEY=sk-ant-...          # for claude-opus-4-7
 export GOOGLE_API_KEY=...                    # for gemini-3-flash-preview
+export META_API_KEY=...                      # for muse-spark-1.1
 export YUTORI_API_KEY=yt_...                 # for n1.5-latest
 export KERNEL_API_KEY=sk_...                 # always required
 
@@ -115,6 +118,9 @@ cua -p --model claude-opus-4-7 "Same prompt"
 # Gemini 3 Flash (built-in computer use)
 cua -p --model gemini-3-flash-preview "Same prompt"
 
+# Meta Muse Spark
+cua -p --model meta:muse-spark-1.1 "Same prompt"
+
 # Yutori Navigator
 cua -p --model n1.5-latest "Same prompt"
 

package.json

@@ -22,6 +22,7 @@
   },
   "devDependencies": {
     "@types/node": "22.18.4",
+    "smol-toml": "^1.7.0",
     "tsx": "^4.21.0",
     "typescript": "5.9.3"
   }