Python: TypeSafe: support inline images and Box questions (Jev 1.14)

[Zakaria Khchiche (Zakariakhchiche)]

Motivation & Context

TypeSafe released Jev 1.14 (jev-1.14.0, jev-preview) with image inputs and a new Box question type that locates a target in an image. The agent-framework-typesafe connector currently rejects every non-text message content, so Agent Framework users cannot send an image to Jev, and Box questions cannot be used because they require one. TypeSafe also requires the TypeSafe-Danger-Zone-Images: true header on any request with an image or a Box question, which the connector never sends.

Details and proposed scope are in #9410. This PR is opened as a draft so the implementation can be discussed alongside the issue.

Description & Review Guide

• What are the major changes?
  • Inline image data content (Content.from_data(..., media_type="image/...")) is mapped to typesafe_sdk.Image assets inside the existing messages state.
  • The connector sends TypeSafe-Danger-Zone-Images: true through the SDK's extra_headers only when the request contains an image or a Box question. Text-only requests are unchanged (extra_headers=None).
  • Remote image URIs are still rejected with an explicit message (the SDK needs the bytes); non-text function result items are still rejected.
  • README and AGENTS.md document the feature as experimental, following TypeSafe's guidance, with a Noul + Box example. Box is listed as a supported question type.
• What is the impact of these changes? Opt-in by content: nothing changes for existing text-only users. No dependency change: typesafe-sdk 0.7.5, within the current >=0.7.1,<0.8.0 range, already exposes Image, Box and BoxAnswer.
• What do you want reviewers to focus on? Whether the header should be sent automatically when an image or a Box question is present (current approach, since the request fails without it), or require an explicit client option.

Tests

• Inline image is sent as an Image asset with the opt-in header (stub client).
• Box question sends the header and the BoxAnswer is available through response.value.
• Text-only requests send no header.
• Transport-level test with a connector-owned client: the outgoing request carries the header and the image bytes.
• Remote image URI is rejected.
• A Box question alone (SDK object or mapping form, with or without an image) triggers the header.
• The PNG fixture is validated (chunk CRCs and decodable IDAT).

pytest -m "not integration": 95 passed. ruff check and ruff format --check pass. mypy reports the same 8 pre-existing errors in _tool_calls.py as on main, none in the changed file.

Related Issue

Fixes #9410

AI Assistance

• No material AI assistance was used.
• This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details:

Contribution Checklist

• The code builds clean without any errors or warnings
• All unit tests pass, and I have added new tests where possible
• The PR follows the Contribution Guidelines
• This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
• This is not a breaking change.

[Copilot]

🔵 Needs a closer look

Compatibility of the new SDK APIs with the declared minimum and locked dependency versions remains unverified.

2 open findings

• Replace corrupted PNG fixture with a valid 1×1 PNG · New
• Cover Box-triggered header behavior independently from image detection · New

What changed in this PR

Adds experimental Jev 1.14 image and Box-question support to the Python TypeSafe connector, addressing #9410.

Changes:

• Converts inline images to SDK assets and conditionally sends the required opt-in header.
• Adds image transport and Box-response tests.
• Documents experimental usage and remote-image restrictions.

File	Description
python/​packages/​typesafe/​tests/​typesafe/​test_chat_client.py	Tests image handling, headers, and Box responses.
python/​packages/​typesafe/​README.md	Documents experimental image and Box usage.
python/​packages/​typesafe/​AGENTS.md	Updates the connector’s behavioral contract.
python/​packages/​typesafe/​agent_framework_typesafe/​_chat_client.py	Maps inline images and selects opt-in headers.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.