Skip to content

Expose CoreML EP via OrtEpFactory - #31975

Open
maxim-davgalev wants to merge 1 commit into
microsoft:mainfrom
maxim-davgalev:coreml-ep-factory
Open

Expose CoreML EP via OrtEpFactory#31975
maxim-davgalev wants to merge 1 commit into
microsoft:mainfrom
maxim-davgalev:coreml-ep-factory

Conversation

@maxim-davgalev

Copy link
Copy Markdown

Description

This PR registers an internal CoreMLEpFactory with the plugin-EP infrastructure so that,
on supported Apple systems, the CoreML EP participates in device-based EP selection.

The CoreML EP now:

  • appears in GetEpDevices() as NPU and GPU OrtEpDevice entries.
  • can be added with SessionOptionsAppendExecutionProvider_V2 by selecting either or both devices.
  • participates in SetEpSelectionPolicy, so PREFER_NPU can select the Apple Neural Engine instead
    of silently falling back to the CPU EP.

The implementation follows the existing internal-factory pattern used by the CPU, DML, and WebGPU
EPs.

The factory reports "Apple" as its vendor name and the new OrtDevice::VendorIds::APPLE value
(0x106B) as its vendor ID. Apple device discovery uses the same vendor ID, so OrderDevices ranks
CoreML above other EPs for Apple hardware based on the vendor match.

Fixes #29066.

The factory advertises at most one NPU and one GPU from Apple device discovery. Core ML
selects compute-unit classes rather than individual physical devices, so additional devices of the
same type would not be individually addressable and are intentionally ignored.

No devices are advertised below Core ML 5 (macOS 12 / iOS 15), the minimum version accepted by the
current CoreML EP.

The NPU additionally requires Core ML 6 (iOS 16 / macOS 13). Earlier versions do not provide
MLComputeUnitsCPUAndNeuralEngine, the only mode that enables the Neural Engine without also
enabling the GPU. Core ML 5 can still use the ANE through "ALL", but it cannot honor an NPU-only
device selection.

The CPU device is deliberately not advertised. CPU-selection policies select one primary CPU EP
and add the ORT CPU EP as a fallback when needed. If CoreML advertised the CPU, its Apple vendor
match would make it the preferred CPU EP on Apple hardware, so other CPU-based plugin EPs would
not be selected. Models that depend on operators provided by those EPs could then fail.
CoreML-on-CPU remains available through MLComputeUnits=CPUOnly, and Core ML may use its internal
CPU path for operations unsupported by the selected compute units.

When the caller does not provide MLComputeUnits, the factory derives one value from the complete
device selection: CPUAndNeuralEngine for NPU, CPUAndGPU for GPU, and ALL for NPU + GPU.
A user-provided MLComputeUnits value may narrow which selected accelerators CoreML uses, or disable
them all with CPUOnly, but it may not enable an accelerator that was not selected.

PREFER_NPU, MAX_EFFICIENCY, and MIN_OVERALL_POWER can now select CoreML on systems where the
factory advertises the Neural Engine.

In builds that register both CoreML and WebGPU, including the official macOS packages, both EPs
advertise the same Apple GPU. When the EP selection-policy API is used, CoreML's Apple vendor ID
matches the GPU's, giving CoreML priority over WebGPU. As a result, PREFER_GPU and
MAX_PERFORMANCE now select CoreML instead of WebGPU. CoreML uses
MLComputeUnits=CPUAndGPU for that selection, which excludes the Neural Engine. WebGPU remains
available through explicit device selection.

Tests

This patch adds 19 CoreML AutoEP tests covering:

  • factory registration and NPU/GPU advertisement.
  • explicit single device and NPU+GPU V2 selection.
  • PREFER_NPU, PREFER_GPU, and their policy aliases.
  • CoreML assignment and inference with ORT CPU fallback disabled.
  • default MLComputeUnits derivation and user-option propagation.
  • narrowing and incompatible option validation.
  • invalid, duplicate, null, and CPU device selections.
  • Core ML 5/6 version requirements and the limit of one device of each type.
  • cleanup of previously created devices after a subsequent creation failure.
  • vendor affinity taking precedence over EP-name ordering.

Register an internal CoreMLEpFactory with the plugin-EP
infrastructure so that, on supported Apple systems with discovered
accelerator hardware, the CoreML EP appears in
GetEpDevices(), can be added to a session with
SessionOptionsAppendExecutionProvider_V2, and is eligible for EP
selection policies, notably PREFER_NPU, which can now select the Apple
Neural Engine on supported systems instead of silently falling back
to the CPU EP.

Fixes microsoft#29066

The factory registers with vendor ID OrtDevice::VendorIds::APPLE,
matching the vendor ID reported by Apple device discovery, so device
ordering during EP selection prefers CoreML for Apple hardware via
the vendor-affinity rule.

In builds that also register the internal WebGPU EP, including the
official macOS packages, PREFER_GPU and MAX_PERFORMANCE now select
CoreML instead of WebGPU when both EPs claim the same Apple GPU,
because CoreML has matching vendor affinity.

The factory advertises at most one NPU and one GPU from device
discovery. It advertises no devices below Core ML 5, the EP's minimum
supported version. The NPU additionally requires Core ML 6
(iOS 16 / macOS 13), because earlier Core ML versions have no
compute-units mode that enables the Neural Engine without also
enabling the GPU. MLComputeUnitsCPUAndNeuralEngine requires Core ML 6,
so an NPU selection could not otherwise be honored.

The CPU device is deliberately left to the CPU EP. CPU-selecting
policies choose a single preferred EP for the CPU device, so
advertising it would displace the ORT CPU EP and other CPU-based EPs.
CoreML-on-CPU remains available through an explicit
MLComputeUnits=CPUOnly option, and Core ML can use its internal CPU
path for operations unsupported by the selected compute units.

When MLComputeUnits is not specified, the factory derives it from the
selected devices: CPUAndNeuralEngine for NPU, CPUAndGPU for GPU, and
ALL for NPU+GPU. User options may narrow this set, but may not enable
an accelerator that was not selected.

Tests in onnxruntime/test/autoep/test_selection.cc cover factory
registration, explicit GPU and NPU+GPU selection, inference,
PREFER_NPU/PREFER_GPU and their policy aliases, compute-unit
defaulting and conflict validation, version checks, device validation
and selection limits, CPU-only narrowing, and transactional cleanup
after partial device-creation failures.
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@maxim-davgalev

Copy link
Copy Markdown
Author

maxim-davgalev please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

@microsoft-github-policy-service agree company="Parallels"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Expose the CoreML EP via OrtEpFactory so it appears in GetEpDevices() and participates in automatic EP selection

1 participant