Expose CoreML EP via OrtEpFactory - #31975
Open
maxim-davgalev wants to merge 1 commit into
Open
Conversation
Register an internal CoreMLEpFactory with the plugin-EP infrastructure so that, on supported Apple systems with discovered accelerator hardware, the CoreML EP appears in GetEpDevices(), can be added to a session with SessionOptionsAppendExecutionProvider_V2, and is eligible for EP selection policies, notably PREFER_NPU, which can now select the Apple Neural Engine on supported systems instead of silently falling back to the CPU EP. Fixes microsoft#29066 The factory registers with vendor ID OrtDevice::VendorIds::APPLE, matching the vendor ID reported by Apple device discovery, so device ordering during EP selection prefers CoreML for Apple hardware via the vendor-affinity rule. In builds that also register the internal WebGPU EP, including the official macOS packages, PREFER_GPU and MAX_PERFORMANCE now select CoreML instead of WebGPU when both EPs claim the same Apple GPU, because CoreML has matching vendor affinity. The factory advertises at most one NPU and one GPU from device discovery. It advertises no devices below Core ML 5, the EP's minimum supported version. The NPU additionally requires Core ML 6 (iOS 16 / macOS 13), because earlier Core ML versions have no compute-units mode that enables the Neural Engine without also enabling the GPU. MLComputeUnitsCPUAndNeuralEngine requires Core ML 6, so an NPU selection could not otherwise be honored. The CPU device is deliberately left to the CPU EP. CPU-selecting policies choose a single preferred EP for the CPU device, so advertising it would displace the ORT CPU EP and other CPU-based EPs. CoreML-on-CPU remains available through an explicit MLComputeUnits=CPUOnly option, and Core ML can use its internal CPU path for operations unsupported by the selected compute units. When MLComputeUnits is not specified, the factory derives it from the selected devices: CPUAndNeuralEngine for NPU, CPUAndGPU for GPU, and ALL for NPU+GPU. User options may narrow this set, but may not enable an accelerator that was not selected. Tests in onnxruntime/test/autoep/test_selection.cc cover factory registration, explicit GPU and NPU+GPU selection, inference, PREFER_NPU/PREFER_GPU and their policy aliases, compute-unit defaulting and conflict validation, version checks, device validation and selection limits, CPU-only narrowing, and transactional cleanup after partial device-creation failures.
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Author
@microsoft-github-policy-service agree company="Parallels" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR registers an internal CoreMLEpFactory with the plugin-EP infrastructure so that,
on supported Apple systems, the CoreML EP participates in device-based EP selection.
The CoreML EP now:
of silently falling back to the CPU EP.
The implementation follows the existing internal-factory pattern used by the CPU, DML, and WebGPU
EPs.
The factory reports "Apple" as its vendor name and the new OrtDevice::VendorIds::APPLE value
(0x106B) as its vendor ID. Apple device discovery uses the same vendor ID, so OrderDevices ranks
CoreML above other EPs for Apple hardware based on the vendor match.
Fixes #29066.
The factory advertises at most one NPU and one GPU from Apple device discovery. Core ML
selects compute-unit classes rather than individual physical devices, so additional devices of the
same type would not be individually addressable and are intentionally ignored.
No devices are advertised below Core ML 5 (macOS 12 / iOS 15), the minimum version accepted by the
current CoreML EP.
The NPU additionally requires Core ML 6 (iOS 16 / macOS 13). Earlier versions do not provide
MLComputeUnitsCPUAndNeuralEngine, the only mode that enables the Neural Engine without also
enabling the GPU. Core ML 5 can still use the ANE through "ALL", but it cannot honor an NPU-only
device selection.
The CPU device is deliberately not advertised. CPU-selection policies select one primary CPU EP
and add the ORT CPU EP as a fallback when needed. If CoreML advertised the CPU, its Apple vendor
match would make it the preferred CPU EP on Apple hardware, so other CPU-based plugin EPs would
not be selected. Models that depend on operators provided by those EPs could then fail.
CoreML-on-CPU remains available through MLComputeUnits=CPUOnly, and Core ML may use its internal
CPU path for operations unsupported by the selected compute units.
When the caller does not provide MLComputeUnits, the factory derives one value from the complete
device selection: CPUAndNeuralEngine for NPU, CPUAndGPU for GPU, and ALL for NPU + GPU.
A user-provided MLComputeUnits value may narrow which selected accelerators CoreML uses, or disable
them all with CPUOnly, but it may not enable an accelerator that was not selected.
PREFER_NPU, MAX_EFFICIENCY, and MIN_OVERALL_POWER can now select CoreML on systems where the
factory advertises the Neural Engine.
In builds that register both CoreML and WebGPU, including the official macOS packages, both EPs
advertise the same Apple GPU. When the EP selection-policy API is used, CoreML's Apple vendor ID
matches the GPU's, giving CoreML priority over WebGPU. As a result, PREFER_GPU and
MAX_PERFORMANCE now select CoreML instead of WebGPU. CoreML uses
MLComputeUnits=CPUAndGPU for that selection, which excludes the Neural Engine. WebGPU remains
available through explicit device selection.
Tests
This patch adds 19 CoreML AutoEP tests covering: