Repository navigation
Let a Core ML model read and write bound pixel buffers in place (#24261) - #24261
Open
sivapatibandla wants to merge 2 commits into
Open
sivapatibandla wants to merge 2 commits into
sivapatibandla wants to merge 2 commits into
Conversation
sivapatibandla
requested review from
JacobSzwejbka and
larryliu0820
as code owners
October 28, 2026 21:25
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24261
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 32837f6 with merge base 337aa81 ( BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
Contributor
|
@sivapatibandla has exported this pull request. If you are a Meta employee, you can view the originating Diff in D127640931. |
This PR needs a
|
meta-codesync
Bot
changed the title
sivapatibandla
added a commit
to sivapatibandla/executorch
that referenced
this pull request
Oct 28, 2026
…rch#24261) Summary: Pull Request resolved: pytorch#24261 Lets a Core ML model take inputs and outputs in IOSurface-backed fp16 pixel buffers bound to it, so Core ML reads and writes them in place instead of copying them into and out of its own buffers. The saving grows with tensor size, so it matters most for large fp16 tensors such as video frames. `-[ETCoreMLModel bindPixelBuffer:error:]` binds a `kCVPixelFormatType_OneComponent16Half` IOSurface-backed buffer to one model instance, and refuses it on OSes older than macOS 12 / iOS 16, which lack the pixel-buffer `MLMultiArray` API, or if the buffer can't be locked or has no base address. There is no global registry, and two models can bind the same buffer, e.g. to chain one model's output into another's input. Callers don't have the delegate's headers, so the next diff lets them bind buffers through load-time options. Bound buffers are matched by base address: when an input's or output's data pointer is a bound buffer's base address, `prepareArgs` hands Core ML an array wrapping that buffer (`-[MLMultiArray initWithPixelBuffer:shape:]`) as the input or the output backing. The model's input or output has to be fp16 too: against a feature of another type, a bound buffer fails the arg instead of reaching Core ML, while unbound args still convert as before. The model keeps one array per bound buffer, made on first use and remade only when the arg's shape changes, since wrapping costs Core ML a setup per array. After the prediction, the output step recognizes the in-place write by Core ML returning that same array (it hands back the output backing itself) and doesn't touch the buffer. When Core ML doesn't write a backing in place, the fallback copy writes through the pixel buffer's row stride (rows are padded to 64 bytes, e.g. 720 -> 768 bytes for 360 fp16 values) instead of treating the buffer as dense, and a buffer whose shape or dtype doesn't match the arg fails the prediction. Only models with bound buffers see any of this: with nothing bound, the per-arg lookup walks an empty list and returns, and the copy paths are unchanged. This diff was authored with Claude Code. Reviewed By: metascroy Differential Revision: D127640931
sivapatibandla
force-pushed
the
export-D127640931
branch
from
October 28, 2026 21:34
f629990 to
d25c875
Compare
metascroy
approved these changes
Oct 28, 2026
sivapatibandla
added a commit
to sivapatibandla/executorch
that referenced
this pull request
Oct 29, 2026
…rch#24261) Summary: Lets a Core ML model take inputs and outputs in IOSurface-backed fp16 pixel buffers bound to it, so Core ML reads and writes them in place instead of copying them into and out of its own buffers. The saving grows with tensor size, so it matters most for large fp16 tensors such as video frames. `-[ETCoreMLModel bindPixelBuffer:error:]` binds a `kCVPixelFormatType_OneComponent16Half` IOSurface-backed buffer to one model instance, and refuses it on OSes older than macOS 12 / iOS 16, which lack the pixel-buffer `MLMultiArray` API, or if the buffer can't be locked or has no base address. There is no global registry, and two models can bind the same buffer, e.g. to chain one model's output into another's input. Callers don't have the delegate's headers, so the next diff lets them bind buffers through load-time options. Bound buffers are matched by base address: when an input's or output's data pointer is a bound buffer's base address, `prepareArgs` hands Core ML an array wrapping that buffer (`-[MLMultiArray initWithPixelBuffer:shape:]`) as the input or the output backing. The model's input or output has to be fp16 too: against a feature of another type, a bound buffer fails the arg instead of reaching Core ML, while unbound args still convert as before. The model keeps one array per bound buffer, made on first use and remade only when the arg's shape changes, since wrapping costs Core ML a setup per array. After the prediction, the output step recognizes the in-place write by Core ML returning that same array (it hands back the output backing itself) and doesn't touch the buffer. When Core ML doesn't write a backing in place, the fallback copy writes through the pixel buffer's row stride (rows are padded to 64 bytes, e.g. 720 -> 768 bytes for 360 fp16 values) instead of treating the buffer as dense, and a buffer whose shape or dtype doesn't match the arg fails the prediction. Only models with bound buffers see any of this: with nothing bound, the per-arg lookup walks an empty list and returns, and the copy paths are unchanged. This diff was authored with Claude Code. Differential Revision: D127640931
sivapatibandla
force-pushed
the
export-D127640931
branch
from
October 29, 2026 04:44
d25c875 to
b9c9e56
Compare
sivapatibandla
added a commit
to sivapatibandla/executorch
that referenced
this pull request
Oct 29, 2026
…rch#24261) Summary: Lets a Core ML model take inputs and outputs in IOSurface-backed fp16 pixel buffers bound to it, so Core ML reads and writes them in place instead of copying them into and out of its own buffers. The saving grows with tensor size, so it matters most for large fp16 tensors such as video frames. `-[ETCoreMLModel bindPixelBuffer:error:]` binds a `kCVPixelFormatType_OneComponent16Half` IOSurface-backed buffer to one model instance, and refuses it on OSes older than macOS 12 / iOS 16, which lack the pixel-buffer `MLMultiArray` API, or if the buffer can't be locked or has no base address. There is no global registry, and two models can bind the same buffer, e.g. to chain one model's output into another's input. Callers don't have the delegate's headers, so the next diff lets them bind buffers through load-time options. Bound buffers are matched by base address: when an input's or output's data pointer is a bound buffer's base address, `prepareArgs` hands Core ML an array wrapping that buffer (`-[MLMultiArray initWithPixelBuffer:shape:]`) as the input or the output backing. The model's input or output has to be fp16 too: against a feature of another type, a bound buffer fails the arg instead of reaching Core ML, while unbound args still convert as before. The model keeps one array per bound buffer, made on first use and remade only when the arg's shape changes, since wrapping costs Core ML a setup per array. After the prediction, the output step recognizes the in-place write by Core ML returning that same array (it hands back the output backing itself) and doesn't touch the buffer. When Core ML doesn't write a backing in place, the fallback copy writes through the pixel buffer's row stride (rows are padded to 64 bytes, e.g. 720 -> 768 bytes for 360 fp16 values) instead of treating the buffer as dense, and a buffer whose shape or dtype doesn't match the arg fails the prediction. Only models with bound buffers see any of this: with nothing bound, the per-arg lookup walks an empty list and returns, and the copy paths are unchanged. This diff was authored with Claude Code. Differential Revision: D127640931
Summary: `BackendInitContext::method_meta()` gives a delegate's `init` the `MethodMeta` of the method being loaded, or null when the context is built without one. `Method::init` passes its own; the view is borrowed and valid only during `init`. With it, a delegate can check the method around it, such as its delegates, its instruction count and whether its inputs and outputs are memory planned, before it accepts options that depend on them. The Core ML delegate uses this later in the stack to accept IOSurface load options only for a method that is just its own call on unplanned inputs and outputs. This diff was authored with Claude Code. Reviewed By: JacobSzwejbka Differential Revision: D127716377
…rch#24261) Summary: Lets a Core ML model take inputs and outputs in IOSurface-backed fp16 pixel buffers bound to it, so Core ML reads and writes them in place instead of copying them into and out of its own buffers. The saving grows with tensor size, so it matters most for large fp16 tensors such as video frames. `-[ETCoreMLModel bindPixelBuffer:error:]` binds a `kCVPixelFormatType_OneComponent16Half` IOSurface-backed buffer to one model instance, and refuses it on OSes older than macOS 12 / iOS 16, which lack the pixel-buffer `MLMultiArray` API, or if the buffer can't be locked or has no base address. There is no global registry, and two models can bind the same buffer, e.g. to chain one model's output into another's input. Callers don't have the delegate's headers, so the next diff lets them bind buffers through load-time options. Bound buffers are matched by base address: when an input's or output's data pointer is a bound buffer's base address, `prepareArgs` hands Core ML an array wrapping that buffer (`-[MLMultiArray initWithPixelBuffer:shape:]`) as the input or the output backing. The model's input or output has to be fp16 too: against a feature of another type, a bound buffer fails the arg instead of reaching Core ML, while unbound args still convert as before. The model keeps one array per bound buffer, made on first use and remade only when the arg's shape changes, since wrapping costs Core ML a setup per array. After the prediction, the output step recognizes the in-place write by Core ML returning that same array (it hands back the output backing itself) and doesn't touch the buffer. When Core ML doesn't write a backing in place, the fallback copy writes through the pixel buffer's row stride (rows are padded to 64 bytes, e.g. 720 -> 768 bytes for 360 fp16 values) instead of treating the buffer as dense, and a buffer whose shape or dtype doesn't match the arg fails the prediction. Only models with bound buffers see any of this: with nothing bound, the per-arg lookup walks an empty list and returns, and the copy paths are unchanged. This diff was authored with Claude Code. Differential Revision: D127640931
sivapatibandla
force-pushed
the
export-D127640931
branch
from
October 29, 2026 11:10
b9c9e56 to
32837f6
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Lets a Core ML model take inputs and outputs in IOSurface-backed fp16 pixel buffers bound to it, so Core ML reads and writes them in place instead of copying them into and out of its own buffers. The saving grows with tensor size, so it matters most for large fp16 tensors such as video frames.
-[ETCoreMLModel bindPixelBuffer:error:]binds akCVPixelFormatType_OneComponent16HalfIOSurface-backed buffer to one model instance, and refuses it on OSes older than macOS 12 / iOS 16, which lack the pixel-bufferMLMultiArrayAPI, or if the buffer can't be locked or has no base address. There is no global registry, and two models can bind the same buffer, e.g. to chain one model's output into another's input. Callers don't have the delegate's headers, so the next diff lets them bind buffers through load-time options.Bound buffers are matched by base address: when an input's or output's data pointer is a bound buffer's base address,
prepareArgshands Core ML an array wrapping that buffer (-[MLMultiArray initWithPixelBuffer:shape:]) as the input or the output backing. The model's input or output has to be fp16 too: against a feature of another type, a bound buffer fails the arg instead of reaching Core ML, while unbound args still convert as before. The model keeps one array per bound buffer, made on first use and remade only when the arg's shape changes, since wrapping costs Core ML a setup per array. After the prediction, the output step recognizes the in-place write by Core ML returning that same array (it hands back the output backing itself) and doesn't touch the buffer. When Core ML doesn't write a backing in place, the fallback copy writes through the pixel buffer's row stride (rows are padded to 64 bytes, e.g. 720 -> 768 bytes for 360 fp16 values) instead of treating the buffer as dense, and a buffer whose shape or dtype doesn't match the arg fails the prediction.Only models with bound buffers see any of this: with nothing bound, the per-arg lookup walks an empty list and returns, and the copy paths are unchanged.
This diff was authored with Claude Code.
Differential Revision: D127640931