Repository navigation
Fix crashes, wrong outputs and lowering failures in several backends and in exir - #24222
Merged
Merged
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24222
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ❌ 1 New Failure, 70 PendingAs of commit 204bbc8 with merge base 254635c ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
Contributor
Author
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 22m 42s —— View job Approve: nothing blocks. The main open point is that on OpenVINO before 2026.4, the delegate blob now stays in memory next to OpenVINO's own copy for the whole life of the method. Three smaller robustness and test gaps are noted below. PR Review: #24222SummaryThe PR does two things:
Nothing blocks merging. The OpenVINO memory cost on versions before 2026.4 is worth settling, and three smaller points follow. The PyTorch Inductor and OpenVINO sources are not in this checkout, so I mark statements about their internals as low confidence. I ran no tests. Code Quality
Testing
Performance
RecommendationApprove Nothing here blocks the merge. The one point to settle is the OpenVINO memory cost: either free the blob on versions before 2026.4, or document the cost. That can happen here or in a follow-up.
|
Gasoonjia
approved these changes
Oct 28, 2026
shoumikhin
force-pushed
the
fix-lerobot-export-bugs
branch
from
October 28, 2026 09:44
35f7db1 to
55b4840
Compare
shoumikhin
changed the title
Contributor
Author
|
Thanks for the review. Points 1, 2 and 4 are fixed in the commits that own the code. Point 3 has no code change, see below.
|
shoumikhin
changed the title
shoumikhin
force-pushed
the
fix-lerobot-export-bugs
branch
from
October 28, 2026 15:04
55b4840 to
52e8dcf
Compare
shoumikhin
requested review from
JacobSzwejbka and
larryliu0820
as code owners
October 28, 2026 15:04
shoumikhin
force-pushed
the
fix-lerobot-export-bugs
branch
from
October 28, 2026 19:42
52e8dcf to
c39d57f
Compare
shoumikhin
requested review from
SS-JIA,
digantdesai,
metascroy and
rascani
as code owners
October 28, 2026 19:42
shoumikhin
changed the title
shoumikhin
force-pushed
the
fix-lerobot-export-bugs
branch
from
October 28, 2026 21:47
c39d57f to
d11bd4b
Compare
XNNPACKQuantizer annotated ops that read or produce bfloat16 tensors, such
as linears in a model that casts its activations to bfloat16. The quantize,
dequantize and choose_qparams nodes it produced then ran on bfloat16
tensors. XNNPACK does not take those patterns, so they stayed on the
portable quantized kernels, which do not take bfloat16 and abort the
process at run time ("Expected input to be Float tensor received: 15").
Skip nodes that read or produce a bfloat16 tensor, both in the filter that
every annotator applies (cat included, since the previous commit) and when
annotations are propagated to ops that share their input's quantization
parameters. Such ops stay unquantized and run correctly. Float32 and
float16 models quantize as before; the test checks that a float32 linear
feeding a bfloat16 op is still quantized.
to_backend could fail with "Expected output node, got op=call_function" when a runtime check outside a partition reads a value the partition computes, for example torch._check(t.all().item()) where the tensor ops are delegated and the item() and the check are not. After fusing the partition, create_submodule_from_nodes re-sorted the graph with legalize_graph. That sort places a node as soon as its inputs are ready, so the output node could be placed before check nodes that have no users. Later code calls graph.output_node(), which requires the output node to be last. Use stable_topological_sort instead. It keeps the original order wherever the dependencies allow it, so the output node stays last. This changes the order of the remaining top-level nodes of every program that to_backend touches: they now keep their original order instead of the order the old sort produced.
A model that uses a tensor-wide check, for example torch._check_tensor_all or transformers' torch_compilable_check, could not go through to_edge. Two things stopped it: 1. torch.export turns the check into aten._is_all_true, which is not a core ATen op and has no decomposition, so the ATen verifier rejected it. 2. The check then reads the result with item() and compares it through torch.sym_ite. The Edge verifier rejected every Python symbolic function (torch.sym_ite, sym_max, sym_min, sym_not, sym_float, sym_int), on data-dependent values and on shapes alike, even though ExecuTorch lowers each of them to its own executorch_prim op, which has a runtime kernel. Like other runtime asserts, the check is removed later by to_executorch unless a backend takes it into a delegate; the problem was that the program never got that far. Decompose aten._is_all_true to aten.all in the default decomposition table, and let the Edge verifier accept the Python symbolic functions that ExecuTorch maps to prim ops, the same way it already accepts those prim ops. The runtime's sym_max and sym_min kernels only take ints, so the verifier still rejects them, also inside control flow branches, when an argument is a float or a bool.
A program lowered with the CUDA backend could pass lowering and then kill the process on its first run: undefined symbol: aoti_torch_as_strided Inductor's C++ wrapper calls aoti_torch_as_strided when it allocates a buffer with a larger shape than its logical shape and then gives it the logical shape. ExecuTorch's CUDA shim library did not provide it, so the compiled model could not be loaded. Add aoti_torch_as_strided with the PyTorch shim's signature. It returns a view with the given sizes and strides and the tensor's own storage offset, built on the existing aoti_torch__reinterpret_tensor. Also regenerate the checked-in Windows import library, backends/cuda/runtime/aoti_cuda_shims.lib, as its README describes, so a wrapper lowered for a Windows target can link against the new shim. Its symbol list is the old one plus aoti_torch_as_strided, and its members carry no timestamps.
… other Lowering a dynamically quantized pi0 model to XNNPACK failed with "Invalid partition, found dependency cycles". Before merging two partitions, GroupBasedPartitioner checks that no node outside them is both downstream and upstream of the merged partition. That check looks at single nodes. But every other partition also becomes a single node once it is fused. So a path can leave the merged partition, enter another partition at one node, leave it from a different node that the first one does not reach, and come back. The two partitions then depend on each other, and fusing them fails. Before accepting a merge, also follow paths through the other partitions: whenever the merged partition reaches a node of another partition, add everything that partition reaches as a whole, and reject the merge as soon as that reaches something upstream of it. Only partitions that will be returned count, so partitions that are later filtered out (for example ones without enough compute nodes) do not block a merge. Their compute node count now comes from one helper shared with the filter. To stay fast on large graphs, the check only visits the partitions it actually reaches, and caches what each partition reaches; a merge only drops the cached entries that reach the merged nodes. On synthetic graphs of 4k to 16k nodes the partitioner takes about twice as long as before. The merge loop itself is unchanged and still tries every remaining pair, so a graph where the check rejects most merges now keeps many partitions and pays for all their pairs (about 2 s for 1600 partitions on a 4k-node chain of groups); before, such graphs had few partitions only because merges that cannot be fused were accepted. If caller-provided node groups already depend on each other, what a partition reaches is found by walking around that cycle directly. Two partitions whose nodes all already lie on one cycle through such groups can still be merged, since every cycle through the result already existed. A partition that will be fused counts as one node there; any other partition counts as its separate nodes, so a group that is later filtered out (for example an add with a getitem) can still merge to remove a cycle. Other merges that would leave the merged partition on a cycle are refused. Because a merge that would need a later merge to become valid is now rejected, a few graphs end up with a different number of partitions. The merge loop is still a single greedy pass, so caller node_groups that depend on each other through a plain node can still give partitions that cannot be fused, as before this change.
MatmulToBmmPass read the operand shapes with ProxyValue.to_tensor(). When
an operand of the matmul is a constant (a parameter, a buffer or a tensor
attribute), ExportPass passes it to call_operator as a plain torch.Tensor,
so CortexMQuantizer crashed in prepare_pt2e for every such matmul:
AttributeError: 'Tensor' object has no attribute 'to_tensor'
For example softmax(x) @ codebook, where codebook is a tensor attribute.
Read the shapes through .data, which works for both kinds of operand. A
constant operand of a shape the pass rewrites, such as a batched rank-3
codebook used as the right-hand side, now becomes bmm like any other operand
and lowers to cortex_m.quantized_batch_matmul, which already supports a
constant right-hand side. Shapes the pass does not rewrite, such as
rank-3 @ rank-2, no longer crash but stay matmul.
When ExportPass meets a real tensor, for example a get_attr constant, it
makes a fake tensor for the node metadata. Only parameters were faked with
static_shapes=True. Any other tensor that the fake mode had not seen before
got a fresh symbol for every dimension when the fake mode has a ShapeEnv,
which is the case for exported programs.
Such tensors are common: constant folding (for example the Arm quantizer's
ConstantFoldingPass) creates new constant tensors, and the next ExportPass
gives them symbolic sizes like (s45, s96) instead of (9, 2). Later passes
then see symbolic shapes where everything is static. With the LeRobot
diffusion policy, the Arm quantizer then failed in DecomposeVarPass:
AssertionError: a.node.constant should not be None
A real tensor is constant data with known sizes, so fake all of them with
static shapes, as the existing TODO suggested.
aten.slice_scatter is functional: it returns a new tensor and leaves its base alone. The MLX builder lowers it to a SliceUpdateNode whose output slot differs from its base slot, and the runtime handles that case without writing the base. Still, exec_slice_update always fetched the base through tensor_ref(), which is the accessor for tensors an op writes, and which refuses constants. So a slice_scatter whose base is a buffer or another constant failed at run time with "tensor_ref: cannot mutate constant tensor". Read the base with const_tensor_ref(), and use tensor_ref() only for the in-place form, which is the one that writes it.
The partitioner keeps aten.stack from being decomposed so that it can lower to the QNN Pack op. HTP's Pack takes inputs of rank 1 to 4 (QNN HTP op definition), so its output has 2 to 5 dims. A stack of 0-d or 5-D tensors is rejected at op validation. The node then stays on CPU, but because it was kept from decomposition, to_edge_transform_and_lower fails with "aten.stack ... was not decomposed or delegated". Let filter_fn decompose a stack whose output rank is outside 2 to 5, the same way it already does for integer stacks. The unsqueeze and cat it decomposes into are checked by the partitioner like any other op.
When an op stays on CPU, the ops before it can form a partition that is
fed only by constants (parameters, buffers, lifted constants), or by
nothing at all, for example a lone aten.full or a chain of casts of
lifted scalars. HTP refuses such a graph ("No graph inputs present for
graph", or "Graph has zero node"), so context binary generation failed
and the whole lowering stopped.
For HTP, drop a partition when none of its inputs is a runtime value,
so its ops run on CPU. A mutable buffer counts as a runtime input, since
QNN reads it as a graph input in the partition that updates it.
tag_nodes no longer assumes at least one partition is left, which
dropping can now cause.
Other backends keep their partitions. GPU accepts graphs without
inputs, and LPAI places its quantize and dequantize boundaries for CPU
fallback before partitioning, so dropping a partition there afterwards
would hand a float tensor to a delegate that expects a quantized one.
LPAI refuses graphs without inputs too ("Graph has no input tensors"),
so such a partition still fails there as before. Fixing that needs
the decision to move before LPAI places its boundaries.
Two HTP cases still fail as before: a dropped partition that holds an
op the partitioner keeps from decomposition, such as a linear on a
buffer (that op cannot run on CPU undecomposed, so lowering now fails
with "was not decomposed or delegated"), and a partition whose only
runtime input is a mutable buffer that a different partition updates.
…an replace it Remove0DTensor drops every select that returns a 0-d tensor and feeds its users the whole input instead. That is only right when the input holds just the selected element and every user broadcasts it, so that a one-element tensor gives the same result as the 0-d element. The pass was written for that case (torch.zeros(1)[0] added to a tensor). Otherwise: - For a lookup such as table[t] on a longer buffer, the users got the whole table: lowering failed later with a broadcast error, or, when the shapes happened to broadcast, the program silently computed the wrong values. - For a one-element input, a user that depends on the rank, such as torch.stack, got a tensor of shape (1,) instead of a 0-d one and produced a different shape. Only drop the select when its input is known to have one element and every user, looking through quantize and dequantize, is an add, sub, mul or div whose output has at least one dimension. Other 0-d selects are left in the graph and run on CPU. That includes a select used by another op such as sqrt, or by a dtype cast (x * int_buffer[0] with a float x): the result is still right, but some ops run on CPU instead of in the delegate. Casts are not looked through, because a dim-order cast keeps the 0-d layout in its own arguments and fails at runtime on CPU when given a one-element tensor. Two kinds of graph then lower only with other changes in place. A stack of such selects needs the partitioner to decompose stacks that HTP Pack cannot take. A select and cast left on CPU can leave an HTP partition fed only by constants, which has to stay on CPU as well. So this change lands after both. The pass's own test in tests/rework selected from a four-element tensor and checked that the select was removed, so it asserted the wrong behavior; it now uses a one-element input.
…with its weight RecomposeRmsNorm takes any constant operand of the last multiply as the RMSNorm weight, uses its shape as normalized_shape, and never looks at how the mean reduces. Two things go wrong: - When a norm is followed by a multiply with a 0-d constant (a scalar scale), it builds rms_norm(x, normalized_shape=[]), which fails the next time the graph is traced: "Expected normalized_shape to be at least 1-dimensional". - When the mean reduces a dim other than the last one, several dims, or drops the reduced dim (keepdim=False), it still builds an rms_norm over the last dim, so the program silently computes different values. The backend's RmsNorm builder normalizes the last dim with a weight of that size, so only recompose that form: the mean must reduce exactly the last dim and keep it (a 1-D input may reduce all dims or drop it), and the constant must have the shape of that dim. Anything else is left as the decomposed ops.
The Qualcomm backend has no bfloat16 tensor type (QNN_TENSOR_TYPE_MAP has no entry for it), but nothing checked data types before a node visitor was asked to define the node's tensors. An unquantized bfloat16 node (for example an explicit x.to(torch.bfloat16) region in a float model) therefore crashed lowering with KeyError: torch.bfloat16 in NodeVisitor.get_data_type instead of staying on CPU. Add has_unsupported_dtype, true when the node's output or one of its inputs is an unquantized tensor of a data type the backend cannot represent, and use it in two places: - The partitioner skips such a node, so it falls back to CPU like any other unsupported node. - filter_fn does not keep such a node from decomposition unless every such tensor is quantized: each such input comes from a dequantize op and the output feeds a quantize op. Otherwise an op the backend normally lowers whole, such as a bfloat16 linear, would stay on CPU undecomposed. That fails with "was not decomposed or delegated", or, for a linear whose weight alone is dequantized (weight-only quantization), needs a linear kernel that the portable CPU kernels do not have. A quantized tensor takes its QNN type from its encoding, so quantized bfloat16 regions are still delegated as before. A bfloat16 graph input or output of a quantized model is still not supported. The test covers a quantized, an fp16 and a weight-only quantized bfloat16 linear. It lowers the quantized model first, because some QNN SDK versions crash on a quantized lowering that follows an fp16 one in the same process.
… CPU QNN needs a size for every tensor dimension. For a dynamic dimension the node visitor passes the SymInt's hint, but a data-dependent size (for example the result of boolean mask indexing, x[mask]) has no hint. Nothing checked for that before a node visitor was asked to define such a tensor, and lowering crashed with a TypeError from the TensorWrapper constructor instead of leaving the node on CPU. Add has_data_dependent_shape, true when the node's output or one of its inputs has a dimension without a hint, and use it in two places: - The partitioner skips such a node, so it falls back to CPU like any other unsupported node. - filter_fn does not keep such a node from decomposition. Otherwise an op the backend normally lowers whole, such as a stack of masked tensors, would stay on CPU undecomposed and fail with "was not decomposed or delegated". A linear on masked rows also needs partitions without a runtime input to stay on CPU on HTP: once the linear is decomposed on CPU, its weight permute is fed only by the weight. That is handled by "Keep partitions with no runtime input on CPU", so this change lands after it.
… its user Passes that build a scalar constant with create_const_node (for example DecomposeRemainder for i % 3 on an int64 tensor) insert its get_attr at the very top of the graph, above the placeholders. I64toI32 then put the int32 cast right after that get_attr, also above the placeholders. When lift_constant_tensor_pass later turned the get_attr into a placeholder, it placed it with the other placeholders, below the cast, and lowering failed: "Argument _lifted_tensor_constant0 of Node aten__to_copy_default was used before it has been defined". Insert the cast for a get_attr right before its first user instead, which is below the placeholders no matter where the get_attr sits. Placeholders keep the cast right after them, as before.
The XNNPACK partitioner never delegated aten.elu, so ELU ran on the portable kernel even though XNNPACK has an ELU node and ExecuTorch already has its partitioner config and serializer. The config was commented out of the partitioner's config list in 2024 while waiting for a PyTorch pin update, and was never turned back on. Turn the config on. XNNPACK's ELU only has alpha, while aten.elu also has scale and input_scale, and SELU decomposes to elu with scale != 1. XNNPACK also stores alpha in the compute dtype: an alpha that overflows it gives NaN, and on x86 an infinite alpha gives NaN and a zero or subnormal alpha gives +0 where eager gives -0. So the config keeps elu nodes on the CPU kernel unless alpha is a positive normal number in the compute dtype and scale and input_scale are 1. The config and the serializer read these values from positional or keyword arguments, and the serializer handles an elu call that leaves alpha at its default. As with other XNNPACK ops, delegated results can still differ from eager for values near zero: subnormal inputs and results are flushed to zero, and the fp16 kernel returns +0 for an input of -0. The config already listed static quantization, so in a statically quantized model an elu between quantized ops is now delegated as a quantized XNNPACK ELU as well (checked to match the reference on a conv, elu, conv model). Enable the existing fp32 and fp16 ELU tests. The fp32 test also checks that SELU, an input_scale of 2 and an infinite alpha stay out of XNNPACK and that an elu without alpha is delegated, and the fp16 test checks that an alpha too large for fp16 stays out, all matching eager.
A linear quantized with torchao's IntxWeightOnlyConfig gave wrong outputs after lowering to XNNPACK. With int4 weights in groups of 32 (the llama "4w" mode) the largest error on a small MLP against the eager quantized model was over 4000, and int8 per-channel weights were just as wrong. The quantized weight is a tensor subclass, so its dequantize_affine runs inside the linear call and lands in the linear's source partition. When the linear is decomposed to mm or addmm, the partitioner finds the weight by walking back from the gemm until it leaves the source partition. It walked straight through the dequant to the int8 data, saw a plain parameter, and delegated the gemm as fp32. ConvertToLinearPass does the same walk when it rebuilds the linear inside the delegate, so the int8 values were used as float weights, with the scales dropped. Stop both walks at a dequant node. With float activations the gemm is then seen as quantized, and since the XNNPACK partitioner has no path for float activations with quantized weights, it stays out of the delegate. Lowering such a model now fails loudly at to_executorch (no kernel for dequantize_affine) instead of running with wrong weights. With dynamically quantized activations the gemm is delegated and keeps its weight scales. Dynamically quantized linears from to_edge_transform_and_lower (8da4w and 8da8w: 8-bit dynamic activations with 4-bit or 8-bit weights) are unchanged, because the linear itself is delegated and is not decomposed. With the older to_edge() and to_backend() flow, 8da4w and 8da8w linears that used to be delegated with wrong weights now fail loudly the same way. Add a test with an int4 weight-only linear, with and without bias: with float activations its gemm stays out of the delegate, and with dynamically quantized activations it is delegated and matches eager.
torch.cummin has no ExecuTorch kernel. It is not part of the Core ATen opset and has no decomposition, so to_edge rejects it by default; a program that keeps it anyway (with cummin in _core_aten_ops_exception_list, or with IR validity checks turned off) then fails to load with "kernel 'aten::cummin.out' not found". This happens whenever cummin is left to the CPU, for example in a model lowered with XNNPACK or OpenVINO, or a TensorRT program whose cummin falls outside the engine. Add cummin.out to the portable kernels, following cumsum. For each position it writes the running minimum along dim and the index where that minimum was found. It matches PyTorch: on a tie the index moves to the later element, once a NaN appears it stays the minimum (with the index of the latest NaN), and a zero-dimensional input works with dim 0 or -1. All real types, Bool, Half and BFloat16 are supported. Test: op_cummin_test.cpp checks values and indices along both dims for every real type, Bool, NaN propagation, a zero-dimensional input, and that an invalid dim fails.
The OpenVINO backend converted each partition with openvino_compile, a helper written for torch.compile. That helper converts the graph first and sets the input types only afterwards. Two things break because of that: 1. Indexing with a boolean mask, like x[mask] or x[mask] = v, is lowered wrong. OpenVINO's converter looks at the index type to choose the boolean-mask lowering, but during conversion the type is still unknown, so the mask is treated as integer indices. A read gives wrong results, and a write fails with "Check 'args_et.is_dynamic() || is_supported_et' failed" once the boolean type is set. 2. A bfloat16 partition input fails with "KeyError: torch.bfloat16", because the helper's type table has no bfloat16 entry. Now the backend builds OpenVINO's PyTorch FX decoder with the partition input types and converts it with the same PyTorch frontend calls the helper used, then compiles the result with openvino.compile_model on the device from the compile specs. The converter sees the real types from the start, and its own type table already covers bfloat16. The helper's other options (model caching and an extra config dictionary) cannot be reached from ExecuTorch compile specs, so nothing else changes. A partition input can also be a SymInt, such as a data-dependent size. It has no dtype, so it is given to the converter as an int64 scalar, the form OpenVINO's FX translators expect. The runtime passes such an input as an int64 tensor of shape [1], so a PrePostProcessor step declares it that way and squeezes it back to a scalar. The runtime takes no other non-tensor input, so a float or bool one now fails at preprocess with a clear error instead of when the program runs. The new tests write through a boolean mask with float32 and bfloat16 data, and slice and view a tensor by a data-dependent size. They fail before this change and pass after it.
Exporting a model with ConvTranspose1d or ConvTranspose2d to the CUDA backend fails at the end of the AOTInductor compile with "missing fallback kernels: aoti_torch_cuda_convolution". This backend lets Inductor use only Triton kernels, and on CUDA Inductor has no Triton template for transposed convolution (it has none for 1-D, and the 2-D one is disabled on CUDA), so it falls back to ATen's convolution, which the ExecuTorch CUDA runtime does not have. The diffusion policy in LeRobot (ConditionalUnet1D upsampling) hits this. Add aten.conv_transpose1d and aten.conv_transpose2d to the backend's decomposition table, next to the existing conv1d entry. For finite weights, a transposed convolution equals a stride 1 convolution with the flipped kernel (input and output channels swapped per group) over the input with stride - 1 zeros inserted between elements and padded on each side by dilation * (kernel - 1) minus padding (plus output_padding at the end). Inductor generates Triton code for the zero insertion, the padding and the resulting convolution. The inserted zeros cost extra work (about stride times the multiply-adds per spatial dimension), which is the price of having a Triton path at all. On ROCm, Inductor still uses its Triton template for 2-D transposed convolution, so the 2-D entry is only added for CUDA. A transposed convolution with dilation > 1 becomes a dilated convolution, which has no Triton template, so on CUDA it still reports the missing kernel, as a regular dilated convolution does today. Test: test_conv_transpose lowers a ConvTranspose1d to the CUDA backend and checks the decompositions against conv_transpose1d and conv_transpose2d with stride, padding, output_padding, groups and dilation set, an unbatched input, and padding that crops more than the input.
When a constant is used by more than one partition, duplicate_constant_node
gives every extra user of the constant its own placeholder, and every copy
keeps the target of the original. So several placeholders can point at the
same state_dict or constants entry, and two of them can end up in the same
partition. exir already handles this when it removes delegated constants (it
skips targets that are still in use).
delete_constant_placeholder deleted the entry on the first call. That left
the other placeholders pointing at a missing tensor, and deleting a second
placeholder with the same target failed with a KeyError. The Arm
FuseEqualPlaceholdersPass hits this when it fuses such copies, for example
for a weight reused around an op that stays on the CPU:
KeyError: 'lin.bias'
Now the entry is removed only when no remaining placeholder refers to its
target, the same rule exir uses when it removes delegated constants.
duplicate_constant_node also stored a deep copy of the tensor under each
copy's node name (for example p_weight_copy_0). No input spec refers to
that key, so the data was never read, and it stayed in the program after
every placeholder was deleted. Stop storing it.
A model with a tensor-wide check, for example torch._check_tensor_all or transformers' torch_compilable_check, could not be lowered to the CUDA backend: Method forward missing fallback kernels: aten::_is_all_true torch.export turns the check into aten._is_all_true. The AOTI partitioner keeps every op for Inductor to handle, but Inductor has no lowering for this one, and ExecuTorch's CUDA shim has no fallback kernel for it. Add it to the CUDA backend's decomposition table as aten.all, which Inductor lowers, next to the existing conv1d entry.
Every CUDA program writes its weights to a file named aoti_cuda_blob.ptd, and keys each weight by its parameter name. When two programs with the same parameter names are exported into one folder, the second export overwrites the first program's weights file. The first program still loads, finds every key it asks for in the other program's file, and silently runs with the other program's weights. For example, a Linear layer exported twice with different seeds: the first program then returned the second one's outputs. Add each weight's SHA-256, which the exporter already computes while it writes the weight, to the weight's key. A program then finds only keys written for its own weights, and loading it with another program's file fails with "CUDA FQN storage ... is missing from named data". The bytes are not hashed again at load, so this catches files that belong to another program, not files corrupted in place. Identical weights still get the same key, so methods that share a weight still store it once. Within one export, a later method that stores different bytes under the same name is still rejected. The file name stays aoti_cuda_blob.ptd, because runners and scripts look for it. Two programs still need their own folders or file names, but a mix-up now fails at load instead of giving wrong results. This also means a .ptd with different weights, for example from a fine-tuned model, needs the .pte exported with it. test_missing_weights_blob.py exports two programs with different weights into one folder and checks that the first one refuses the second one's file. Without this change it loads. Tests that read the stored keys now take them from the delegate's manifest, which is what the runtime reads.
shoumikhin
force-pushed
the
fix-lerobot-export-bugs
branch
from
October 29, 2026 15:09
9136f25 to
204bbc8
Compare
Contributor
Author
|
@pytorchbot cherry-pick --onto release/1.6 -c critical |
Collaborator
Cherry picking #24222The cherry pick PR is at #24298 and it is recommended to link a critical cherry pick PR with an issue. Details for Dev Infra teamRaised by workflow job |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes found while exporting models to ExecuTorch backends.
Fixes
constant_prop_passkeeps tensors that in-place ops write, also through views or inno_grad/autocastblocks.to_backendkeeps the output node last.to_edgelowerstorch._check_tensor_allandtorch.sym_ite.GroupBasedPartitionerno longer makes partitions that depend on each other, unless caller node groups already do. Tensor constants get static shapes; deduplicated constants keep their data.detachlowers, also insidetorch.cond. Strided outputs are copied. Transposed convolutions (1-D, 2-D, no dilation) andtorch._check_tensor_alllower.catrespects scoped configs. ELU is delegated when XNNPACK can run it.MatmulToBmmPasstakes a constant operand; Vulkan constants above 4 dims; non-strict programs in the NXP quantizer;cummin.out.Behavior changes
.pteloads only the weights it was exported with. A.ptdwith other weights, for example after fine-tuning, needs a newly exported.pte.GroupBasedPartitioneris about 2x slower on large graphs, and some graphs get a different number of partitions.to_backendkeeps the original order of top-level nodes.to_backend, fail at export instead of giving wrong results. Delegated ELU flushes subnormal values to zero. Cats outside a scoped quantizer config are no longer quantized.We would like to cherry-pick this PR to release/1.6 after it merges. It needs #23915 there first.
Testing
Each fix has a test that fails without it. Run on an A100 (CUDA, Vulkan), an x86 CPU (OpenVINO, QNN SDK), macOS (Core ML, MLX), the TOSA reference model and VGF emulation (Arm), and the eIQ Neutron SDK (NXP). Not run: Metal, ROCm, Windows CUDA.