[Core ML] Rank-0 Input

[YifanShenSZ]

When we try to partition certain models on to Core ML delegation, we were hit by

ValueError: Rank-0 (input None) is unsupported

This is because Core ML does not support rank-0 (scalar) input, yet ExecuTorch tries to send a graph with scalar input to Core ML backend

We will need to wrap those scalars into rank-1 shape-1 tensors before sending to Core ML backend

cc @JacobSzwejbka @angelayi @kimishpatel @cymbalrush

[cccclai]

From top of my brain, we can have a pass to convert all scalar to tensor before lower it to CoremL

[cymbalrush]

@YifanShenSZ is this still an issue?

[YifanShenSZ]

No longer urgent.

Concretely, the rank-0 "inputs" we encountered were constants, that ExecuTorch decided to keep and feed to backend during runtime. By letting Core ML backend own those constants (i.e. feed at AOT), we no longer see them as inputs.

[cccclai]

No longer urgent.

Concretely, the rank-0 "inputs" we encountered were constants, that ExecuTorch decided to keep and feed to backend during runtime. By letting Core ML backend own those constants (i.e. feed at AOT), we no longer see them as inputs.

Does it mean if there is real rank-0 input, the issue might still be there? If it is, it will be great to share an example, and I will change the tag from coreml to exir, as it sounds like a pass we can provide.

[YifanShenSZ]

Yes, here is a minimum reproduce based on executorch 0.2.0

import torch
import executorch.exir

import coremltools as ct


class Model(torch.nn.Module):
    def forward(self, x, y):
        return x + y
    

model = Model()
model.eval()


example_inputs = (torch.tensor(1.0), torch.tensor(2.0))
exported_program_manager_aten = torch.export.export(model, example_inputs)
exported_program_manager_edge = executorch.exir.to_edge(exported_program_manager_aten)
print(exported_program_manager_edge.exported_program())


coreml_model = ct.convert(exported_program_manager_edge.exported_program())

The exported program has scalar inputs

ExportedProgram:
    class GraphModule(torch.nn.Module):
        def forward(self, arg0_1: "f32[]", arg1_1: "f32[]"):
            # File: /Volumes/Models/LLM/Framework/CoreMLTools-Dev_ExecuTorch-0.2/Debug/rank-0_input/rank-0_input.py:9 in forward, code: return x + y
            aten_add_tensor: "f32[]" = executorch_exir_dialects_edge__ops_aten_add_Tensor(arg0_1, arg1_1);  arg0_1 = arg1_1 = None
            return (aten_add_tensor,)

and CoreML conversion would fail

ValueError: Rank-0 (input None) is unsupported

[cccclai]

This one still needs to be fixed..

[cccclai]

Here is the pull request to solve the issue #8298

[cbilgin]

@cccclai, @YifanShenSZ looks like PR merged. Is this resolved and can it be closed?

[YifanShenSZ]

Yes, this is now fully fixed

1. In CoreMLTools we already do the wrapping
2. Before Add a pass to convert rank-0 tensor to rank-1 tensor #8298 ExecuTorch would be surprised that "scalar input becomes tensor", but now it is fixed