[pybind] Improve pip package build#6142

pytorchbot wants to merge 1 commit intorelease/0.4pytorch/executorch:release/0.4from

cherry-pick-5965-by-pytorch_bot_bot_pytorch/executorch:cherry-pick-5965-by-pytorch_bot_bot_Copy head branch name to clipboard