The transaction CBOR produced by serialiseToRawBytes does not emit the transaction body map keys in ascending order. Although the transaction body is a CBOR map, key order is still significant for hardware wallets.
Devices like Ledger do not use the provided CBOR directly. They extract the fields, reconstruct the body internally, and then compute the transaction hash. Their reconstruction uses a sorted (ascending) key order. When the output is unsorted, the hardware wallet computes a different hash and produces an invalid signature.
To maintain hardware-wallet compatibility, the transaction body map should be sorted before serialisation. This is a straightforward change.
Note: The Cardano network itself is order-agnostic. The issue is strictly related to hardware-wallet signing behavior.