Shelley mempool per-tx size check: do not include perTxOverhead (#1352)

With the previous logic, a tx that has less than `perTxOverhead = 4`
bytes less than the max tx size will be rejected by the mempool even
though it is perfectly valid. This bug was introduced in #1175.

This behavior could be rather surprising for users who create a big (but
not too big) tx and then can't submit it to any node.

The fix is to use the "raw" size (without adding `perTxOverhead`) for
the check, but to still add `perTxOverhead` when returning the
`TxMeasure`.

Related: This code shouldn't live in Consensus (also see the module
header), cf IntersectMBO/cardano-ledger#4820.

Author: amesgen

ouroboros-consensus-cardano/changelog.d/20250107_124744_alexander.esgen_shelley_mempool_txsize_check.md

@@ -0,0 +1,4 @@
+### Patch
+
+- Fixed a bug where a valid tx with less than `4` bytes less than the max tx
+  size would be incorrectly rejected by the mempool.

ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Ledger/Mempool.hs

@@ -328,9 +328,9 @@ txInBlockSize ::
 txInBlockSize st (ShelleyTx _txid tx') =
     validateMaybe (maxTxSizeUTxO txsz limit) $ do
       guard $ txsz <= limit
-      Just $ IgnoringOverflow $ ByteSize32 $ fromIntegral txsz
+      Just $ IgnoringOverflow $ ByteSize32 $ fromIntegral txsz + perTxOverhead
   where
-    txsz = perTxOverhead + (tx' ^. sizeTxF)
+    txsz = tx' ^. sizeTxF
 
     pparams = getPParams $ tickedShelleyLedgerState st
     limit   = fromIntegral (pparams ^. L.ppMaxTxSizeL) :: Integer