Initial draft of the Partial Fanout changes in the spec

src/Hydra/Protocol/Figures/head-protocol-states.tex

@@ -21,7 +21,8 @@
     \path[->] (open) edge [bend left=20] node {$\stClose$} (closed);
     \path[->] (open) edge [loop above] node {$\mathsf{increment}$} (open);
     \path[->] (open) edge [loop below] node {$\mathsf{decrement}$} (open);
-    \path[->] (closed) edge [bend left=20] node {$\stFanout$} (final);
+    \path[->] (closed) edge [bend left=20] node {$\stFanout$ (complete)} (final);
+    \path[->] (closed) edge [loop right] node {$\stFanout$ (partial)} (closed);
     \path[->] (closed) edge [loop above] node {$\stContest$} (closed);
     \path[->] (initial) edge [bend right=20] node {$\stAbort$} (final);
   \end{tikzpicture}

src/Hydra/Protocol/Figures/offchain-protocol.tex

@@ -129,9 +129,9 @@
 								$U \gets U_{\mathsf{active}} \applytx \underline{\tx}_{\mathsf{req}}$ \;
 								$\hats \gets s$ \;
 								% TODO: DRY message creation
-								$\eta \gets \combine(U)$ \;
-								$\eta_\alpha \gets \mathsf{combine}(U_\alpha)$ \;
-								$\eta_\omega \gets \mathsf{combine}(\mathsf{outputs}(\tx_\omega))$ \;
+								$\eta \gets \accUTxO(U)$ \;
+								$\eta_\alpha \gets \accUTxO(U_\alpha)$ \;
+								$\eta_\omega \gets \accUTxO(\mathsf{outputs}(\tx_\omega))$ \;
 								$\msSig_i \gets \msSign(\hydraSigningKey, (\cid || v || \hats || \eta || \eta_\alpha || \eta_\omega))$ \;
 								% TODO: use a seen snapshot to keep track of things easier
 								$\hatSigma \gets \emptyset$ \;
@@ -165,11 +165,11 @@
 									$\msCSig \gets \msComb(\hydraKeys^{setup}, \hatSigma)$ \;
 
 									% TODO: DRY message creation
-									$\eta \gets \combine(\hatmU)$ \;
+									$\eta \gets \accUTxO(\hatmU)$ \;
 
-									$\eta_\alpha \gets \mathsf{combine}(U_\alpha)$ \;
+									$\eta_\alpha \gets \accUTxO(U_\alpha)$ \;
 									$U_\omega \gets \mathsf{outputs}(\tx_\omega)$ \;
-									$\eta_\omega \gets \mathsf{combine}(U_\omega)$ \;
+									$\eta_\omega \gets \accUTxO(U_\omega)$ \;
 									% NOTE: Implementation differs here and
 									% below as it stores seen version in seen
 									% snapshot and uses that to verify
@@ -261,9 +261,9 @@
 				\begin{walgo}{0.6}
 					% CLOSE from client
 					\On{$(\hpClose)$ from client}{
-						$\eta \gets \combine(\bar{\mc S}.U)$ \;
-						$\eta_\alpha \gets \combine(\bar{\mc S}.U_\alpha$) \;
-						$\eta_\omega \gets \combine(\bar{\mc S}.U_\omega)$ \;
+						$\eta \gets \accUTxO(\bar{\mc S}.U)$ \;
+						$\eta_\alpha \gets \accUTxO(\bar{\mc S}.U_\alpha$) \;
+						$\eta_\omega \gets \accUTxO(\bar{\mc S}.U_\omega)$ \;
 						$\xi \gets \bar{\mc S}.\sigma$ \;
 						% XXX: \hatv needed to distinguish between CloseType redeemer, explain how exactly?
 						$\PostTx{}~(\mtxClose, \hatv, \bar{\mc S}.v, \bar{\mc S}.s, \eta , \eta_\alpha, \eta_\omega, \xi)$ \;
@@ -275,9 +275,9 @@
 					% CLOSE TX
 					\On{$(\gcChainClose, \eta) \lor (\gcChainContest, s_{c}, \eta)$ from chain}{
 						\If{$\bar{\mc S}.s > s_{c}$}{
-							$\eta \gets \combine(\bar{\mc S}.U)$ \;
-							$\eta_\alpha \gets \combine(\bar{\mc S}.U_\alpha$) \;
-							$\eta_\omega \gets \combine({\bar{\mc S}.U_\omega})$ \;
+							$\eta \gets \accUTxO(\bar{\mc S}.U)$ \;
+							$\eta_\alpha \gets \accUTxO(\bar{\mc S}.U_\alpha$) \;
+							$\eta_\omega \gets \accUTxO({\bar{\mc S}.U_\omega})$ \;
 							$\xi \gets \bar{\mc S}.\sigma$ \;
 							% XXX: \hatv needed to distinguish between CloseType redeemer, explain how exactly?
 							$\PostTx{}~(\mtxContest, \hatv, \bar{\mc S}.v, \bar{\mc S}.s, \eta , \eta_\alpha, \eta_\omega , \xi)$ \;

src/Hydra/Protocol/OffChain.tex

@@ -7,8 +7,16 @@ \section{Off-Chain Protocol}\label{sec:offchain}
 full life-cycle of a Hydra head on-chain, this section completes the picture by
 defining how the protocol behaves off-chain and notably the relationship between
 on- and off-chain semantics. Participants of the protocol are also called Hydra
-head members, parties or simply protocol actors. The protocol is specified as a
-reactive system that processes three kinds of inputs:
+head members, parties or simply protocol actors.
+
+\noindent The off-chain protocol coordinates BLS accumulator operations (see Section~\ref{sec:bls-accumulators})
+for partial fanout support. When UTxO sets are too large for single transactions,
+the off-chain protocol automatically falls back to partial distribution using
+exclusion proofs, ensuring all UTxOs can be distributed across multiple transactions
+while maintaining cryptographic integrity.
+
+The protocol is specified as a reactive system that processes three kinds of inputs:
+
 \begin{enumerate}
   \item On-chain protocol transactions as introduced in
         Section~\ref{sec:on-chain}, which are posted to the mainchain and can be
@@ -23,8 +31,7 @@ \section{Off-Chain Protocol}\label{sec:offchain}
           \item $\mathtt{decrementTx}$: removes UTxO from an open head
           \item $\mathtt{closeTx}$: closes a head
           \item $\mathtt{contestTx}$: contests a closed head
-          % NOTE: fanout not mentioned because not needed in off-chain protocol
-          % description
+          \item $\mathtt{fanoutTx}$: distributes UTxOs from a closed head (partial or complete)
         \end{itemize}
         Also, a special input when time advanced on chain may be used:
         \begin{itemize}
@@ -306,6 +313,16 @@ \subsubsection{Processing transactions off-chain}
   is incremented on each \mtxIncrement{} transaction as described in
   Section~\ref{sec:increment-tx}
 
+\dparagraph{$\mathtt{fanoutTx}$.}\quad Upon observing a \mtxFanout{}
+transaction, the distributed UTxOs are removed from the local state tracking.
+If this was a partial fanout (not all UTxOs were distributed), the head remains
+in $\stClosed$ state with updated accumulator containing the remaining UTxOs.
+If this was a complete fanout (all UTxOs distributed), the head transitions to
+$\stFinal$ state. The off-chain protocol must coordinate partial fanout to
+ensure proper selection of UTxO subsets and generation of exclusion proofs.
+\todo{Add detailed off-chain protocol for partial fanout coordination, including
+selection strategy for UTxO subsets.}
+
 \subsubsection{Closing the head}
 
 \dparagraph{$\hpClose$.}\quad In order to close a head, a client issues the
@@ -388,6 +405,7 @@ \subsection{Rollbacks and protocol changes}\label{sec:rollbacks}
 \input{Hydra/Protocol/Figures/offchain-protocol}
 
 \todo{In figure: $\combine$ on UTxO slightly different than on commits}
+\todo{Update off-chain protocol to support partial fanout coordination}
 
 %%% Local Variables:
 %%% mode: latex

src/Hydra/Protocol/OnChain.tex

@@ -1,9 +1,14 @@
 \clearpage
 \section{On-chain Protocol}\label{sec:on-chain}
 \todo{Update figures}
+\todo{Update fanoutTx.svg figure to show partial fanout concept - either complete distribution or partial distribution with remaining UTxOs}
 
 \todo{Open problem: ensure abort is always possible. e.g. by individual aborts or undoing commits}
 \todo{Open problem: ensure fanout is always possible, e.g. by limiting complexity of $U_0$}
+\todo{Update UTxO set digest mechanism to support partial fanout when transaction size limits are exceeded}
+\todo{Update combine function: change from simple hash to commitment scheme that supports partial distribution proofs}
+\todo{Implement fallback mechanism: try full fanout first, fall back to partial fanout if transaction size limit exceeded}
+\todo{Consider transaction chaining for seamless partial fanout execution}
 
 \noindent The following sections describe the the \emph{on-chain} protocol
 controlling the life-cycle of a Hydra head, which can be intuitively described
@@ -12,6 +17,12 @@ \section{On-chain Protocol}\label{sec:on-chain}
 protocol transaction
 on-chain: \mtxInit{}~\ref{sec:init-tx}, \mtxCom{}~\ref{sec:commit-tx}, \mtxAbort{}~\ref{sec:abort-tx}, \mtxCollect{}~\ref{sec:collect-tx}, \mtxIncrement{}~\ref{sec:increment-tx}, \mtxDecrement{}~\ref{sec:decrement-tx}, \mtxClose{}~\ref{sec:close-tx}, \mtxContest{}~\ref{sec:contest-tx}, and \mtxFanout{}~\ref{sec:fanout-tx}. \\
 
+\noindent The protocol uses BLS accumulators (see Section~\ref{sec:bls-accumulators}) to enable
+partial fanout when UTxO sets exceed transaction size limits. The \mtxCollect{} transaction
+creates BLS accumulator commitments to UTxO sets, and \mtxFanout{} transactions use
+exclusion proofs to distribute subsets of UTxOs while maintaining cryptographic guarantees
+about the remaining UTxOs. \\
+
 \noindent Besides the main state transitions of the head protocol, there is
   the related ``deposit protocol'' with two transactions in support of
   \mtxIncrement{}: \mtxDeposit{}~\ref{sec:deposit-tx} and \mtxRecover{}~\ref{sec:recover-tx}. \\
@@ -235,6 +246,7 @@ \subsection{Abort Transaction}\label{sec:abort-tx}
   \[
 	\hash(\bigoplus_{j=1}^{m} \bytes(o_{j})) = \combine([C_{i} ~ | ~ \forall [1 \dots |\hydraKeys|], C_{i} \neq \bot])
   \]
+  \todo{Update abort transaction to support partial distribution when all commits together are too big for a single transaction}
 
   \item Transaction is signed by a participant $\exists \{\cid \mapsto \keyHash_{i} \mapsto -1\} \in \txMint \Rightarrow \keyHash_{i} \in \txKeys$.
   \item All tokens are burnt
@@ -279,13 +291,13 @@ \subsection{CollectCom Transaction}\label{sec:collect-tx}
   \]
   where $n = |\hydraKeys|$ and
   \[
-	\combine(\underline{C}) = \hash(\mathsf{concat}({\sortOn(1, \mathsf{concat}(\underline{C}))}^{\downarrow2}))
+	\combine(\underline{C}) = \accUTxO(\mathsf{concat}({\sortOn(1, \mathsf{concat}(\underline{C}))}^{\downarrow2}))
   \]
   That is, given a list of committed UTxO $\underline{C}$, where each element is
   a list of output references and the serialised representation of what was
   committed, $\combine$ first concatenates all commits together, sorts this list
-  by the output references, concatenates all bytes and hashes the
-  result\footnote{Sorting is required to ensure a canonical representation which
+  by the output references, and creates a commitment to the UTxO set that supports
+  partial fanout\footnote{Sorting is required to ensure a canonical representation which
 	can also be reproduced from the UTxO set later in the fanout.}.
 
   \item All committed value captured and no value is extracted
@@ -393,8 +405,9 @@ \subsection{Recover Transaction}\label{sec:recover-tx}
   comparing hashes of serialised representations of the $m$ recovering outputs
   $o_{1} \dots o_{m}$ with the canonically combined committed UTxOs in $C$:
   \[
-    \hash(\bigoplus_{j=1}^{m} \bytes(o_{j})) = \hash(\mathsf{concat}({\sortOn(1, C)}^{\downarrow2}))
+    \hash(\bigoplus_{j=1}^{m} \bytes(o_{j})) = \combine(C)
   \]
+  \todo{Consider partial recovery for deposit transactions when fractional deposits allow multiple UTxO and some are too small to recover individually}
   \item Transaction is posted after the deadline
   \[
     \txValidityMin > t_{\mathsf{recover}}
@@ -465,8 +478,9 @@ \subsection{Increment Transaction}\label{sec:increment-tx}
   where $\eta_\alpha$ is the digest of all deposited UTxO in $C$ sorted by their output
   references
   \[
-	\eta_\alpha = \hash(\mathsf{concat}({\sortOn(1, C)}^{\downarrow2}))
+	\eta_\alpha = \combine(C)
   \]
+  \todo{Update increment transaction to support partial distribution if needed}
   \item The value in the head output is increased accordingly
   \[
 	\valHead \cup \valDeposit = \valHead'
@@ -519,8 +533,9 @@ \subsection{Decrement Transaction}\label{sec:decrement-tx}
   \]
   where $\eta_\omega$ is the digest of all removed UTxO
   \[
-	\eta_\omega = \hash(\bigoplus_{j=2}^{m+1} \bytes(o_{j}))
+	\eta_\omega = \accUTxO(\bigoplus_{j=2}^{m+1} \bytes(o_{j}))
   \]
+  \todo{Consider partial distribution for decrement transactions when the withdrawal set is too big for a single transaction}
   \item The value in the head output is decreased accordingly
   \[
 	\valHead' \cup (\bigcup_{j=2}^{m+1} \val_{j}) = \valHead
@@ -812,51 +827,59 @@ \subsection{Contest Transaction}\label{sec:contest-tx}
 
 \subsection{Fan-Out Transaction}\label{sec:fanout-tx}
 
-Once the contestation phase is over, a head may be finalized by posting a
-\mtxFanout{} transaction (see Figure~\ref{fig:fanoutTx}), which
-distributes UTxOs from the head according to the latest state. It consists of
+Once the contestation phase is over, a head may be finalized by posting one or more
+\mtxFanout{} transactions (see Figure~\ref{fig:fanoutTx}), which
+distribute UTxOs from the head according to the latest state. The protocol first attempts
+a complete fanout of all UTxOs. If this fails due to transaction size limits, it automatically
+falls back to partial fanout, distributing a subset of UTxOs and proving the remaining UTxOs
+are still valid. A fanout transaction consists of
 \begin{itemize}
   \item one input spending from $\nuHead$ holding the $\st$, and
-  \item outputs $o_{1} \dots o_{m+n+n'}$ to distribute UTxOs.
+  \item outputs $o_{1} \dots o_{m+n+n'}$ to distribute a subset of UTxOs.
 \end{itemize}
 
-Note that \mtxFanout{} represents a final transition of the state machine and
-hence there is no state machine output.
+Note that \mtxFanout{} may represent either a final transition of the state machine
+(when all UTxOs are distributed) or an intermediate transition (when only a subset
+is distributed). In the latter case, the state machine remains in $\stClosed$ state
+with updated accumulator containing the remaining UTxOs.
 
 \begin{figure}
   \centering
   \includesvg[width=0.8\textwidth]{Hydra/Protocol/Figures/fanoutTx.svg}
   \caption{\mtxFanout{} transaction spending the $\stClosed$ head output and
-	distributing funds with outputs $o_{1} \dots o_{m+n+n'}$.}\label{fig:fanoutTx}
+	distributing funds with outputs $o_{1} \dots o_{m+n+n'}$. The transaction can
+	distribute either all UTxOs (complete fanout) or a subset (partial fanout) with
+	proofs that remaining UTxOs are still valid.}\label{fig:fanoutTx}
 \end{figure}
 
 \noindent The state-machine validator $\nuHead$ is spent with
-$\redeemerHead = (\mathsf{fanout}, m, n, n')$, where $m$, $n$ and $n'$ are
-outputs to distribute from the $\stClosed$ state, and checks:
+$\redeemerHead = (\mathsf{fanout}, m, n, n', \pi_{\eta}, \pi_{\alpha\Delta}, \pi_{\omega\Delta}, \eta'_{remaining})$, where:
+\begin{itemize}
+  \item $m$, $n$ and $n'$ are outputs to distribute from the $\stClosed$ state
+  \item $\pi_{\eta}$, $\pi_{\alpha\Delta}$, $\pi_{\omega\Delta}$ are partial distribution proofs for the distributed UTxO sets
+  \item $\eta'_{remaining}$ is the updated commitment containing remaining UTxOs
+\end{itemize}
+The validator checks:
 \begin{menumerate}
-  \item State is advanced from $\datumHead \sim \stClosed$ to terminal state
-  $\stFinal$: % XXX: What does this actually mean?
+  \item State transition: If all UTxOs are distributed, transition to $\stFinal$; otherwise remain in $\stClosed$ with updated accumulator:
   \[
-	(\stClosed,\cid,\hydraKeys,\Tcontest,v, s,\eta,\eta_\alpha\Delta,\eta_\omega\Delta,\contesters,\tfinal) \xrightarrow[m,n,n']{\stFanout} \stFinal
+	(\stClosed,\cid,\hydraKeys,\Tcontest,v, s,\eta,\eta_\alpha\Delta,\eta_\omega\Delta,\contesters,\tfinal) \xrightarrow[m,n,n',\pi_{\eta},\pi_{\alpha\Delta},\pi_{\omega\Delta}]{\stFanout} \stFinal \text{ or } \stClosed'
   \]
-  \item The first $m$ outputs are distributing funds according to $\eta$. That is,
-  the outputs exactly correspond to the UTxO canonically combined $U^{\#}$ (see
-  Section~\ref{sec:collect-tx}):
+  \item The first $m$ outputs are distributing a subset of funds according to $\eta$. The exclusion proof $\pi_{\eta}$ proves that the distributed outputs $S_{\eta} = \{o_1, \ldots, o_m\}$ can be removed from accumulator $\eta$:
   \[
-	\eta = U^{\#} = \hash(\bigoplus_{j=1}^{m} \bytes(o_{j}))
+	\accVerifyPartial(\eta, S_{\eta}, \pi_{\eta}) = \true
   \]
-  \item The following $n$ outputs are distributing funds according to
-  $\eta_\alpha\Delta$. That is, the outputs exactly correspond to the UTxO canonically
-  combined $U^{\#}_{\alpha\Delta}$ (see Section~\ref{sec:collect-tx}):
+  \item The following $n$ outputs are distributing a subset of funds according to
+  $\eta_\alpha\Delta$. The exclusion proof $\pi_{\alpha\Delta}$ proves that the distributed outputs $S_{\alpha\Delta} = \{o_{m+1}, \ldots, o_{m+n}\}$ can be removed from accumulator $\eta_{\alpha\Delta}$:
   \[
-	\eta_{\alpha\Delta} = U^{\#}_{\alpha\Delta} = \hash(\bigoplus_{j=m}^{m+n} \bytes(o_{j}))
+	\accVerifyPartial(\eta_{\alpha\Delta}, S_{\alpha\Delta}, \pi_{\alpha\Delta}) = \true
   \]
-  \item The next $n'$ outputs are distributing funds according to
-  $\eta_\omega\Delta$. That is, the outputs exactly correspond to the UTxO canonically
-  combined $U^{\#}_{\omega\Delta}$ (see Section~\ref{sec:collect-tx}):
+  \item The next $n'$ outputs are distributing a subset of funds according to
+  $\eta_\omega\Delta$. The exclusion proof $\pi_{\omega\Delta}$ proves that the distributed outputs $S_{\omega\Delta} = \{o_{m+n+1}, \ldots, o_{m+n+n'}\}$ can be removed from accumulator $\eta_{\omega\Delta}$:
   \[
-	\eta_{\omega\Delta} = U^{\#}_{\omega\Delta} = \hash(\bigoplus_{j=m'}^{m+n'} \bytes(o_{j}))
+	\accVerifyPartial(\eta_{\omega\Delta}, S_{\omega\Delta}, \pi_{\omega\Delta}) = \true
   \]
+  \item If not all UTxOs are distributed, the remaining accumulator $\eta'_{remaining}$ is correctly computed by removing the distributed UTxOs from the original accumulators.
   \item Transaction is posted after contestation deadline $\txValidityMin > \tfinal$.
   \item All tokens are burnt
   $|\{\cid \mapsto \cdot \mapsto -1\} \in \txMint| = n + 1$.

src/Hydra/Protocol/Preliminaries.tex

@@ -37,6 +37,7 @@ \subsection{Notation}
 	\item $\concat : \tyBytes^* \to \tyBytes$ is concatenating bytes, we also use operator $\bigoplus$ for this
 	\item $\hash : x \to \tyBytes$ denotes a collision-resistant
 	      hashing function and $x^{\#}$ indicates the hash of $x$
+	      \todo{Update hash function usage to support partial fanout where needed}
 	\item $\bytes : x \to \tyBytes$ denotes an invertible serialisation function
 	      mapping arbitrary data to bytes
 	\item $a || b = \concat(\bytes(a), \bytes(b))$ is an operator which concatenates the $\bytes(b)$ to the $\bytes(a)$
@@ -83,6 +84,39 @@ \subsection{Public key multi-signature scheme}\label{sec:multisig}
 out the $ver$ suffix for verification key such that $k = k^{ver}$ for better
 readability.
 
+\subsection{BLS Accumulators for Partial Fanout}\label{sec:bls-accumulators}
+
+\noindent To enable partial fanout when UTxO sets are too large for a single transaction,
+the protocol uses BLS (Boneh-Lynn-Shacham) accumulators that support partial distribution
+proofs. This allows distributing a subset of UTxOs while proving that the remaining
+UTxOs are still valid and can be distributed in subsequent transactions.
+
+\begin{definition}[BLS Accumulator]
+A BLS accumulator scheme provides the following operations:
+\begin{itemize}
+	\item $\accSetup$ generates public parameters for the accumulator system
+	\item $\accCommit(U)$ creates a commitment to a UTxO set $U$
+	\item $\accWitness(C, u)$ generates a membership witness for UTxO $u$ in commitment $C$
+	\item $\accVerify(C, u, w)$ verifies that witness $w$ proves $u$ is in commitment $C$
+	\item $\accExclude(C, S)$ generates an exclusion proof for subset $S$ from commitment $C$
+	\item $\accVerifyExclude(C, S, \pi)$ verifies that exclusion proof $\pi$ allows removal of $S$ from $C$
+\end{itemize}
+\end{definition}
+
+\begin{definition}[Partial Distribution]
+For a UTxO set $U$ and a subset $S \subseteq U$ to be distributed:
+\begin{itemize}
+	\item $\accUTxO(U)$ creates a BLS accumulator commitment to the UTxO set $U$
+	\item $\accPartial(U, S)$ creates an exclusion proof showing that subset $S$ can be distributed
+	\item $\accVerifyPartial(C, S, \pi)$ verifies that exclusion proof $\pi$ allows distribution of $S$ from commitment $C$
+\end{itemize}
+\end{definition}
+
+\noindent The BLS accumulator enables the protocol to handle large UTxO sets by automatically falling back
+to partial distribution when full fanout exceeds transaction size limits. The accumulator provides
+cryptographic guarantees that distributed UTxOs are valid and that remaining UTxOs are still
+available for future distribution.
+
 \subsection{Extended UTxO}\label{sec:eutxo}
 The Hydra Head protocol is specified to work on the so-called Extended UTxO (EUTxO) ledgers
 like Cardano.