Allow Group Channels to contain Extended Channels

[plebhash]

There's been a lot of discussions in the Sv2 community around some fundamental UX aspect of Translator Proxy.

It can work in two different modes:

• aggregated: where all Sv1 workers receive jobs and submit work under one single Extended Channel
• non-aggregated: where each Sv1 worker establishes a dedicated Extended Channel

Agrregated mode brings a lot of efficiency in terms of bandwidth and job broadcasting. However, it has the downside of losing data granularity, meaning that the user cannot look at the Pool dashboard and visualize metrics on a per-worker basis. This is the UX that miners are used to today, and forcing them to move away from it could become a barrier for protocol adoption.

Non-aggregated mode allows for such granularity, but loses all the efficiency optimizations of aggregation.

Sv2 has the notion of Group Channels, which are meant to make job broadcasting more efficient. This could be an interesting option to optimize non-aggregated mode.

And now the main problem:

• Currently, the Sv2 spec only allows for Group'ing of Standard Channels, not Extended Channels.
• Sv1 blocks the possibility of using Standard Channels (with a fixed merkle root) on a Translator Proxy.

The fact that Group Channels can only contain Standard Channels (but not Extended) is a bit counter-intuitive at first sight, because when it comes to the design space in front of us, there's no fundamental blocker preventing the protocol from allowing this.

But this restriction becomes clear when we read the following sections:

• 5.2.3 Group Channel
• 5.3.16 NewExtendedMiningJob
• 5.3.22 SetGroupChannel

This Discussion is meant to try to get consensus on changing the Mining Spec, in order to allow Group Channels to also contain Extended Channels.

This would enable a Translator Proxy that retains a legacy-friendly UX (with per-worker data granularity via non-aggregated mode) that can still receive jobs broadcasts from a Pool in an efficient way.

The proposal is split in two parts, where the second is optional and complimentary to the first.

[plebhash]

Non-breaking changes

Here, by non-breaking, I mean that we're not adding or removing fields from any message. It's merely changing conceptual definitions.

Implementations that pre-date to this change would remain spec-compliant, only missing new functionality. This is akin to a soft-fork in bitcoin consensus.

Definition of Group Channel

This would change how the definition of how Group Channel is formally established in the spec.

### 5.2.3 Group Channel
 
-Standard channels opened within one particular connection can be grouped together to be addressable by a common communication group channel.
+Standard and Extended channels opened within one particular connection can be grouped together to be addressable by a common communication group channel.
 
-Whenever a Standard Channel is created, it is always put into some Group Channel identified by its `group_channel_id`.
-Group Channel ID namespace is the same as Standard Channel ID namespace on a particular connection.
+Whenever a Standard or Extended Channel is created, it is always put into some Group Channel identified by its `group_channel_id`.
+Group Channel ID namespace is the same as Standard and Extended Channel ID namespace on a particular connection.

Definition of Extended Job

This would rephrase how we're defining Extended Job with regards to Group Channel.

### 5.3.16 `NewExtendedMiningJob` (Server -> Client)

(Extended and group channels only)

For an **extended channel**:
The whole search space of the job is owned by the specified channel.
If the `min_ntime` field is set to some nTime, the client MUST start to mine on the new job as soon as possible after receiving this message.

 For a **group channel**:
-This is a broadcast variant of `NewMiningJob` message with the `merkle_root` field replaced by `merkle_path` and coinbase transaction prefix and suffix, for further traffic optimization.
-The Merkle root is then defined deterministically for each channel by the common `merkle_path` and unique `extranonce_prefix` serialized into the coinbase.
-The full coinbase is then constructed as follows: `coinbase_tx_prefix + extranonce_prefix + coinbase_tx_suffix`.
+This acts as a broadcast message that distributes work to all channels under the same group.
 
 The proxy MAY transform this multicast variant for downstream standard channels into `NewMiningJob` messages by computing the derived Merkle root for them.
-A proxy MUST translate the message for all downstream standard channels belonging to the group which don’t signal that they accept extended mining jobs in the `SetupConnection` message (intended and expected behavior for end mining devices).
+A proxy MUST translate the message into `NewMiningJob` for all downstream standard channels belonging to the group in case the `SetupConnection` message had the `REQUIRES_STANDARD_JOB` flag set (intended and expected behavior for end mining devices).
+
+For conversion into different `NewMiningJob` messages, the Merkle root is then defined deterministically for each standard channel by the common `merkle_path` and unique `extranonce_prefix` serialized into the coinbase.
+The full coinbase is then constructed as follows: `coinbase_tx_prefix + extranonce_prefix + coinbase_tx_suffix`.

 | Field Name              | Data Type      | Description                                                                                                                                                                                   
                                                                                                                                                                                                                                            
                                                       |
 | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-| channel_id              | U32            | For a group channel, the message is broadcasted to all standard channels belonging to the group. Otherwise, it is addressed to the specified extended channel.                                
                                                                                                                                                                                                                                            
                                                       |
+| channel_id              | U32            | For a group channel, the message is broadcasted to all standard and extended channels belonging to the group. Otherwise, it is addressed to the specified extended channel.  

Definition of SetGroupChannel message

This would rephrase how we're defining the SetGroupChannel message.

### 5.3.22 `SetGroupChannel` (Server -> Client)
 
-Every standard channel is a member of a group of standard channels, addressed by the upstream server's provided identifier.
-The group channel is used mainly for efficient job distribution to multiple standard channels at once.
+The group channel is used mainly for efficient job distribution to multiple standard and extended channels at once.
 
-If we want to allow different jobs to be served to different standard channels (e.g. because of different [BIP 8](https://github.com/bitcoin/bips/blob/master/bip-0008.mediawiki) version bits) and still be able to distribute the work by sending `NewExtendendedMiningJob` instead of a repeated `NewMiningJob`, we need a more fine-grained grouping for standard channels.
+If we want to allow different jobs to be served to different standard and extended channels (e.g. because of different [BIP 8](https://github.com/bitcoin/bips/blob/master/bip-0008.mediawiki) version bits) and still be able to distribute the work by sending `NewExtendendedMiningJob` instead of multiple `NewMiningJob` and/or `NewExtendedMiningJob`, we need a more fine-grained grouping for channels.
 
-This message associates a set of standard channels with a group channel.
+This message associates a set of standard and extended channels with a group channel.
 A channel (identified by particular ID) becomes a group channel when it is used by this message as `group_channel_id`.
 The server MUST ensure that a group channel has a unique channel ID within one connection. Channel reinterpretation is not allowed.
 
 This message can be sent only to connections that don’t have `REQUIRES_STANDARD_JOBS` flag in `SetupConnection`.
 
| Field Name       | Data Type     | Description                                                                               |
| ---------------- | ------------- | ----------------------------------------------------------------------------------------- |
-| group_channel_id | U32           | Identifier of the group where the standard channel belongs                                |
-| channel_ids      | SEQ0_64K[U32] | A sequence of opened standard channel IDs, for which the group channel is being redefined |
+| group_channel_id | U32           | Identifier of the group where the standard or extended channel belongs                                |
+| channel_ids      | SEQ0_64K[U32] | A sequence of opened standard or extended channel IDs, for which the group channel is being redefined |

[GitGab19]

According to this overview, it seems the most problematic scenarios are A and D.

In those scenarios I guess the only thing which could happen is a fallback to a secondary Pool:

• Scenario A -> proxy would not find any channel with id=0, so it would fallback
• Scenario D -> Pool would not find any channel with id=0, it would send a SetCustomMiningJob.Error, and proxy would fallback

Am I correct on this? Is there any other better way to handle those two scenarios?

[plebhash]

I'd say that's correct.

But with a reminder that A is purely theoretical. At least to our knowledge, the only non-aggregating proxy in existence today is the one about to be merged in stratum-mining/stratum#1791

As long as we adapt it soon after this potential spec change happens, scenario A would not cause any problems in the real world.

[plebhash]

but in case everyone feels the "non-breaking" label is inaccurate after reading the analysis above, we could also consider the versioning on SetupConnection.{min_version,max_version}, which was explained by @jakubtrnka here #82 (comment)

[plebhash]

@Fi3 still NACK?

[Fi3]

ACK

[plebhash]

Breaking changes

This is an optional part of the proposal. It could face a bit more resistance, since it would break existing implementations.

Right now, OpenStandardMiningChannel.Success message has a group_channel_id field, while OpenExtendedMiningChannel.Success does not.

If we are allowing Extended Channels to belong to a Group Channel, ideally OpenExtendedMiningChannel.Success should have a group_channel_id field as well.

### 5.3.5 `OpenExtendedMiningChannel.Success` (Server -> Client)

Sent as a response for opening an extended channel.

| Field Name        | Data Type | Description                                                                                                                                                  |
| ----------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| request_id        | U32       | Client-specified request ID from OpenExtendedMiningChannel message, so that the client can pair responses with open channel requests                         |
| channel_id        | U32       | Newly assigned identifier of the channel, stable for the whole lifetime of the connection, e.g. it is used for broadcasting new jobs by NewExtendedMiningJob |
| target            | U256      | Initial target for the mining channel                                                                                                                        |
| extranonce_size   | U16       | Extranonce size (in bytes) set for the channel                                                                                                               |
| extranonce_prefix | B0_32     | Bytes used as implicit first part of extranonce                                                                                                              |
+| group_channel_id  | U32       | Group channel into which the new channel belongs. See SetGroupChannel for details.                                                                           |

Otherwise, it's hard to say which Group Channel some specific Extended Channel belongs to once it's open, and could lead to undefined states.

If this breaking change is rejected, but we still move forward with the non-breaking proposal, then Proxies could still assume some value for group_channel_id (e.g.: 0), until a SetGroupChannel message assigns a different one. But we should still formalize this as a convention into the non-breaking changes, in order to avoid undefined states.

[jbesraa]

Agrregated mode brings a lot of efficiency in terms of bandwidth and job broadcasting. However, it has the downside of losing data granularity, meaning that the user cannot look at the Pool dashboard and visualize metrics on a per-worker basis. This is the UX that miners are used to today, and forcing them to move away from it could become a barrier for protocol adoption.

Can you please specify to what data or metrics are you referring here?

A general note here is that any data that cannot be collected by the pool, can always be collected by the proxy. If the pool does not need this data to operate, I don't think it should know about.

[Fi3]

btw as I already said in per discussion about monitoring extension. We should aggregate information about every worker at proxy level and sending upstream the aggreagated infos. For example {difficulty, number_of_shares_accepted, number_of_share_rejected} you can do that every n minutes. And still send info about last hashratre per worker every 10 sec if you want to be more responsive. You want also to send info about temperature voltage ecc ecc, so the extension should account also for that info. With such extension you don't need to change the protocol, you will get a lot less traffic to the pool, you will get much more infos. (For example you can send errors for share rejection and things like that)

[colerob1776]

The proposal isnt about miscellaneous data, it's about the pool validating shares of each individual miner so the miners can compare the pool's payout data and the miner's share data to ensure pools are paying out correctly. Just as a GroupChannel does

[Fi3]

still not getting what this proposal achieve that the extension can not achieve

[Fi3]

The proposal isnt about miscellaneous data, it's about the pool validating shares of each individual miner so the miners can compare the pool's payout data and the miner's share data to ensure pools are paying out correctly. Just as a GroupChannel does

for doing that you really don't need to send EVERY share to the pool. You can achieve it without extension and without what proposed here. You just need a proxy that aggregate shares and you verify that shares sent up by the proxy are payed. You really do not need anything else. Ofc this is enough fro FPPS if you do PPLNS yuo need to trust the pool to not inflate pool's hashrate (and this proposal certainly do not solve this issue)

[Fi3]

and ofc if you really want to send EVERY share upstream for whatever reason you can do with the extension so what I'm saying is:

1. this use case it do not make much sense.
2. you can achieve the same exact result with an extension.
3. I don't want to change mining spec at this point.

still concept nack

[colerob1776]

Considering the original specification specifically includes GroupChannels, and the proposal is to allow for extended group channels, i think all pros and cons of GroupChannels have to be applied to the proposed extended group channel. Considering the pros outweighed the cons in the original spec definition, I think it's fair to keep those comparisons in mind when evaluating the proposal.

In addition, given the benchmarks here stratum-mining/stratum#1841, it's likely extended channels are more performant in most cases, especially on a local network with sufficient bandwidth. Forcing the proxy (TProxy or Sv2Proxy) to calculate merkle roots sequentially is much less efficient than letting individual devices calculate their own merkle roots in parallel and both ExtendedJobs and StandardJobs both easily fall within a standard Tcp Packet, so there should be little to no networking hit. Sure there is more data to copy over Tcp but copying a buffer is much more efficient than caculating a merkle root. Given these points, i believe it's in Firmwares best interest to support connecting with extended jobs directly, and therefore this proposal may not ONLY affect the TProxy

Cons for ExtendedGroupChannels:

• Same as standard GroupChannels
• requires time and effort to implement
• Firmware doesnt support ExtendedChannels so only benefits TProxy for now

Pros for ExtendedGroupChannels:

• Same as standard GroupChannels
• Probably more efficient than standard GroupChannels in most scenarios
• Without them, less Sv1 miners will want to run the TProxy

[Fi3]

you don't need extended group channel for that. A miner will open an extended channel with (or standard non HOM) with proxy. So the proxy will have multiple extended channel one for each miner, and one extended channel opened with the pool, exactly like in sv1.

[colerob1776]

I think this disregards the use of GroupChannels. Why not just have an Sv2 proxy connect to the pool on an extended channel and let StandardChannels connect downstream, while aggregating shares? Then GroupChannels aren't even necessary

[plebhash]

framing @colerob1776 's argument above from a slightly different perspective:

saying "just aggregate" misses the point, because as far as I can infer on the protocol design, Group Channels are an optimization meant for non-aggregating proxies

if the protocol had been designed such that every proxy was forced to aggregate, the notion of Group Channels wouldn't be a part of it

the issue at hand here is that Extended Channels is left out of the picture, while there's actually no fundamental issue preventing them from belonging to Group Channels as well (aside from the fact that, well, the spec says so)

overall, with one small exception (group_channel_id on Open*.Success messages), this is merely a matter of how things are worded in the spec, and as stated above, can be easily mitigated in a non-breaking way

[pavlenex]

I've done some digging on this topic. One thing we often face at SRI when engaging with early adopters is that most of them always asks if there's a way to get individual workers share data.

Originally, I really didn't understand the reasons for this, and like @Fi3 I was against adding this into specs/protocol for this, because why would we settle for anything that is not the most efficient outcome. However, over time, it became obvious that there are users who want to have granularity in shares. I will talk about WHY users want this, a bit later, but this lead us to develop two modes in tProxy:

• Aggregating: Aggregates SV1 connections and translates SV1 into SV2 messages, maximizing efficiency.
• Non-aggregating: Performs translation only, providing miners with full data granularity but less efficiency.

Now the users who want to have this data for individual level at the Pool, can have it. However this option is pretty inefficient.

My understanding of this spec change is that it would alloExw non-aggregating proxy users to have a more efficient setup by allowing them to not aggregate shares, but do aggregate job dispatched which is a significant efficiency improvement.

Now, I get that it's hard to reason with this from technical perspective, but interaction with users actually shows that a very large portion of users doesn't want to aggregate shares, but still would like to have efficiency boost, which I think makes sense to cater for this use-case, especially since Braiins already uses Group Extended Channels.

There are two compelling reasons on why miners and Pools want to have individual granularity of the worker share data on the Pool level.

• User Experience: Miners rely on the pool’s dashboard to monitor each worker’s performance, including hashrate, accepted shares, payouts, and in some cases firmware-level metrics. With aggregation, this visibility is lost on the Pool level, since the pool now sees a single big worker, and pushed into the proxy, forcing miners to change established workflows and adopt new tools instead of using the familiar pool interface.
• Auditing & Compliance: For large or publicly traded miners, auditing relies on reconciling worker-level shares with pool records and payouts. Aggregation forces changes to established audit processes, requiring lawyers and auditors to revise documentation. This adds cost, bureaucracy, and friction that many prefer to avoid.

One thing that has always concerned me about the plugin route, especially this early in SV2 adoption phase is compatibility. How will things work if one pool supports one, but the fallback happens and the other doesn't? Then your setup becomes quite fragile. Imagine your auditing breaks ,because fallback happened. We can of course wait to find out, but should we really push users into using experimental extension system, especially if majority that we've interacted with expressed the necessity for it, and they would be better supported if it worked right out of the box?

My point here, is that upon having quite a few interactions with miners, mining pools and some of the early adopters, I think this spec change would allow us to move forward with a behaviour that I would say over 70% of adopters asked for in the most efficient way possible, and it seems we can move into a non-breaking manner. I think extensions should be used for features that are opinionated and only a handful users want, but if a feature is something that majority of current and future adopters want, I think it's sensible to consider making a spec change.

[colerob1776]

we solved it in another way as @jbesraa described above. Absolutely no need to change the protocol for that.

This extension is just for monitoring. There is no proof behind the hashrate. If the pool does not know about the assigned difficulty of each individual device, any hashrate related data from the client is tentative at best, so in my opinion this is a fair fit for an extension only.

You just don't use pool that do not support it as fallback pool. What if you do FPPS and the fallback pool do PPLNS. You can make this arguments for a lot of other things that are at the end implementation details and do not belong in the protocol. As SRI you can provide an easy way to achieve that and it make sense. But changing the protocol make very little sense to me.

How does this proposal not support PPLNS? This proposal simply allows all necessary components in the system to have visibility into the work and submissions that a miner gets paid out for. PPLNS just relies on a sliding window as opposed to all shares with FPPS, so as long as device shares are connected into a timeseries DB (most likely by some client side agent) it's still possible to reconcile given the pools window and blocks mined.

It's still odd to me how GroupChannels received a spot in the protocol, but now when we realize ExtendedGroupChannels were left out, grouping doesnt make sense. Like I mentioned previously, you could just mine StandardChannel -> Proxy -> ExtendedChannel and aggregate hashrate with Sv2Firmware devices without GroupChannels, so any efficiency argument against the proposal does not resonate with me because the same argument can be made against regular GroupChannels.

[pavlenex]

I think with PPLNS and FFPS example Fi3 was just making a general observation on how extensions could be incompatible, highlighting that the incompatibility example I gave may happen in other cases as well, with different extensions. The difference here is just that this is likely to be a core feature used by many, so ensuring interoperability may be a good idea, which may be hard to to through extensions at this stage of adoptions. More extensions, narrow down on fallback pool choice, which I think we all agree isn't ideal for early stage of adoption.

It seems that we're getting close to consensus that this is needed, but what remains (at least from what I am reading) is agreeing on technical implementation and how (if) it , a protocol change affects existing Pool implementation.

This threading of discussions can sometimes be tricky to get freshest context, as some things mentioned by fi3 seems to be changed in more recent messages.

[colerob1776]

ahh sorry, i may have misinterpreted

[Fi3]

I think with PPLNS and FFPS example Fi3 was just making a general observation on how extensions could be incompatible

It was more like about fallback. If a miner expect/want/need FPPS it can't fallback to a pool the is PPLNS. Same thing can be said for a lot of other little thing. Btw if we reaaly need to save every share produced by every worker, I start thinking that maybe this little patch could be the easiest and way and also very efficient. Said that (1) we need to support custom job (2) I think that eventually we want to push for a system that make sense, so having things like that deeply rooted in the protocol is not the best imo.

For example if a miner want accountability what it change if it have it at proxy level or at device level? From a technical perspective absolutly nothing but the fact that one is much much more effecient than the other. And the big difference is for the pool operator here, so if we want a world where operating a pool is as easiest and cheapest as possible we should for sure push for a direction and not the other. And I think that as maintainers of SRI this should be our mission. So IMO an extension here that can be deprecated in the future is the best option.

[Fi3]

I would rather be more happy to completly remove standard channels and group channels from the mining protocol and make it simpler lol

[jakubtrnka]

IMO the entire concept of Group Channels is discussed as something very sophisticated, while the idea behind it is fairly simple.

first of all, it's actually not a real channel. Client doesn't request a group channel opening. There are just two channel types - standard and extended. Standard channels have group_id assigned at the creation. The purpose of group id is explained in the spec. Extended channels don't have any group_id assigned implicitly. And further relations to the groups are not mentioned in the spec. (In fact, I'd prefer calling it channel-groups not group channels, but whatever)

And our question is whether extended channels should be allowed in a group.
So let's go through things:

1. In case of multiple mining channels in a single connection, there are cases where some information would be sent repeatedly. To every channel separately. Like prev hash info. new target, mining job (sometimes)…
2. So in order to improve that, a (real) channel, besides its main id, also has a group_id - which makes it member of group channel (or channel group, as I'd prefer). Group is meant as a mechanism for multicast. Channels with id and group_id receive and interpret messages routed to both ids. The namespace of groups ids and ids is shared and there is no ambiguity. And that's pretty much it - channel has assigned one or two ids and receives message routed to those ids.

so unless I'm missing something I'm fine with using groups also for extended channels. I don't see reason for being overly restrictive in the spec. Every channel may have two ids - one for unicast, other for multicast (group). For standard channels it's implicit. For extended it's optional.

Then there is a completely different topic of whether it makes sense to use groups to distribute various things. For example using groups to distribute extended jobs to extended channels is a mistake. But using groups to distribute prev hash is fine IMO.

[plebhash]

using groups to distribute extended jobs to extended channels is a mistake

why?

the ability to broadcast one NewExtendedMiningJob to multiple extended channels is one of the main goals of this proposal

[jakubtrnka]

You are right it's not, sry

[plebhash]

I think your hesitancy has some merit, though:

one important implication here would be that all Extended Channels that belong to the same Group Channel would need to have:

• extranonce_prefix of identical sizes AND identical values of extranonce_len
  OR
• full_extranonce = extranonce_prefix + extranonce that always has the exact same size

that's because the coinbase has a scriptSig length field, which is directly influenced by the size of full_extranonce

when it comes to Extended Jobs, this field is serialized as part of NewExtendedMiningJob.coinbase_tx_prefix

imagine we try to broadcast the same NewExtendedMiningJob to an Extended Channel where full_extranonce has size A and another Extended Channel where full_extranonce has size B

for at least one of them, the resulting scriptSig would end up violating what was serialized as scriptSig length inside NewExtendedMiningJob.coinbase_tx_prefix

[plebhash]

here's an analysis of different messages, and whether they could be addressed to Group Channel Ids or not:

Message	Can be addressed to Group Channel Id?	Comment
NewExtendedMiningJob	YES	Allows broadcasting jobs to multiple channels with a single message.
NewMiningJob	NO	Fixed merkle root only allows message to be addressed to some specific Standard Channel.
UpdateChannel	NO	Hashrates are set per channel, individually.
SetTarget	YES	Allows setting the same target to multiple channels with a single message.
CloseChannel	YES	Allows closing multiple channels with a single message.
SetExtranoncePrefix	NO	Group Channels do not have Extranonce Prefix.
SubmitSharesStandard	NO	A standard share is associated with a Standard Channel's unique Merkle Root, and that information is lost if the share is addressed to a Group Channel Id.
SubmitSharesExtended	NO	An extended share is associated with an Extended Channel's unique Extranonce Prefix, and that information is lost if the share is addressed to a Group Channel Id.
SetNewPrevHash	YES*	Allows broadcasting a new PrevHash to multiple channels with a single message. *The job_id must be some future job that was broadcast to the same Group Channel Id.
SetCustomMiningJob	YES*	*Custom Jobs are set on Extended Channels, so if the Group Channel contains Standard Channels, those would be ignored.

[GitGab19]

Maybe the SetTarget could be also sent from a server through a Group Channel?

In the case the server wants to set the same target to all of its Extended/Standard Channels?

[plebhash]

I think that could give rise to many potentially tricky edge cases, but I guess it's possible. Adapted the table.