Replies: 6 comments 26 replies
-
Non-breaking changesHere, by non-breaking, I mean that we're not adding or removing fields from any message. It's merely changing conceptual definitions. Implementations that pre-date to this change would remain spec-compliant, only missing new functionality. This is akin to a soft-fork in bitcoin consensus. Definition of Group ChannelThis would change how the definition of how Group Channel is formally established in the spec. ### 5.2.3 Group Channel
-Standard channels opened within one particular connection can be grouped together to be addressable by a common communication group channel.
+Standard and Extended channels opened within one particular connection can be grouped together to be addressable by a common communication group channel.
-Whenever a Standard Channel is created, it is always put into some Group Channel identified by its `group_channel_id`.
-Group Channel ID namespace is the same as Standard Channel ID namespace on a particular connection.
+Whenever a Standard or Extended Channel is created, it is always put into some Group Channel identified by its `group_channel_id`.
+Group Channel ID namespace is the same as Standard and Extended Channel ID namespace on a particular connection.Definition of Extended JobThis would rephrase how we're defining Extended Job with regards to Group Channel. ### 5.3.16 `NewExtendedMiningJob` (Server -> Client)
(Extended and group channels only)
For an **extended channel**:
The whole search space of the job is owned by the specified channel.
If the `min_ntime` field is set to some nTime, the client MUST start to mine on the new job as soon as possible after receiving this message.
For a **group channel**:
-This is a broadcast variant of `NewMiningJob` message with the `merkle_root` field replaced by `merkle_path` and coinbase transaction prefix and suffix, for further traffic optimization.
-The Merkle root is then defined deterministically for each channel by the common `merkle_path` and unique `extranonce_prefix` serialized into the coinbase.
-The full coinbase is then constructed as follows: `coinbase_tx_prefix + extranonce_prefix + coinbase_tx_suffix`.
+This acts as a broadcast message that distributes work to all channels under the same group.
The proxy MAY transform this multicast variant for downstream standard channels into `NewMiningJob` messages by computing the derived Merkle root for them.
-A proxy MUST translate the message for all downstream standard channels belonging to the group which don’t signal that they accept extended mining jobs in the `SetupConnection` message (intended and expected behavior for end mining devices).
+A proxy MUST translate the message into `NewMiningJob` for all downstream standard channels belonging to the group in case the `SetupConnection` message had the `REQUIRES_STANDARD_JOB` flag set (intended and expected behavior for end mining devices).
+
+For conversion into different `NewMiningJob` messages, the Merkle root is then defined deterministically for each standard channel by the common `merkle_path` and unique `extranonce_prefix` serialized into the coinbase.
+The full coinbase is then constructed as follows: `coinbase_tx_prefix + extranonce_prefix + coinbase_tx_suffix`.
| Field Name | Data Type | Description
|
| ----------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-| channel_id | U32 | For a group channel, the message is broadcasted to all standard channels belonging to the group. Otherwise, it is addressed to the specified extended channel.
|
+| channel_id | U32 | For a group channel, the message is broadcasted to all standard and extended channels belonging to the group. Otherwise, it is addressed to the specified extended channel. Definition of
|
Beta Was this translation helpful? Give feedback.
-
|
I don't have the time now to think at how a pool that implement the proposed changes interact with a miner that do not and vice versa. Can you provide some analysis on that so that we can see that the proposed changes are actually non breaking? Also I think that if we go with this proposal, we need to say that SetCustomMJob can belong to group as well. |
Beta Was this translation helpful? Give feedback.
-
|
The scenarios below assume we did not modify So we would establish some kind of convention that reserves a special channel id (e.g.: Now let's establish the different scenarios: Scenario A
Pool would send one Proxy would not work as expected. So for this scenario, yes, this would be a breaking change. However, to our knowledge, there is no non-aggregating proxy implemented so far. Scenario B
Pool would simply send Scenario C
JDC would send one Scenario D
JDC would send one |
Beta Was this translation helpful? Give feedback.
-
Breaking changesThis is an optional part of the proposal. It could face a bit more resistance, since it would break existing implementations. Right now, If we are allowing Extended Channels to belong to a Group Channel, ideally ### 5.3.5 `OpenExtendedMiningChannel.Success` (Server -> Client)
Sent as a response for opening an extended channel.
| Field Name | Data Type | Description |
| ----------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| request_id | U32 | Client-specified request ID from OpenExtendedMiningChannel message, so that the client can pair responses with open channel requests |
| channel_id | U32 | Newly assigned identifier of the channel, stable for the whole lifetime of the connection, e.g. it is used for broadcasting new jobs by NewExtendedMiningJob |
| target | U256 | Initial target for the mining channel |
| extranonce_size | U16 | Extranonce size (in bytes) set for the channel |
| extranonce_prefix | B0_32 | Bytes used as implicit first part of extranonce |
+| group_channel_id | U32 | Group channel into which the new channel belongs. See SetGroupChannel for details. |Otherwise, it's hard to say which Group Channel some specific Extended Channel belongs to once it's open, and could lead to undefined states. If this breaking change is rejected, but we still move forward with the non-breaking proposal, then Proxies could still assume some value for |
Beta Was this translation helpful? Give feedback.
-
Can you please specify to what data or metrics are you referring here? A general note here is that any data that cannot be collected by the pool, can always be collected by the proxy. If the pool does not need this data to operate, I don't think it should know about. |
Beta Was this translation helpful? Give feedback.
-
|
btw as I already said in per discussion about monitoring extension. We should aggregate information about every worker at proxy level and sending upstream the aggreagated infos. For example {difficulty, number_of_shares_accepted, number_of_share_rejected} you can do that every n minutes. And still send info about last hashratre per worker every 10 sec if you want to be more responsive. You want also to send info about temperature voltage ecc ecc, so the extension should account also for that info. With such extension you don't need to change the protocol, you will get a lot less traffic to the pool, you will get much more infos. (For example you can send errors for share rejection and things like that) |
Beta Was this translation helpful? Give feedback.
-
|
The proposal isnt about miscellaneous data, it's about the pool validating shares of each individual miner so the miners can compare the pool's payout data and the miner's share data to ensure pools are paying out correctly. Just as a GroupChannel does |
Beta Was this translation helpful? Give feedback.
-
|
still not getting what this proposal achieve that the extension can not achieve |
Beta Was this translation helpful? Give feedback.
-
for doing that you really don't need to send EVERY share to the pool. You can achieve it without extension and without what proposed here. You just need a proxy that aggregate shares and you verify that shares sent up by the proxy are payed. You really do not need anything else. Ofc this is enough fro FPPS if you do PPLNS yuo need to trust the pool to not inflate pool's hashrate (and this proposal certainly do not solve this issue) |
Beta Was this translation helpful? Give feedback.
-
|
and ofc if you really want to send EVERY share upstream for whatever reason you can do with the extension so what I'm saying is:
still concept nack |
Beta Was this translation helpful? Give feedback.
-
|
Considering the original specification specifically includes GroupChannels, and the proposal is to allow for extended group channels, i think all pros and cons of GroupChannels have to be applied to the proposed extended group channel. Considering the pros outweighed the cons in the original spec definition, I think it's fair to keep those comparisons in mind when evaluating the proposal. In addition, given the benchmarks here stratum-mining/stratum#1841, it's likely extended channels are more performant in most cases, especially on a local network with sufficient bandwidth. Forcing the proxy (TProxy or Sv2Proxy) to calculate merkle roots sequentially is much less efficient than letting individual devices calculate their own merkle roots in parallel and both ExtendedJobs and StandardJobs both easily fall within a standard Tcp Packet, so there should be little to no networking hit. Sure there is more data to copy over Tcp but copying a buffer is much more efficient than caculating a merkle root. Given these points, i believe it's in Firmwares best interest to support connecting with extended jobs directly, and therefore this proposal may not ONLY affect the TProxy Cons for ExtendedGroupChannels:
Pros for ExtendedGroupChannels:
|
Beta Was this translation helpful? Give feedback.
-
|
you don't need extended group channel for that. A miner will open an extended channel with (or standard non HOM) with proxy. So the proxy will have multiple extended channel one for each miner, and one extended channel opened with the pool, exactly like in sv1. |
Beta Was this translation helpful? Give feedback.
-
|
I think this disregards the use of GroupChannels. Why not just have an Sv2 proxy connect to the pool on an extended channel and let StandardChannels connect downstream, while aggregating shares? Then GroupChannels aren't even necessary |
Beta Was this translation helpful? Give feedback.
-
|
framing @colerob1776 's argument above from a slightly different perspective: saying "just aggregate" misses the point, because as far as I can infer on the protocol design, Group Channels are an optimization meant for non-aggregating proxies if the protocol had been designed such that every proxy was forced to aggregate, the notion of Group Channels wouldn't be a part of it the issue at hand here is that Extended Channels is left out of the picture, while there's actually no fundamental issue preventing them from belonging to Group Channels as well (aside from the fact that, well, the spec says so) overall, with one small exception ( |
Beta Was this translation helpful? Give feedback.
-
|
I've done some digging on this topic. One thing we often face at SRI when engaging with early adopters is that most of them always asks if there's a way to get individual workers share data. Originally, I really didn't understand the reasons for this, and like @Fi3 I was against adding this into specs/protocol for this, because why would we settle for anything that is not the most efficient outcome. However, over time, it became obvious that there are users who want to have granularity in shares. I will talk about WHY users want this, a bit later, but this lead us to develop two modes in tProxy:
Now the users who want to have this data for individual level at the Pool, can have it. However this option is pretty inefficient. My understanding of this spec change is that it would alloExw non-aggregating proxy users to have a more efficient setup by allowing them to not aggregate shares, but do aggregate job dispatched which is a significant efficiency improvement. Now, I get that it's hard to reason with this from technical perspective, but interaction with users actually shows that a very large portion of users doesn't want to aggregate shares, but still would like to have efficiency boost, which I think makes sense to cater for this use-case, especially since Braiins already uses Group Extended Channels. There are two compelling reasons on why miners and Pools want to have individual granularity of the worker share data on the Pool level.
One thing that has always concerned me about the plugin route, especially this early in SV2 adoption phase is compatibility. How will things work if one pool supports one, but the fallback happens and the other doesn't? Then your setup becomes quite fragile. Imagine your auditing breaks ,because fallback happened. We can of course wait to find out, but should we really push users into using experimental extension system, especially if majority that we've interacted with expressed the necessity for it, and they would be better supported if it worked right out of the box? My point here, is that upon having quite a few interactions with miners, mining pools and some of the early adopters, I think this spec change would allow us to move forward with a behaviour that I would say over 70% of adopters asked for in the most efficient way possible, and it seems we can move into a non-breaking manner. I think extensions should be used for features that are opinionated and only a handful users want, but if a feature is something that majority of current and future adopters want, I think it's sensible to consider making a spec change. |
Beta Was this translation helpful? Give feedback.
-
This extension is just for monitoring. There is no proof behind the hashrate. If the pool does not know about the assigned difficulty of each individual device, any hashrate related data from the client is tentative at best, so in my opinion this is a fair fit for an extension only.
How does this proposal not support PPLNS? This proposal simply allows all necessary components in the system to have visibility into the work and submissions that a miner gets paid out for. PPLNS just relies on a sliding window as opposed to all shares with FPPS, so as long as device shares are connected into a timeseries DB (most likely by some client side agent) it's still possible to reconcile given the pools window and blocks mined. It's still odd to me how GroupChannels received a spot in the protocol, but now when we realize ExtendedGroupChannels were left out, grouping doesnt make sense. Like I mentioned previously, you could just mine StandardChannel -> Proxy -> ExtendedChannel and aggregate hashrate with Sv2Firmware devices without GroupChannels, so any efficiency argument against the proposal does not resonate with me because the same argument can be made against regular GroupChannels. |
Beta Was this translation helpful? Give feedback.
-
|
I think with PPLNS and FFPS example Fi3 was just making a general observation on how extensions could be incompatible, highlighting that the incompatibility example I gave may happen in other cases as well, with different extensions. The difference here is just that this is likely to be a core feature used by many, so ensuring interoperability may be a good idea, which may be hard to to through extensions at this stage of adoptions. More extensions, narrow down on fallback pool choice, which I think we all agree isn't ideal for early stage of adoption. It seems that we're getting close to consensus that this is needed, but what remains (at least from what I am reading) is agreeing on technical implementation and how (if) it , a protocol change affects existing Pool implementation. This threading of discussions can sometimes be tricky to get freshest context, as some things mentioned by fi3 seems to be changed in more recent messages. |
Beta Was this translation helpful? Give feedback.
-
|
ahh sorry, i may have misinterpreted |
Beta Was this translation helpful? Give feedback.
-
It was more like about fallback. If a miner expect/want/need FPPS it can't fallback to a pool the is PPLNS. Same thing can be said for a lot of other little thing. Btw if we reaaly need to save every share produced by every worker, I start thinking that maybe this little patch could be the easiest and way and also very efficient. Said that (1) we need to support custom job (2) I think that eventually we want to push for a system that make sense, so having things like that deeply rooted in the protocol is not the best imo. For example if a miner want accountability what it change if it have it at proxy level or at device level? From a technical perspective absolutly nothing but the fact that one is much much more effecient than the other. And the big difference is for the pool operator here, so if we want a world where operating a pool is as easiest and cheapest as possible we should for sure push for a direction and not the other. And I think that as maintainers of SRI this should be our mission. So IMO an extension here that can be deprecated in the future is the best option. |
Beta Was this translation helpful? Give feedback.
-
|
I would rather be more happy to completly remove standard channels and group channels from the mining protocol and make it simpler lol |
Beta Was this translation helpful? Give feedback.
-
|
IMO the entire concept of Group Channels is discussed as something very sophisticated, while the idea behind it is fairly simple. first of all, it's actually not a real channel. Client doesn't request a group channel opening. There are just two channel types - standard and extended. Standard channels have group_id assigned at the creation. The purpose of group id is explained in the spec. Extended channels don't have any group_id assigned implicitly. And further relations to the groups are not mentioned in the spec. (In fact, I'd prefer calling it channel-groups not group channels, but whatever) And our question is whether extended channels should be allowed in a group.
so unless I'm missing something I'm fine with using groups also for extended channels. I don't see reason for being overly restrictive in the spec. Every channel may have two ids - one for unicast, other for multicast (group). For standard channels it's implicit. For extended it's optional. Then there is a completely different topic of whether it makes sense to use groups to distribute various things. For example using groups to distribute extended jobs to extended channels is a mistake. But using groups to distribute prev hash is fine IMO. |
Beta Was this translation helpful? Give feedback.
-
why? the ability to broadcast one |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
There's been a lot of discussions in the Sv2 community around some fundamental UX aspect of Translator Proxy.
It can work in two different modes:
Agrregated mode brings a lot of efficiency in terms of bandwidth and job broadcasting. However, it has the downside of losing data granularity, meaning that the user cannot look at the Pool dashboard and visualize metrics on a per-worker basis. This is the UX that miners are used to today, and forcing them to move away from it could become a barrier for protocol adoption.
Non-aggregated mode allows for such granularity, but loses all the efficiency optimizations of aggregation.
Sv2 has the notion of Group Channels, which are meant to make job broadcasting more efficient. This could be an interesting option to optimize non-aggregated mode.
And now the main problem:
The fact that Group Channels can only contain Standard Channels (but not Extended) is a bit counter-intuitive at first sight, because when it comes to the design space in front of us, there's no fundamental blocker preventing the protocol from allowing this.
But this restriction becomes clear when we read the following sections:
5.2.3 Group Channel5.3.16 NewExtendedMiningJob5.3.22 SetGroupChannelThis Discussion is meant to try to get consensus on changing the Mining Spec, in order to allow Group Channels to also contain Extended Channels.
This would enable a Translator Proxy that retains a legacy-friendly UX (with per-worker data granularity via non-aggregated mode) that can still receive jobs broadcasts from a Pool in an efficient way.
The proposal is split in two parts, where the second is optional and complimentary to the first.
Beta Was this translation helpful? Give feedback.
All reactions