Deal / Data Transfer Rate-limiting

[jennijuju]

As described in #7084 (comment), as the client demands in the Filecoin network, a better deal throttling in lotus is required so that storage provider system doesn't get overwhelmed by excessive deals and impact their service quality. One of the top requests for improvement here is to allow storage providers to config data transfer rate-limiting easily based on their hardware resources, setup and service operations. Lotus-miner currently has a SimultaneousTransfers that applies to the incoming retrieval deals data transfers. That being said, such a feature is missing for controlling incoming storage deal data transfers(#7030).

There are multiple approaches that can be done here, and this discussion is created to ask our storage providers about which solution would be the most ideal to support your service.

Option 1: N for SimultaneousTransfers means exactly N in progress Storage or Retrieval deals combined at once for a miner (where sending or receiving data)
Option 2: N for SimultaneousTransfers means exactly N in progress storage deals and N in progress Retrieval deals at once for a miner
Option 3: N for SimultaneousTransfers means exactly N/2 in progress Storage or N/2 Retrieval deals at once for a miner (where sending or receiving data)
Option 4: N in progress simultaneous Storage Deals and M in progress simultaneous Retrieval Deals separately.

Please let us know what's your preferred approach in a comment below in a form of:
Preferred Option:
Miner Setup(describe your system architecture, including subsystem setup and worker setup):
Reason:
Other consideration(special consideration you think worth calling out):

[jsign]

I'm not a miner but my 2 cents.

As to share my experience with SimultaneousTransfers from the client-side, it has a drawback that can cause problems. And that's slow uploads/downloads. Slow progress on transfers eats 1 slot of that N quota.

You can be trapped in situations where the other side is too slow and you're not maximizing your throughput. I've experienced this as a client before, so is a tricky knob in the configuration file.

You can try patching the situation with aggressive thresholds to drop transfers or similar or considering other dimensions of bytes/second limits in balanced ways.

[kernelogic]

I think we need separate configs for incoming and outgoing. Also this is more like a quantity control, not speed control.

[rjan90]

Some additional thoughts from a storage-providers perspective about rate limiting:

Most storage providers knows their theoretical maximum sealing capacity per 24 hours. When the storage provider is only sealing CC-sectors it's possible to go at max capacity, since you can time when you pledge a CC-sector to exactly fit your sealing cluster (in terms of how many PC1, PC2 & C2 your cluster can handle).

Going at the max sealing capacity with only storage deals introduces a couple of hurdles currently for a storage provider:

• If you have 64GiB of storage deals that are being published in a message, those will start AddPiece consecutively, and thus start two PC1 almost at the same time, which potentially can bottleneck your PC2, C2 later (especially if we are talking about 640+ GiB of deals in a publish-message). An ability to "release a staged deal" at a certain time interval could be helpful to optimize this.

• Getting XX-TiB of data-transfers might be higher then the storage providers theoretical maximum sealing capacity per 24 hours, and disk-space, which can lead to deal errors later if the startEpoch is not set long enough on the deals.

[dd45e640b42e6da7da96faee3996ef7c]

option 4.

more important in my eyes is to decouple data transfer from data processing. enable miners to handle 2 possible bottle necks independent from each other

[jennijuju]

What do you mean by data processing?

[dd45e640b42e6da7da96faee3996ef7c]

What do you mean by data processing?

sealing. if you could transfer (not sure if it does AP on the fly already or not...) and let the deals enter a state DealPostTransfer and then be able to run a lotus-miner sectors pledge like command - eg. lotus-miner sectors seal-deal XYZ - we would have transfer and sealing decoupled and gain a huge amount of flexibility into the whole online deal process

[dd45e640b42e6da7da96faee3996ef7c]

cross linking stuff

#3482

[William8Work]

Option 4.

[stuberman]

I would like to see data transfer as quickly as possible, however, I would like to see the following:
A. Per client simultaneous transfers (limit to one per client), because some lotus clients are very slow and they can fill all of the market data transfers and block the faster clients. This protects the market node from traffic congestion.
B. Aggregate max data transfers in GiB/day for all clients. This protects our sealing pipeline from running over capacity.
C. Prioritized clients, ability to reserve data transfer quotas to specific clients. (This ensures the market node stays busy but ensures client commitments may be made.)
D. Prioritized data types, ability to prioritize verified deals over unverified deals. This ensures that the Filecoin network lives up to its mission of storing important data.

[jennijuju]

Aggregate max data transfers in GiB/day for all clients. This protects out

which option proposed above do your prefer for this and why?

[stuberman]

I think we need all 4 (A-D) of my options.
For B. (Aggregate max data transfers in GiB/day for all clients. This protects our sealing pipeline from running over capacity.) None of the original options (1-4) addresses this important need as this has nothing to do with SimultaneousTransfers but rather with total GB of all data transferred during a 24 hour period.

I strongly believe that the SimultaneousTransfers need to be based on limiting each client to one SimultaneousTransfer.
I would also only allow ONE SimultaneousTransfer for all retrieval deals until we have more experience.

[funktafide]

I have a small setup, but I would go with Option 4. I also agree with what @stuberman has written above.

[TippyFlitsUK]

I would go for option 4 in terms of managing data transfers. but I also strongly agree with the comments above. We need to be able to limit deals by max data transfers in GiB/day.

[SBudo]

Option 4
Data transfer for deals relies mainly on bandwidth capacity for us, and since download and upload are different things, the throttling should be separate as well.

A few additional ideas to make online deals better:

• Stop accepting deals when a FIL threshold has been reached on the worker address (doing this via script at the moment, but would be great to have this integrated in Lotus)
• As Patrick mentioned, creating a state between publishing and AP would be great as it would ensure the sealing workers don't get overrun. It would also allow to "park" deals in case of problems with the workers. etc.
• Add the "total amount of data to be transferred" on the "lotus-miner data-transfers list" output.
• Right now, the publishing of deals is only dependent on the number of deals in the queue and a time threshold. It would be great to have a base fee threshold with this too (again, I do this via script today, but having it as part of Lotus would be better)
• Adding the ability to decide how much to add to the market balance when "adding funding" kicks off
• As Sturbeman mentioned, prioritizing data transfer and deal acceptance based on client address and deal types
• On the "lotus-miner storage-deals list" output, add which sectors the deal has been stored in. There is a disconnect between sector numbers (referred in publishing messages for example) and deal ID, making it harder to report on deals profitability.

Our miner setup:
11 servers as follow:

• 1x Miner storage (GPU)
• 1x Miner deals
• 3x PC1 workers
• 4x PC2/C1/C2 workers (GPU)
• 1x Lotus node
• 1x Tools and monitoring servers (scripts run, farcaster node, bidbot node, etc.)

[dd45e640b42e6da7da96faee3996ef7c]

* As Patrick mentioned, creating a state between publishing and AP would be great as it would ensure the sealing workers don't get overrun. It would also allow to "park" deals in case of problems with the workers. etc.

it would allow to sort the sealing too! deal 2 before deal 1 etc.

[nonsense]

Having read this discussion and went over it with the Ignite team, we are exploring various options on how to improve:

1. Storage deal making, and not have it overwhelm the sealing pipeline, i.e. getting deals that are over the storage providers' maximum sealing capacity, and disk space - until we come up with a full-fledged solution, we are going to investigate stopgaps, such as adding a configuration option to the markets service to reject new deals when there are too many staged deals. Issue for this is at: Throttle staging deals on the lotus-miner markets service and reject new deals, until staged deals are added to sectors. #7241
2. Data transfer rate-limiting - we are aware of the problem where a few slow clients can take over all the available slots for data transfers and this is not ideal. We are unsure about adding rate-limiting functionality to Lotus itself - it seems like this can be handled outside of Lotus too, for example marking traffic with a given pid owner, and then using tc to shape the traffic.

[nonsense]

There is a point at which the markets service should reject deals - if my client tried to make 1PiB deals with you, given a sealing capacity of 1TiB per day, the markets service should come back to the client and say "i am at capacity, try again later".

[dd45e640b42e6da7da96faee3996ef7c]

but we are talking about resources here that can be used in parallel. i can transfer and seal at the same time. just because i am at capacity with sealing doesn't mean i cannot transfer.

these are basically 2 steps of a process that do not need to be connected. i can transfer without immediately start sealing. thats all i am talking about. whats the reasoning behind that? it has so many disadvantages!

[dd45e640b42e6da7da96faee3996ef7c]

just because i am at capacity with sealing for 4 days shouldn't mean i cannot accept and transfer deals to seal on day 5! a miner could stack deals for years to come and transfer them in without sealing them for month....years...whatevs if the start dates are right

[dd45e640b42e6da7da96faee3996ef7c]

whats the advantage of coupling the transfer and the sealing process into a unit that cannot be stopped/paused?

[dd45e640b42e6da7da96faee3996ef7c]

divide and conquer two problems that to not need to be connected. how often a transfer succeeds and then it fails because it starts sealing right away? start epoch stuff. it's all around. data cap stuff is a curse right now...

the transfer itself has enough problems - connecting them to the sealing problems is nothing we do out of need, we do it because of - well - who knows why?

[tmyuu]

I don't know how often to be used in storing and retrieving data in the future, but I prefer option 4 for flexibility.

I think controlling the number of simultaneous transfers means that when N occupy a limited bandwidth of the storage provider, the average can be set according to a fast client. From a storage provider's perspective, a good client will be able to transfer or get data smoothly without having to wait.
However, if clients have waited in chronological order at this time, the slow client might gradually monopolize N, and they make the client with the potential for high-speed transfer wait longer than necessary.
Therefore lotus needs proper logic, such as establishing a connection with each other, then doing a simple speed test and ranking the clients. (Setting the number of clients M that do not meet the criteria.)

About the relationship between sealing and data transfer that some have mentioned:
Estuary, for example, currently specifies a deal start epoch after 7 days, so I think daily transfer size is not much of a problem for storage providers even though a daily sealing capability is low.
They can already select client with Expected SealDuration, and if over capacity of the daily sealing limit we would just reject the online deal.
It's not related to the topic here, but it would be nice to be able to specify WaitDeals? even if one deal is a data size that just fits the sector.