Fast retrieval

[arajasek]

Cross-component task: Miners can optionally store an extra, unsealed, copy of the data for fast retrieval. Needs work on Lotus, markets, and storage miner.

• Markets Sub Issue: StorageClient can request a fast retrieval go-fil-markets#285

[hannahhoward]

Markets Sub Issue: filecoin-project/go-fil-markets#285

Maybe that goes in a todo list for this ticket?

[hannahhoward]

Proposed Intra-component design

Initiating Fast Retrieval In Lotus Software

When proposing a storage deal, the lotus client deal can optionally accept a flag specifying fast retrieval

The default value for fast retrieval is TBD
Could be:

• if the client is a verified client, the default is yes, if not no
• always yes
• always no

Markets accepts fast retrieval parameter

ProposeStorageDeal on the StorageClient interface will be augmented to take a fast retrieval parameter. This value will be transmitted to the miner when the deal is proposed. The value will be stored in the deal state inside markets but live off chain

Accepting a deal with fast retrieval

By default, whether or not a deal requests fast retrieval will not effect whether it's accepted in Markets code.

However, this can be changed through custom deal decision logic

Handoff

When provider publishes a deal, it calls OnDealComplete on the StorageProviderNode interface to handoff the deal to but put in a sector. We will add the fast retrieval parameter to this method. The markets adapter will pass the fast retrieval attribute to SectorBlocks.AddPiece.

If the attribute is set, SectorBlocks will not delete the unsealed sector once a sealed sector is written.

Retrieval

Fast retrieval is transparent from the point of view of the markets code and command line. The only change is when SectorManager.ReadPiece is called, it will run quickly assuming the unsealed sector is still there.

cc: @magik6k @arajasek @momack2

[arajasek]

I'd think the default value should be "no", but I'm curious to hear other people's thoughts. Regardless, I think this is one of those values that every client has to think about, so the default isn't as important (if you're happy to just go with "whatever it does automatically" for whether you ask miners to store a bunch of extra data, you're doing something wrong).

[momack2]

I think the default should be yes (at least for verified clients) because I think the vast majority of web3 clients will want fast retrieval, and many non-web3 users will benefit from this as well. Clients that know they don't want fast retrieval can easily change this flag, but I think users will have a better experience if the default is set with that flow in mind, and then more advanced/specialized users can get price improvements/optimizations by configuring these values.

My order of preference for default values would be:

• Always yes: represents that most users want this by default, but requires configuration for users who don't care about fast retrieval to open up all storage options (ex if miners accept fewer deals with this param set from non-VCs)
• Always yes for verified clients, no otherwise: requires configuration for users who end up as non-VCs by accident but want fast retrieval, but might make testing easier for clients intentionally in this state that don't care about fast retrieval
• Always no: requires configuration for expected use case, doesn't set users up for successful deal-making flow

(@pooja for the product side perspective as well)

[magik6k]

Finalized in #2323