Chain: Underlying Store fails on PUT - what to do?

[VojtechVitek]

Let's say I have 3 stores: Memstore, BoltDB (async) and s3 (async).

given chain.Put(ctx, key, data)

• Memstore succeeds
• BoltDB fails
• s3 succeeds

What should we do? Should we automatically re-try storing the value into the BoltDB again? What if the DB is down for a bit? Should we have a retry queue per Store?

Or since we have LRU, should we have something like PUT down ON DELETE functionality on chain?

[VojtechVitek]

Idea:

err, errCh := chain.Put(ctx, key, value)
if err != nil {
    // handle errors coming from blocking (sync) stores
}
go func() {
      for _, err := range errCh {
            // handle errors coming from async stores
            err.(StoreErr).Retry() // ?
      }
}