Skip to content

Commit cec7432

Browse files
lidellidel
authored
feat: fast provide support in dag import (#11058)
* fix(add): respect Provide config in fast-provide-root fast-provide-root should honor the same config settings as the regular provide system: - skip when Provide.Enabled is false - skip when Provide.DHT.Interval is 0 - respect Provide.Strategy (all/pinned/roots/mfs/combinations) This ensures fast-provide only runs when appropriate based on user configuration and the nature of the content being added (pinned vs unpinned, added to MFS or not). * feat(config): options to adjust global defaults Add Import.FastProvideRoot and Import.FastProvideWait configuration options to control default behavior of fast-provide-root and fast-provide-wait flags in ipfs add command. Users can now set global defaults in config while maintaining per-command flag overrides. - Add Import.FastProvideRoot (default: true) - Add Import.FastProvideWait (default: false) - Add ResolveBoolFromConfig helper for config resolution - Update docs with configuration details - Add log-based tests verifying actual behavior * refactor: extract fast-provide logic into reusable functions Extract fast-provide logic from add command into reusable components: - Add config.ShouldProvideForStrategy helper for strategy matching - Add ExecuteFastProvide function reusable across add and dag import commands - Move DefaultFastProvideTimeout constant to config/provide.go - Simplify add.go from 72 lines to 6 lines for fast-provide - Move fast-provide tests to dedicated TestAddFastProvide function Benefits: - cleaner API: callers only pass content characteristics - all strategy logic centralized in one place - better separation of concerns - easier to add fast-provide to other commands in future * feat(dag): add fast-provide support for dag import Adds --fast-provide-root and --fast-provide-wait flags to `ipfs dag import`, mirroring the fast-provide functionality available in `ipfs add`. Changes: - Add --fast-provide-root and --fast-provide-wait flags to dag import command - Implement fast-provide logic for all root CIDs in imported CAR files - Works even when --pin-roots=false (strategy checked internally) - Share ExecuteFastProvide implementation between add and dag import - Move ExecuteFastProvide to cmdenv package to avoid import cycles - Add logging when fast-provide is disabled - Conditional error handling: return error when wait=true, warn when wait=false - Update config docs to mention both ipfs add and ipfs dag import - Update changelog to use "provide" terminology and include dag import examples - Add comprehensive test coverage (TestDagImportFastProvide with 6 test cases) The fast-provide feature allows immediate DHT announcement of root CIDs for faster content discovery, bypassing the regular background queue. * docs: improve fast-provide documentation Refine documentation to better explain fast-provide and sweep provider working together, and highlight the performance improvement. Changelog: - add fast-provide to sweep provider features list - explain performance improvement: root CIDs discoverable in <1s vs 30+ seconds - note this uses optimistic DHT operations (faster with sweep provider) - simplify examples, point to --help for details Config docs: - fix: --fast-provide-roots should be --fast-provide-root (singular) - clarify Import.FastProvideRoot focuses on root CIDs while sweep handles all blocks - simplify Import.FastProvideWait description Command help: - ipfs add: explain sweep provider context upfront - ipfs dag import: add fast-provide explanation section - both explain the split: fast-provide for roots, sweep for all blocks * test: add tests for ShouldProvideForStrategy add tests covering all provide strategy combinations with focus on bitflag OR logic (the else-if bug fix). organized by behavior: - all strategy always provides - single strategies match only their flag - combined strategies use OR logic - zero strategy never provides * refactor: error cmd on error and wait=true change ExecuteFastProvide() to return error, enabling proper error propagation when --fast-provide-wait=true. in sync mode, provide failures now error the command as expected. in async mode (default), always returns nil with errors logged in background goroutine. also remove duplicate ExecuteFastProvide() from provide.go (75 lines), keeping single implementation in cmdenv/env.go for reuse across add and dag import commands. call sites simplified: - add.go: check and propagate error from ExecuteFastProvide - dag/import.go: return error from ForEach callback, remove confusing conditional error handling semantics: - precondition skips (DHT unavailable, etc): return nil (not failure) - async mode (wait=false): return nil, log errors in goroutine - sync mode (wait=true): return wrapped error on provide failure
1 parent d56fe3a commit cec7432

File tree

12 files changed

+740
-96
lines changed

12 files changed

+740
-96
lines changed

‎config/import.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ const (
1616
DefaultUnixFSRawLeaves = false
1717
DefaultUnixFSChunker = "size-262144"
1818
DefaultHashFunction = "sha2-256"
19+
DefaultFastProvideRoot = true
20+
DefaultFastProvideWait = false
1921

2022
DefaultUnixFSHAMTDirectorySizeThreshold = 262144 // 256KiB - https://github.com/ipfs/boxo/blob/6c5a07602aed248acc86598f30ab61923a54a83e/ipld/unixfs/io/directory.go#L26
2123

@@ -48,6 +50,8 @@ type Import struct {
4850
UnixFSHAMTDirectorySizeThreshold OptionalBytes
4951
BatchMaxNodes OptionalInteger
5052
BatchMaxSize OptionalInteger
53+
FastProvideRoot Flag
54+
FastProvideWait Flag
5155
}
5256

5357
// ValidateImportConfig validates the Import configuration according to UnixFS spec requirements.

‎config/provide.go‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,11 @@ const (
2222
DefaultProvideDHTMaxProvideConnsPerWorker = 20
2323
DefaultProvideDHTKeystoreBatchSize = 1 << 14 // ~544 KiB per batch (1 multihash = 34 bytes)
2424
DefaultProvideDHTOfflineDelay = 2 * time.Hour
25+
26+
// DefaultFastProvideTimeout is the maximum time allowed for fast-provide operations.
27+
// Prevents hanging on network issues when providing root CID.
28+
// 10 seconds is sufficient for DHT operations with sweep provider or accelerated client.
29+
DefaultFastProvideTimeout = 10 * time.Second
2530
)
2631

2732
type ProvideStrategy int
@@ -175,3 +180,25 @@ func ValidateProvideConfig(cfg *Provide) error {
175180

176181
return nil
177182
}
183+
184+
// ShouldProvideForStrategy determines if content should be provided based on the provide strategy
185+
// and content characteristics (pinned status, root status, MFS status).
186+
func ShouldProvideForStrategy(strategy ProvideStrategy, isPinned bool, isPinnedRoot bool, isMFS bool) bool {
187+
if strategy == ProvideStrategyAll {
188+
// 'all' strategy: always provide
189+
return true
190+
}
191+
192+
// For combined strategies, check each component
193+
if strategy&ProvideStrategyPinned != 0 && isPinned {
194+
return true
195+
}
196+
if strategy&ProvideStrategyRoots != 0 && isPinnedRoot {
197+
return true
198+
}
199+
if strategy&ProvideStrategyMFS != 0 && isMFS {
200+
return true
201+
}
202+
203+
return false
204+
}

‎config/provide_test.go‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,3 +105,87 @@ func TestValidateProvideConfig_MaxWorkers(t *testing.T) {
105105
})
106106
}
107107
}
108+
109+
func TestShouldProvideForStrategy(t *testing.T) {
110+
t.Run("all strategy always provides", func(t *testing.T) {
111+
// ProvideStrategyAll should return true regardless of flags
112+
testCases := []struct{ pinned, pinnedRoot, mfs bool }{
113+
{false, false, false},
114+
{true, true, true},
115+
{true, false, false},
116+
}
117+
118+
for _, tc := range testCases {
119+
assert.True(t, ShouldProvideForStrategy(
120+
ProvideStrategyAll, tc.pinned, tc.pinnedRoot, tc.mfs))
121+
}
122+
})
123+
124+
t.Run("single strategies match only their flag", func(t *testing.T) {
125+
tests := []struct {
126+
name string
127+
strategy ProvideStrategy
128+
pinned, pinnedRoot, mfs bool
129+
want bool
130+
}{
131+
{"pinned: matches when pinned=true", ProvideStrategyPinned, true, false, false, true},
132+
{"pinned: ignores other flags", ProvideStrategyPinned, false, true, true, false},
133+
134+
{"roots: matches when pinnedRoot=true", ProvideStrategyRoots, false, true, false, true},
135+
{"roots: ignores other flags", ProvideStrategyRoots, true, false, true, false},
136+
137+
{"mfs: matches when mfs=true", ProvideStrategyMFS, false, false, true, true},
138+
{"mfs: ignores other flags", ProvideStrategyMFS, true, true, false, false},
139+
}
140+
141+
for _, tt := range tests {
142+
t.Run(tt.name, func(t *testing.T) {
143+
got := ShouldProvideForStrategy(tt.strategy, tt.pinned, tt.pinnedRoot, tt.mfs)
144+
assert.Equal(t, tt.want, got)
145+
})
146+
}
147+
})
148+
149+
t.Run("combined strategies use OR logic (else-if bug fix)", func(t *testing.T) {
150+
// CRITICAL: Tests the fix where bitflag combinations (pinned+mfs) didn't work
151+
// because of else-if instead of separate if statements
152+
tests := []struct {
153+
name string
154+
strategy ProvideStrategy
155+
pinned, pinnedRoot, mfs bool
156+
want bool
157+
}{
158+
// pinned|mfs: provide if EITHER matches
159+
{"pinned|mfs when pinned", ProvideStrategyPinned | ProvideStrategyMFS, true, false, false, true},
160+
{"pinned|mfs when mfs", ProvideStrategyPinned | ProvideStrategyMFS, false, false, true, true},
161+
{"pinned|mfs when both", ProvideStrategyPinned | ProvideStrategyMFS, true, false, true, true},
162+
{"pinned|mfs when neither", ProvideStrategyPinned | ProvideStrategyMFS, false, false, false, false},
163+
164+
// roots|mfs
165+
{"roots|mfs when root", ProvideStrategyRoots | ProvideStrategyMFS, false, true, false, true},
166+
{"roots|mfs when mfs", ProvideStrategyRoots | ProvideStrategyMFS, false, false, true, true},
167+
{"roots|mfs when neither", ProvideStrategyRoots | ProvideStrategyMFS, false, false, false, false},
168+
169+
// pinned|roots
170+
{"pinned|roots when pinned", ProvideStrategyPinned | ProvideStrategyRoots, true, false, false, true},
171+
{"pinned|roots when root", ProvideStrategyPinned | ProvideStrategyRoots, false, true, false, true},
172+
{"pinned|roots when neither", ProvideStrategyPinned | ProvideStrategyRoots, false, false, false, false},
173+
174+
// triple combination
175+
{"all-three when any matches", ProvideStrategyPinned | ProvideStrategyRoots | ProvideStrategyMFS, false, false, true, true},
176+
{"all-three when none match", ProvideStrategyPinned | ProvideStrategyRoots | ProvideStrategyMFS, false, false, false, false},
177+
}
178+
179+
for _, tt := range tests {
180+
t.Run(tt.name, func(t *testing.T) {
181+
got := ShouldProvideForStrategy(tt.strategy, tt.pinned, tt.pinnedRoot, tt.mfs)
182+
assert.Equal(t, tt.want, got)
183+
})
184+
}
185+
})
186+
187+
t.Run("zero strategy never provides", func(t *testing.T) {
188+
assert.False(t, ShouldProvideForStrategy(ProvideStrategy(0), false, false, false))
189+
assert.False(t, ShouldProvideForStrategy(ProvideStrategy(0), true, true, true))
190+
})
191+
}

‎config/types.go‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,16 @@ func (f Flag) String() string {
117117
}
118118
}
119119

120+
// ResolveBoolFromConfig returns the resolved boolean value based on:
121+
// - If userSet is true, returns userValue (user explicitly set the flag)
122+
// - Otherwise, uses configFlag.WithDefault(defaultValue) (respects config or falls back to default)
123+
func ResolveBoolFromConfig(userValue bool, userSet bool, configFlag Flag, defaultValue bool) bool {
124+
if userSet {
125+
return userValue
126+
}
127+
return configFlag.WithDefault(defaultValue)
128+
}
129+
120130
var (
121131
_ json.Unmarshaler = (*Flag)(nil)
122132
_ json.Marshaler = (*Flag)(nil)

‎core/commands/add.go‎

Lines changed: 27 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,13 @@
11
package commands
22

33
import (
4-
"context"
54
"errors"
65
"fmt"
76
"io"
87
"os"
98
gopath "path"
109
"strconv"
1110
"strings"
12-
"time"
1311

1412
"github.com/ipfs/kubo/config"
1513
"github.com/ipfs/kubo/core/commands/cmdenv"
@@ -74,11 +72,6 @@ const (
7472

7573
const (
7674
adderOutChanSize = 8
77-
78-
// fastProvideTimeout is the maximum time allowed for async fast-provide operations.
79-
// Prevents hanging on network issues when providing root CID in background.
80-
// 10 seconds is sufficient for DHT operations with sweep provider or accelerated client.
81-
fastProvideTimeout = 10 * time.Second
8275
)
8376

8477
var AddCmd = &cmds.Command{
@@ -89,21 +82,21 @@ Adds the content of <path> to IPFS. Use -r to add directories (recursively).
8982
9083
FAST PROVIDE OPTIMIZATION:
9184
92-
When you add content to IPFS, it gets queued for announcement on the DHT.
93-
The background queue can take some time to process, meaning other peers
94-
won't find your content immediately after 'ipfs add' completes.
85+
When you add content to IPFS, the sweep provider queues it for efficient
86+
DHT provides over time. While this is resource-efficient, other peers won't
87+
find your content immediately after 'ipfs add' completes.
9588
96-
To make sharing faster, 'ipfs add' does an extra immediate announcement
97-
of just the root CID to the DHT. This lets other peers start discovering
98-
your content right away, while the regular background queue still handles
99-
announcing all the blocks later.
89+
To make sharing faster, 'ipfs add' does an immediate provide of the root CID
90+
to the DHT in addition to the regular queue. This complements the sweep provider:
91+
fast-provide handles the urgent case (root CIDs that users share and reference),
92+
while the sweep provider efficiently provides all blocks according to
93+
Provide.Strategy over time.
10094
101-
By default, this extra announcement runs in the background without slowing
102-
down the command. If you need to be certain the root CID is discoverable
103-
before the command returns (for example, sharing a link immediately),
104-
use --fast-provide-wait to wait for the announcement to complete.
105-
Use --fast-provide-root=false to skip this optimization and rely only on
106-
the background queue (controlled by Provide.Strategy and Provide.DHT.Interval).
95+
By default, this immediate provide runs in the background without blocking
96+
the command. If you need certainty that the root CID is discoverable before
97+
the command returns (e.g., sharing a link immediately), use --fast-provide-wait
98+
to wait for the provide to complete. Use --fast-provide-root=false to skip
99+
this optimization.
107100
108101
This works best with the sweep provider and accelerated DHT client.
109102
Automatically skipped when DHT is not available.
@@ -245,8 +238,8 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import
245238
cmds.UintOption(modeOptionName, "Custom POSIX file mode to store in created UnixFS entries. WARNING: experimental, forces dag-pb for root block, disables raw-leaves"),
246239
cmds.Int64Option(mtimeOptionName, "Custom POSIX modification time to store in created UnixFS entries (seconds before or after the Unix Epoch). WARNING: experimental, forces dag-pb for root block, disables raw-leaves"),
247240
cmds.UintOption(mtimeNsecsOptionName, "Custom POSIX modification time (optional time fraction in nanoseconds)"),
248-
cmds.BoolOption(fastProvideRootOptionName, "Immediately provide root CID to DHT for fast content discovery. When disabled, root CID is queued for background providing instead.").WithDefault(true),
249-
cmds.BoolOption(fastProvideWaitOptionName, "Wait for fast-provide-root to complete before returning. Ensures root CID is discoverable when command finishes.").WithDefault(false),
241+
cmds.BoolOption(fastProvideRootOptionName, "Immediately provide root CID to DHT in addition to regular queue, for faster discovery. Default: Import.FastProvideRoot"),
242+
cmds.BoolOption(fastProvideWaitOptionName, "Block until the immediate provide completes before returning. Default: Import.FastProvideWait"),
250243
},
251244
PreRun: func(req *cmds.Request, env cmds.Environment) error {
252245
quiet, _ := req.Options[quietOptionName].(bool)
@@ -317,8 +310,8 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import
317310
mode, _ := req.Options[modeOptionName].(uint)
318311
mtime, _ := req.Options[mtimeOptionName].(int64)
319312
mtimeNsecs, _ := req.Options[mtimeNsecsOptionName].(uint)
320-
fastProvideRoot, _ := req.Options[fastProvideRootOptionName].(bool)
321-
fastProvideWait, _ := req.Options[fastProvideWaitOptionName].(bool)
313+
fastProvideRoot, fastProvideRootSet := req.Options[fastProvideRootOptionName].(bool)
314+
fastProvideWait, fastProvideWaitSet := req.Options[fastProvideWaitOptionName].(bool)
322315

323316
if chunker == "" {
324317
chunker = cfg.Import.UnixFSChunker.WithDefault(config.DefaultUnixFSChunker)
@@ -355,6 +348,9 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import
355348
maxHAMTFanout = int(cfg.Import.UnixFSHAMTDirectoryMaxFanout.WithDefault(config.DefaultUnixFSHAMTDirectoryMaxFanout))
356349
}
357350

351+
fastProvideRoot = config.ResolveBoolFromConfig(fastProvideRoot, fastProvideRootSet, cfg.Import.FastProvideRoot, config.DefaultFastProvideRoot)
352+
fastProvideWait = config.ResolveBoolFromConfig(fastProvideWait, fastProvideWaitSet, cfg.Import.FastProvideWait, config.DefaultFastProvideWait)
353+
358354
// Storing optional mode or mtime (UnixFS 1.5) requires root block
359355
// to always be 'dag-pb' and not 'raw'. Below adjusts raw-leaves setting, if possible.
360356
if preserveMode || preserveMtime || mode != 0 || mtime != 0 {
@@ -606,65 +602,15 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import
606602
if err != nil {
607603
return err
608604
}
609-
610-
// Parse the provide strategy to check if we should provide based on pin/MFS status
611-
strategyStr := cfg.Provide.Strategy.WithDefault(config.DefaultProvideStrategy)
612-
strategy := config.ParseProvideStrategy(strategyStr)
613-
614-
// Determine if we should provide based on strategy
615-
shouldProvide := false
616-
if strategy == config.ProvideStrategyAll {
617-
// 'all' strategy: always provide
618-
shouldProvide = true
619-
} else {
620-
// For combined strategies (pinned+mfs), check each component
621-
if strategy&config.ProvideStrategyPinned != 0 && dopin {
622-
shouldProvide = true
623-
} else if strategy&config.ProvideStrategyRoots != 0 && dopin {
624-
shouldProvide = true
625-
} else if strategy&config.ProvideStrategyMFS != 0 && toFilesSet {
626-
shouldProvide = true
627-
}
605+
if err := cmdenv.ExecuteFastProvide(req.Context, ipfsNode, cfg, lastRootCid.RootCid(), fastProvideWait, dopin, dopin, toFilesSet); err != nil {
606+
return err
628607
}
629-
630-
switch {
631-
case !cfg.Provide.Enabled.WithDefault(config.DefaultProvideEnabled):
632-
log.Debugw("fast-provide-root: skipped", "reason", "Provide.Enabled is false")
633-
case cfg.Provide.DHT.Interval.WithDefault(config.DefaultProvideDHTInterval) == 0:
634-
log.Debugw("fast-provide-root: skipped", "reason", "Provide.DHT.Interval is 0")
635-
case !shouldProvide:
636-
log.Debugw("fast-provide-root: skipped", "reason", "strategy does not match content", "strategy", strategyStr, "pinned", dopin, "to-files", toFilesSet)
637-
case !ipfsNode.HasActiveDHTClient():
638-
log.Debugw("fast-provide-root: skipped", "reason", "DHT not available")
639-
default:
640-
rootCid := lastRootCid.RootCid()
641-
642-
if fastProvideWait {
643-
// Synchronous mode: block until provide completes
644-
log.Debugw("fast-provide-root: providing synchronously", "cid", rootCid)
645-
if err := provideCIDSync(req.Context, ipfsNode.DHTClient, rootCid); err != nil {
646-
log.Warnw("fast-provide-root: sync provide failed", "cid", rootCid, "error", err)
647-
} else {
648-
log.Debugw("fast-provide-root: sync provide completed", "cid", rootCid)
649-
}
650-
} else {
651-
// Asynchronous mode (default): fire-and-forget, don't block
652-
log.Debugw("fast-provide-root: providing asynchronously", "cid", rootCid)
653-
go func() {
654-
// Use detached context with timeout to prevent hanging on network issues
655-
ctx, cancel := context.WithTimeout(context.Background(), fastProvideTimeout)
656-
defer cancel()
657-
if err := provideCIDSync(ctx, ipfsNode.DHTClient, rootCid); err != nil {
658-
log.Warnw("fast-provide-root: async provide failed", "cid", rootCid, "error", err)
659-
} else {
660-
log.Debugw("fast-provide-root: async provide completed", "cid", rootCid)
661-
}
662-
}()
663-
}
608+
} else if !fastProvideRoot {
609+
if fastProvideWait {
610+
log.Debugw("fast-provide-root: skipped", "reason", "disabled by flag or config", "wait-flag-ignored", true)
611+
} else {
612+
log.Debugw("fast-provide-root: skipped", "reason", "disabled by flag or config")
664613
}
665-
} else if fastProvideWait && !fastProvideRoot {
666-
// Log that wait flag is ignored when provide-root is disabled
667-
log.Debugw("fast-provide-root: wait flag ignored", "reason", "fast-provide-root disabled")
668614
}
669615

670616
return nil

0 commit comments

Comments
 (0)