Skip to content

Backport index and ipni pdp tasks to PDPv0 #670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 22 commits into
base: pdpv0
Choose a base branch
from
Draft

Backport index and ipni pdp tasks to PDPv0 #670

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
eae8874
update pieceref schema and indexing task rough draft
ZenGround0 Sep 17, 2025
84d02c3
Bring in state view
ZenGround0 Sep 25, 2025
94a2db1
WIP
ZenGround0 Sep 30, 2025
77de42f
IPNI task done
ZenGround0 Oct 1, 2025
4c85996
Fixes
ZenGround0 Oct 1, 2025
c9cbc7b
Move alteration below creation
ZenGround0 Oct 1, 2025
13ea33e
Fix lint errors and actually use new tasks
ZenGround0 Oct 1, 2025
e9fb2f2
review response
ZenGround0 Oct 5, 2025
401b135
Fix calibnet curio bug
ZenGround0 Oct 5, 2025
b2879aa
Better errors to debug pdptool
ZenGround0 Oct 6, 2025
b726468
allow simple pdp service listener for testing
ZenGround0 Oct 6, 2025
286f80d
fix
ZenGround0 Oct 6, 2025
1d94cc4
Allow 0 listener
ZenGround0 Oct 6, 2025
42307e8
SimpleService
ZenGround0 Oct 7, 2025
940f49c
sql syntax error
ZenGround0 Oct 7, 2025
6bce99f
SQL missing AND keyword
ZenGround0 Oct 7, 2025
a61f473
Chill out ipni announce
ZenGround0 Oct 7, 2025
d934d38
More and less logging
ZenGround0 Oct 7, 2025
741f8f9
more debug
ZenGround0 Oct 7, 2025
927c000
Only publish new heads
ZenGround0 Oct 7, 2025
0499503
fix
ZenGround0 Oct 7, 2025
b7e8705
init
ZenGround0 Oct 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions cmd/curio/tasks/tasks.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,7 @@ func StartTasks(ctx context.Context, dependencies *deps.Deps, shutdownChan chan
return nil, err
}
var sdeps cuhttp.ServiceDeps
idxMax := taskhelp.Max(cfg.Subsystems.IndexingMaxTasks)

if cfg.Subsystems.EnablePDP {
es := getSenderEth()
Expand All @@ -299,11 +300,11 @@ func StartTasks(ctx context.Context, dependencies *deps.Deps, shutdownChan chan
pdpNextProvingPeriodTask := pdp.NewNextProvingPeriodTask(db, must.One(dependencies.EthClient.Val()), dependencies.Chain, chainSched, es)
pdpInitProvingPeriodTask := pdp.NewInitProvingPeriodTask(db, must.One(dependencies.EthClient.Val()), dependencies.Chain, chainSched, es)
pdpNotifTask := pdp.NewPDPNotifyTask(db)
activeTasks = append(activeTasks, pdpNotifTask, pdpProveTask, pdpNextProvingPeriodTask, pdpInitProvingPeriodTask)
pdpIndexingTask := indexing.NewPDPIndexingTask(db, iStore, dependencies.CachedPieceReader, cfg, idxMax)
pdpIpniTask := indexing.NewPDPIPNITask(db, sc, dependencies.CachedPieceReader, cfg, idxMax)
activeTasks = append(activeTasks, pdpNotifTask, pdpProveTask, pdpNextProvingPeriodTask, pdpInitProvingPeriodTask, pdpIndexingTask, pdpIpniTask)
}

idxMax := taskhelp.Max(cfg.Subsystems.IndexingMaxTasks)

indexingTask := indexing.NewIndexingTask(db, sc, iStore, pp, cfg, idxMax)
ipniTask := indexing.NewIPNITask(db, sc, iStore, pp, cfg, idxMax)
activeTasks = append(activeTasks, ipniTask, indexingTask)
Expand Down
2 changes: 1 addition & 1 deletion harmony/harmonydb/sql/20240823-ipni.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ CREATE TABLE ipni_peerid (
CREATE TABLE ipni (
order_number BIGSERIAL PRIMARY KEY, -- Unique increasing order number
ad_cid TEXT NOT NULL,
context_id BYTEA NOT NULL, -- abi.PieceInfo in Curio
context_id BYTEA NOT NULL, -- abi.PieceInfo || PDPIPNIContext in Curio
-- metadata column in not required as Curio only supports one type of metadata(HTTP)
is_rm BOOLEAN NOT NULL,

Expand Down
5 changes: 5 additions & 0 deletions harmony/harmonydb/sql/20251004-pdp-indexing.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
-- fields tracking indexing and ipni jobs over pdp pieces
ALTER TABLE pdp_piecerefs ADD COLUMN indexing_task_id BIGINT DEFAULT NULL;
ALTER TABLE pdp_piecerefs ADD COLUMN needs_indexing BOOLEAN DEFAULT FALSE;
ALTER TABLE pdp_piecerefs ADD COLUMN ipni_task_id BIGINT DEFAULT NULL;
ALTER TABLE pdp_piecerefs ADD COLUMN needs_ipni BOOLEAN DEFAULT FALSE;
1 change: 1 addition & 0 deletions market/ipni/ipni-provider/ipni-provider.go
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we filter out cid.contract for anything not mainnet. We should remove that check as well.

Original file line number Diff line number Diff line change
Expand Up @@ -489,6 +489,7 @@ func (p *Provider) StartPublishing(ctx context.Context) {
return
}
log.Info("Starting IPNI provider publishing for testnet build")
ticker = time.NewTicker(publishInterval)
if build.BuildType != build.BuildCalibnet {
ticker = time.NewTicker(time.Second * 10)
log.Info("Resetting IPNI provider publishing ticker to 10 seconds for devnet build")
Expand Down
Loading