Skip to content

Backport index and ipni pdp tasks to PDPv0 #670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 22 commits into
base: pdpv0
Choose a base branch
from
+2,375 −12
Draft

Backport index and ipni pdp tasks to PDPv0 #670

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
eae8874
update pieceref schema and indexing task rough draft
ZenGround0 Sep 17, 2025
84d02c3
Bring in state view
ZenGround0 Sep 25, 2025
94a2db1
WIP
ZenGround0 Sep 30, 2025
77de42f
IPNI task done
ZenGround0 Oct 1, 2025
4c85996
Fixes
ZenGround0 Oct 1, 2025
c9cbc7b
Move alteration below creation
ZenGround0 Oct 1, 2025
13ea33e
Fix lint errors and actually use new tasks
ZenGround0 Oct 1, 2025
e9fb2f2
review response
ZenGround0 Oct 5, 2025
401b135
Fix calibnet curio bug
ZenGround0 Oct 5, 2025
b2879aa
Better errors to debug pdptool
ZenGround0 Oct 6, 2025
b726468
allow simple pdp service listener for testing
ZenGround0 Oct 6, 2025
286f80d
fix
ZenGround0 Oct 6, 2025
1d94cc4
Allow 0 listener
ZenGround0 Oct 6, 2025
42307e8
SimpleService
ZenGround0 Oct 7, 2025
940f49c
sql syntax error
ZenGround0 Oct 7, 2025
6bce99f
SQL missing AND keyword
ZenGround0 Oct 7, 2025
a61f473
Chill out ipni announce
ZenGround0 Oct 7, 2025
d934d38
More and less logging
ZenGround0 Oct 7, 2025
741f8f9
more debug
ZenGround0 Oct 7, 2025
927c000
Only publish new heads
ZenGround0 Oct 7, 2025
0499503
fix
ZenGround0 Oct 7, 2025
b7e8705
init
ZenGround0 Oct 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions cmd/curio/tasks/tasks.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,7 @@ func StartTasks(ctx context.Context, dependencies *deps.Deps, shutdownChan chan
return nil, err
}
var sdeps cuhttp.ServiceDeps
idxMax := taskhelp.Max(cfg.Subsystems.IndexingMaxTasks)

if cfg.Subsystems.EnablePDP {
es := getSenderEth()
Expand All @@ -299,11 +300,11 @@ func StartTasks(ctx context.Context, dependencies *deps.Deps, shutdownChan chan
pdpNextProvingPeriodTask := pdp.NewNextProvingPeriodTask(db, must.One(dependencies.EthClient.Val()), dependencies.Chain, chainSched, es)
pdpInitProvingPeriodTask := pdp.NewInitProvingPeriodTask(db, must.One(dependencies.EthClient.Val()), dependencies.Chain, chainSched, es)
pdpNotifTask := pdp.NewPDPNotifyTask(db)
activeTasks = append(activeTasks, pdpNotifTask, pdpProveTask, pdpNextProvingPeriodTask, pdpInitProvingPeriodTask)
pdpIndexingTask := indexing.NewPDPIndexingTask(db, iStore, dependencies.CachedPieceReader, cfg, idxMax)
pdpIpniTask := indexing.NewPDPIPNITask(db, sc, dependencies.CachedPieceReader, cfg, idxMax)
activeTasks = append(activeTasks, pdpNotifTask, pdpProveTask, pdpNextProvingPeriodTask, pdpInitProvingPeriodTask, pdpIndexingTask, pdpIpniTask)
}

idxMax := taskhelp.Max(cfg.Subsystems.IndexingMaxTasks)

indexingTask := indexing.NewIndexingTask(db, sc, iStore, pp, cfg, idxMax)
ipniTask := indexing.NewIPNITask(db, sc, iStore, pp, cfg, idxMax)
activeTasks = append(activeTasks, ipniTask, indexingTask)
Expand Down
10 changes: 9 additions & 1 deletion harmony/harmonydb/sql/20240823-ipni.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,18 @@ CREATE TABLE ipni_peerid (
sp_id BIGINT NOT NULL -- 20241106-market-fixes.sql UNIQUE
);

-- To enable separation of access based on storage service we create a separate provider identity
-- for PDP IPNI requests. This is a practical concern for current IPNI operations with limited resources.
CREATE TABLE ipni_pdp_peerid (
singleton BOOLEAN NOT NULL DEFAULT TRUE PRIMARY KEY CHECK (singleton = TRUE),
priv_key BYTEA NOT NULL,
peer_id TEXT NOT NULL UNIQUE
);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to go in a new file. My suggestion would be not create a new table. MK2.0 also uses the existing table.

CREATE TABLE ipni (
order_number BIGSERIAL PRIMARY KEY, -- Unique increasing order number
ad_cid TEXT NOT NULL,
context_id BYTEA NOT NULL, -- abi.PieceInfo in Curio
context_id BYTEA NOT NULL, -- abi.PieceInfo || PDPIPNIContext in Curio
-- metadata column in not required as Curio only supports one type of metadata(HTTP)
is_rm BOOLEAN NOT NULL,

Expand Down
4 changes: 4 additions & 0 deletions harmony/harmonydb/sql/20240930-pdp.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,10 @@ CREATE TABLE pdp_piecerefs (
FOREIGN KEY (service) REFERENCES pdp_services(service_label) ON DELETE CASCADE,
FOREIGN KEY (piece_ref) REFERENCES parked_piece_refs(ref_id) ON DELETE CASCADE
);
ALTER TABLE pdp_piecerefs ADD COLUMN indexing_task_id BIGINT DEFAULT NULL;
ALTER TABLE pdp_piecerefs ADD COLUMN needs_indexing BOOLEAN DEFAULT FALSE;
ALTER TABLE pdp_piecerefs ADD COLUMN ipni_task_id BIGINT DEFAULT NULL;
ALTER TABLE pdp_piecerefs ADD COLUMN needs_ipni BOOLEAN DEFAULT FALSE;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would need to go into a new file. Always expect the existing SQL files are executed and will not be executed again.


-- PDP hash to piece cid mapping
CREATE TABLE pdp_piece_mh_to_commp (
Expand Down
Loading