Skip to content

fix: do not download all blocks when listing dir #261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SgtPooki merged 7 commits into from
Sep 15, 2025
Merged

fix: do not download all blocks when listing dir #261

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e4d7a15
fix: do not fetch each item in directories
SgtPooki Aug 5, 2025
12f75b3
chore: remove unused extended: false
SgtPooki Aug 5, 2025
290cbe2
fix: createOfflineHelia fixture passes through types
SgtPooki Aug 5, 2025
a02e04a
chore: remove comma dangle
SgtPooki Aug 5, 2025
f25fa11
Update packages/verified-fetch/src/plugins/plugin-handle-dag-pb.ts
SgtPooki Aug 5, 2025
12f5a99
Update packages/verified-fetch/src/utils/walk-path.ts
SgtPooki Aug 5, 2025
c61e9a3
Merge branch 'main' into 260-bug-fsls-seems-to-query-too-many-blocks
SgtPooki Sep 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions packages/verified-fetch/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@
"@helia/interface": "^5.3.1",
"@helia/ipns": "^8.2.2",
"@helia/routers": "^3.1.1",
"@helia/unixfs": "^5.0.2",
"@helia/unixfs": "^5.1.0",
"@ipld/car": "^5.4.2",
"@ipld/dag-cbor": "^9.2.3",
"@ipld/dag-json": "^10.2.4",
Expand All @@ -187,7 +187,7 @@
"helia": "^5.4.1",
"interface-blockstore": "^5.3.1",
"interface-datastore": "^8.3.1",
"ipfs-unixfs-exporter": "^13.6.2",
"ipfs-unixfs-exporter": "^13.7.2",
"ipns": "^10.0.2",
"it-map": "^3.1.3",
"it-pipe": "^3.0.1",
Expand All @@ -211,7 +211,7 @@
"browser-readablestream-to-it": "^2.0.9",
"datastore-core": "^10.0.2",
"helia": "^5.4.1",
"ipfs-unixfs-importer": "^15.3.2",
"ipfs-unixfs-importer": "^15.4.0",
"it-all": "^3.0.8",
"it-last": "^3.0.8",
"it-to-buffer": "^4.0.9",
Expand Down
3 changes: 2 additions & 1 deletion packages/verified-fetch/src/plugins/plugin-handle-dag-pb.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ export class DagPbPlugin extends BasePlugin {
const entry = await handleServerTiming('exporter-dir', '', async () => exporter(`/ipfs/${dirCid}/${rootFilePath}`, helia.blockstore, {
signal: options?.signal,
onProgress: options?.onProgress
// extended: false
}), withServerTiming)

log.trace('found root file at %c/%s with cid %c', dirCid, rootFilePath, entry.cid)
Expand All @@ -111,7 +112,7 @@ export class DagPbPlugin extends BasePlugin {
context.modified++
this.log.trace('attempting to get directory entries because index.html was not found')
try {
for await (const dirItem of fs.ls(dirCid, { signal: options?.signal, onProgress: options?.onProgress })) {
for await (const dirItem of fs.ls(dirCid, { signal: options?.signal, onProgress: options?.onProgress, extended: false })) {
context.directoryEntries.push(dirItem)
}
// dir-index-html plugin or dir-index-json (future idea?) plugin should handle this
Expand Down
2 changes: 1 addition & 1 deletion packages/verified-fetch/src/utils/dir-index-html.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ export const dirIndexHtml = (dir: UnixFSEntry, items: UnixFSEntry[], { gatewayUR
},
listing: items.map((item) => {
return {
size: item.size.toString(),
size: item.size?.toString() ?? '?',
Copy link
Member Author

@SgtPooki SgtPooki Aug 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we no longer have the sizes.. we will need to either fetch them with stat or decide on a better default.

name: item.name,
path: getItemPath(item),
hash: item.cid.toString(),
Expand Down
4 changes: 3 additions & 1 deletion packages/verified-fetch/src/utils/walk-path.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ async function walkPath (blockstore: ReadableStorage, path: string, options?: Pa
const ipfsRoots: CID[] = []
let terminalElement: UnixFSEntry | undefined

for await (const entry of exporterWalk(path, blockstore, options)) {
for await (const entry of exporterWalk(path, blockstore, {
...options
})) {
ipfsRoots.push(entry.cid)
terminalElement = entry
}
Expand Down
4 changes: 2 additions & 2 deletions packages/verified-fetch/test/fixtures/create-offline-helia.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ import { createHeliaHTTP } from '@helia/http'
import { MemoryBlockstore } from 'blockstore-core'
import { IdentityBlockstore } from 'blockstore-core/identity'
import { MemoryDatastore } from 'datastore-core'
import type { HeliaInit } from 'helia'

export async function createHelia (init: Partial<HeliaInit> = {}): Promise<ReturnType<typeof createHeliaHTTP>> {
export async function createHelia (...args: Parameters<typeof createHeliaHTTP>): Promise<ReturnType<typeof createHeliaHTTP>> {
const [init] = args
const datastore = new MemoryDatastore()
const blockstore = new IdentityBlockstore(new MemoryBlockstore())

Expand Down
Loading