Considering the File System API

[andreubotella]

The File System API is a relatively recent web standard, now implemented in all browsers, that allows accessing and modifying files in the file system. All browsers allow access through this API to a virtual filesystem, unique to a page's origin, through await navigator.storage.getDirectory(), that you can't go outside of.

However, Chromium extends this by providing additional APIs (such as window.showOpenFilePicker()) that give you access to arbitrary files and directories in the file system. Although other browsers are opposed to implementing these APIs because of safety and privacy concerns, this is not an issue for server-side runtimes.

I have heard that this API is not a great fit for server-side environments, but (as someone who hasn't looked in depth at the API and considered it) those concerns are not clear to me. So I opened this issue to have a place to discuss what it's missing, whether we can work with WHATWG to fix that, and also to discuss WinterCG extensions similar to showOpenFilePicker().

[ekwoka]

I agree that I'm not too sure what the disagreement would be for the server context (outside of runtimes that inherently don't expose any FS as is).

Everything has unified around the node:fs system ,but the File System API seems to be quite a lot better thought out.

[Qard]

It lacks the full feature set and flexibility of the posix fs model, for example file encodings and permissions. Because the API is designed as a tree structure with methods on each element, such additional capabilities can not be added to it without violating the spec.

Additionally, the use of Web Streams and subsequently EventTarget makes the performance insufficient for Node.js at present.

It may be reasonable to have it as a secondary interface, but not really usable as the only way to read and write files.

[andreubotella]

Additionally, the use of Web Streams and subsequently EventTarget makes the performance insufficient for Node.js at present.

As far as I can tell, none of the FS APIs inherit from EventTarget. The only thing remotely connected is FileReader, and we're already not including that in the common minimum API even though we do include File.

As for streams, my understanding is that it is possible to achieve implementations of web streams with enough performance, so this would only be a temporary blocker for Node.js, right? After all, the usage of web APIs in Node.js is only ever going to increase, and web APIs won't be adopting Node.js's streams, so the pressure on Node.js's implementation to be more performant will do anything but decrease.

[Qard]

Web streams are based on EventTarget, so until the performance of that is resolved it's unlikely we'd be able to get at all comparable performance with the existing fs API.

As I said though, the API simply lacks the capability of the existing API though, and has quite a lot of complexity in its object model, so I don't expect to see a Node.js implementation of it any time soon.

[andreubotella]

Web streams are based on EventTarget, so until the performance of that is resolved it's unlikely we'd be able to get at all comparable performance with the existing fs API.

I don't think that's correct. The streams spec has no mention of EventTarget, and the only mentions of events in normative text are related to passing stream chunks through a worker boundary when transferring streams, which Node.js would do with EventEmitter.

As I said though, the API simply lacks the capability of the existing API though, and has quite a lot of complexity in its object model, so I don't expect to see a Node.js implementation of it any time soon.

Part of the point of this thread is to have a concrete list of capabilities that that API is missing, to see if we can work together with the WHATWG to make that possible.

[mk-pmb]

From a few minutes of reading MDN's OPFS docs and a quick scan of the Node.js fs API ToC, my impression is that polyfills will not be feasible for these Node.js features, unless you implement a full file system on top of OPFS:

[Toggle list]

filehandle.chmod(mode)
filehandle.chown(uid, gid)
filehandle.stat([options])
filehandle.utimes(atime, mtime)
fsPromises.access(path[, mode])
fsPromises.chmod(path, mode)
fsPromises.chown(path, uid, gid)
fsPromises.lchmod(path, mode)
fsPromises.lchown(path, uid, gid)
fsPromises.lutimes(path, atime, mtime)
fsPromises.link(existingPath, newPath)
fsPromises.lstat(path[, options])
fsPromises.readlink(path[, options])
fsPromises.realpath(path[, options])
fsPromises.rename(oldPath, newPath)
fsPromises.stat(path[, options])
fsPromises.symlink(target, path[, type])
fsPromises.utimes(path, atime, mtime)
fsPromises.watch(filename[, options])

A polyfill for fsPromises.mkdtemp(prefix[, options]) will probably be possible only in an insecure way.

[jimmywarting]

The best thing we have so far is the low level:

• node:fs#openAsBlob
  ^{(which i think should have lived on the node:fs/promise#openAsBlob - but beside the point.}

• Bun.js inherently badly written Blob / File / BunFile class you get from calling Bun.file(path)

  • Blob LastModified incompatibility with Node oven-sh/bun#14257
  • Bun.file should extend File not Blob oven-sh/bun#5980

• This Deno hack for getting Blob/Files backed up by the file system.

this ☝️ are the core building blocks for creating something like: await fileHandle.getFile()

---

Really wished we could get something like a core File System Access for the backend, that works the same everywhere.

I have heard that this API is not a great fit for server-side environments

I think the highest thing that got the most backslash was the edit-in-place for large files

See WICG/file-system-access issue #67 for discussion around and desire for a "inPlace" mode for createWritable (where changes will be written to the actual underlying file as they are written to the writer, for example to support in-place modification of large files or things like databases). This is not currently implemented in Chrome. Implementing this is currently blocked on figuring out how to combine the desire to run malware checks with the desire to let websites make fast in-place modifications to existing large files. In-place writes are available for files in a bucket file system via the FileSystemSyncAccessHandle interface.

i suppose servers don't need to have the same concern and it can allow edit-in-place

[jasnell]

I've recently started an implementation for Cloudflare workers. Mainly feeling it out and seeing what issues there are for server environments. There are definitely a few warts and places where some things don't line up but for the most part it's pretty straightforward. Later, once this impl is done I'll probably do one for node.js

[Ethan-Arrowood]

I'd be very interested in trying to contribute FS spec for Node.

[jimmywarting]

I've recently started an implementation for Cloudflare workers. Mainly feeling it out and seeing what issues there are for server environments. There are definitely a few warts and places where some things don't line up but for the most part it's pretty straightforward. Later, once this impl is done I'll probably do one for node.js

@jasnell i don't really know how Cloudflare workers are built if they are using actual chrome(ium) or puppeteer
put if it's of any help i manage to find a way to programmatically load in a file/directory handle with the help of some puppeteer code: utilizing uploading files with drag and drop and then calling item.getAsFileSystemHandle()

just wanted to share this to you if you find it useful. but it seems you have already figure out how to load in a file system access? or maybe you are just using opfs...

  await page.evaluate(() => {
    document.body.draggable = true
    globalThis.ondragover = evt => evt.preventDefault()
    globalThis.ondrop = evt => {
      globalThis.ondrop = globalThis.ondragover = null
      document.body.draggable = false
      globalThis.root = evt.dataTransfer.items[0].getAsFileSystemHandle()
    }
  })
  await page.setDragInterception(true)
  const draggable = await page.$('body')
  const dropzone = await page.$('body')
  const data = await draggable.drag({ x: 0, y: 0 })
  data.files = [ filepath ]
  await dropzone.dragEnter(data)
  await dropzone.drop(data)
  await page.evaluate(() => {
    return globalThis.root.then(root => {
      globalThis.root = root
      console.log(root)
    })
  })