The documented JSON output for `/api/v0/dag/import` is incorrect

[smoyer64]

The current documentation describes /api/v0/dag/import as returning a JSON body with the following structure:

{
  "Root": {
    "Cid": {
      "/": "<cid-string>"
    },
    "PinErrorMsg": "<string>"
  },
  "Stats": {
    "BlockBytesCount": "<uint64>",
    "BlockCount": "<uint64>"
  }
}

What is actually returned is a JSONL body with the following structure:

{"Root":{"Cid":{"/":"<cid-string>"},"PinErrorMsg":"string"}}
{"Stats":{"BlockCount":<uint64>,"BlockBytesCount":<uint64>}}

Note that the Stats "record" is only returned if the stats=true argument is provided.

[welcome]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additiona round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.tech if you just need general support.

[lidel]

@smoyer64 thank you for filling this.

As @2color noted, these docs are generated by code mentioned in #1981 (comment)

Your analysis from #1981 (comment) is spot-on, yes, it is intentional.

I think the only missing piece of the puzzle is streaming and why we have it and that CLI uses it by default. Some context below:

• ipfs CLI makes requests with streaming enabled for all commans by default ( --stream-channels=true is the implicit default and it makes HTTP client append &stream-channels=true to RPC requests).

• This makes RPC endpoint respond in streaming fashion (NDJSON-like), flushing after every object, allowing server to not have to buffer everything in memory before returning result, and client to consume and react to partial results before entire request is processed.

  Click to expand `ipfs dag import` example

  You can see what happens on the wire bit better with some mitm like socat:

  $ echo -n "hello" | ipfs block put --pin=true
  $ ipfs dag export bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq > hello.car
  $ ipfs dag import --pin-roots=true --stats --api /dns/localhost/tcp/7423 hello.car

  And then socat proxy shows what happens on the wire bit better:

  $ socat -v TCP-LISTEN:7423,fork TCP:localhost:5001
  [... sending CAR ... ]
  < 2025/01/16 22:48:40.000221008  length=566 from=0 to=565
  HTTP/1.1 200 OK\r
  Access-Control-Allow-Headers: X-Stream-Output, X-Chunked-Output, X-Content-Length\r
  Access-Control-Expose-Headers: X-Stream-Output, X-Chunked-Output, X-Content-Length\r
  Content-Type: application/json\r
  Server: kubo/0.34.0-dev\r
  Trailer: X-Stream-Error\r
  Vary: Origin\r
  X-Chunked-Output: 1\r
  X-Fookey: BarVal\r
  Date: Thu, 16 Jan 2025 21:48:40 GMT\r
  Connection: close\r
  Transfer-Encoding: chunked\r
  \r
  66\r
  {"Root":{"Cid":{"/":"bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq"},"PinErrorMsg":""}}
  \r
  2f\r
  {"Stats":{"BlockCount":1,"BlockBytesCount":5}}
  \r
  < 2025/01/16 22:48:40.000221600  length=5 from=566 to=570
  0\r
  \r

• In practice, you always want to use streaming if the command may return more than one result object. Commands that require streaming like ping or dag import probably do not even allow you to disable streaming.

The docs for ipfs dag import (and all other commands) generated by /tools/http-api-docs seem to miss the concept of streaming in general, and all possible return objects are wrapped in parent {}.

Next steps

With that context in mind, there are probably various ways of fixing this in /tools/http-api-docs generator.

• We could add a note that when streaming is used, wrapping {} is not present, and objects are separated by \n.

• We could add a StreamingExample flag to https://github.com/ipfs/go-ipfs-cmds similar to Status and annotate commands like dag import to indicate that streaming line-separated output should be included in addition to single JSON?

• Something smarter?

Ideas / PRs against /tools/http-api-docs welcome.