Skip to content

spark-publish: streaming uploadΒ #170

@juliangruber

Description

@juliangruber

Instead of buffering all measurements into memory, creating one JSON blob, and then upload it, we can create a streaming pipeline with less memory needs.

For pg, use https://www.npmjs.com/package/pg-query-stream

For w3up, use

await client.uploadFile({ stream() => createStream() })

Metadata

Metadata

Assignees

No one assigned

    Projects

    Status

    πŸ—ƒ backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions