Skip to content

linehaul parser for download should accept a size parameter #232

@miketheman

Description

@miketheman

In order to annotate how many bytes were transferred in a given download request, the upstream log needs to emit bytes that would be parsed here.

It appears that with parse_all=True, the parser format would need to be updated to allow for the existence of the new field. Does that mean duplicating MESSAGE_V3 to a MESSAGE_V4 with the added param at the end?

V3_HEADER = L("download")
MESSAGE_v3 = (
V3_HEADER + PIPE + REQUEST + PIPE + TLS + PIPE + PROJECT + PIPE + USER_AGENT
)

Will need to add resp.body_bytes_written to https://github.com/pypi/infra/blob/d6702898cbed406cc431a12b80270831ef292557/terraform/file-hosting/vcl/files.vcl#L589 to emit the value when this library can parse for it.

Will likely need coordination with BigQuery team to add column when the data is correctly parsed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions