Skip to content

tokens shouldn't need to track rawkind, rawvalue, only prefixes #2

@bgentry

Description

@bgentry

Here's the current token struct:

type token struct {
    kind      tkType
    macroName string
    value     string
    rawkind   []byte
    rawvalue  []byte
}

The fields rawkind and rawvalue track the original bytes from the token's kind and value, as well as the prefixing spaces (including newlines).

Everything aside from the prefix is redundant w/ the fields like kind and value. This makes for some extra work when updating values or initializing tokens.

Instead, we could rename these fields to valuePrefix and kindPrefix, and keep only the prefixing spaces in them. That would slightly complicate this append in MarshalText() but it seems like it'd be a huge improvement overall.

It would also eliminate the need to do copying on byte slices as they wouldn't be getting modified anymore.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions