Parsing key = value where both can be identical. Unexpected token "key" or unexpected token "value".

[jusito]

Hello participle community,

I stumbled upon an issue which I broke down.
Problematic input: 1 = 2

Whats is my expectation:
The input should be interpreted as Ident(1)Whitespace( )Other(=)Whitespace( )Number(2) but, depending on rule ordering, fails with unexpected 1 or unexpected 2. I would expect that I should put Ident first because wherever Ident is used as Tag there will always be an equals right after.

Whats my assumption of the issue:
Ident and Number can start with a digit which is required. I assume that the parser will get either Ident = Ident or Number = Number, because lexer preprocesses it. And because of that the parser tag of "Key" (Ident '='") has no effect. Therefore a potential workaround could be adding \s*= to Ident pattern and strip it somehow in post processing but this solution feels like I am missing something. Lexer doesn't seem to like positive lookahead, maybe Map as parser option, but no idea how to use it.

Questions:
Is my assumption about this issue correct?
What is the best practice for this?

Example with participle v2.1.4:

package main

import (
	"testing"

	"github.com/alecthomas/participle/v2"
	"github.com/alecthomas/participle/v2/lexer"
)

type Root struct {
	Objects []Pair `parser:"@@*"`
}

type Pair struct {
	Key   string  `parser:"@Ident '='"`
	Value float64 `parser:"@Number"`
}

func TestIssue(t *testing.T) {
	lexer := lexer.MustSimple([]lexer.SimpleRule{
		{Name: "Ident", Pattern: "[0-9_a-zA-Z]+"},
		{Name: "Number", Pattern: `[-+]?(\d*\.)?\d+`},
		{Name: "Whitespace", Pattern: `[ \t]+`},
		{Name: "Other", Pattern: `[-={}@\n\r]`},
	})

	parser := participle.MustBuild[Root](
		participle.Lexer(lexer),
		participle.Elide("Whitespace"),
	)

	config, err := parser.ParseString("", `1 = 2`)
	t.Logf("Got config %v", config)
	if err != nil {
		t.Fatalf("Failed to parse input for issue. Reason: %v", err)
	}
	if len(config.Objects) != 1 {
		t.Fatalf("Failed to find root element. Got %v", config)
	}
}

[alecthomas]

I'm not sure what the issue is, but something that stands out to me is that your Ident rule matches [0-9], so you will likely never see Number at all. You should change Ident to [_a-zA-Z][0-9a-zA-Z_]*

[jusito]

Thanks for the quick reply, that is sadly intentional. Sounds like I need a stateful lexer or a workaround?

Details:
The config files I want to parse in the end looks like this:

@foo = 3 # constant number
1 = { # Tier 1
  base = foo
}
bar = {
  tier = 1 # which is a reference but currently only for the field tier
}

But this is just in one specific folder so one workaround could be just pre and post process all files in that folder.

[alecthomas]

Even with that example, you should not overlap your lexer rules. You solve that at the parser level:

type Pair struct {
	Key   string  `parser:"(@Ident | @Number) '='"`
	Value float64 `parser:"@Number"`
}

[jusito]

Thanks a lot! That makes sense. Probably for optimal case I would need to create some additional lexer rules to catch every case, e.g. 1foo = { ...}. Have a great day!

[alecthomas]

Yeah I would change use match on word boundary in your rules, then everything should work correctly:

		{Name: "Number", Pattern: `\b[-+]?(\d*\.)?\d+\b`},
		{Name: "Ident", Pattern: `\b[0-9_a-zA-Z]+\b`},

I'd change the order as above so that a pure number always matches over an identifier.