question regarding comby's usage in direct-to-datalog

[0xalpharush]

Hi, I was trying to update the go call graph example to a more recent comby version. I came up with the following and omitted the meta DSL and omega engine:

comby 'func :[f](:[_]) {...:[[s]](...)...}' '`edges(:[f], :[s])`' -exclude-dir vendor -timeout 120 -d syncthing -stdout -newline-separated .go -lang .go -stats

This source code

func (c wireFormatConnection) Request(ctx context.Context, folder string, name string, blockNo int, offset int64, size int, hash []byte, weakHash uint32, fromTemporary bool) ([]byte, error) {
	name = norm.NFC.String(filepath.ToSlash(name))
	return c.Connection.Request(ctx, folder, name, blockNo, offset, size, hash, weakHash, fromTemporary)
}

rewrites to

`edges((c wireFormatConnection) Request(ctx context.Context, folder string, name string, blockNo int, offset int64, size int, hash []byte, weakHash uint32, fromTemporary bool) , String)`

How might one also capture that wireFormatConnection.Request invokes Connection.Request? More generally, I guess this is just asking how to grab all possible enumerations of matches and not just the first...

P.S. the paper is fascinating and the use of comby to extract datalog facts is ingenious!

[rvantonder]

Hi, wow never would have thought this would show up

-- So it almost works in the playground except there's a bug, I'll push the fix and link to a live example here. Meanwhile:

The answer is hidden in that repository so it's good you asked. Basically for that example, and trying to follow it as closely as what you would like, I would do it this way:

Match

func (... :[type]) :[f.](...) (:[ret]) {:[body]}

^ captures the type and function and body. what we are going to do output is :[body], but where :[body] as been totally replaced, and where each line output is an edge. so, the rewrite is just:

Rewrite

:[body]

Now the interesting bit, you do:

Rule

where rewrite :[body] {
    :[s.](:[i]) -> edges(:[type].:[f], :[s])
}

This says: "inside :[body], match all function-like syntax :[s](:[_]) and for each, output edges(:[type].:[f], :[s]). Note that you can still reference :[type] and :[f] here).

For your example you will get:

edges(wireFormatConnection.Request, norm.NFC.String)
edges(wireFormatConnection.Request, c.Connection.Request)

but wait there's more! what if from your example you wanted to capture the fact that wireFormatConnection.Request also transitively invokes filepath.ToSlash(name) (since that's in the function body, useful for the graph). Well, you can simply add nested to the rewrite rule:

Rule

where nested, rewrite :[body] {
    :[s.](:[i]) -> edges(:[type].:[f], :[s])
}

This tells comby "okay when you do the rewrite find all :[s.](:[i]) including if they nest.

Running:

Iimportant, on the command-line you put -newline-separated (this means, for each rewrite, output on a separate line)

Caveats/tips:

• the rewrite rule is kind of janky, I recommend explicitly naming all variables like :[a], :[b], :[c], don't use :[_]

• You will need to handle different kinds of Go func definitions like:
  func (c f) Request(ctx context.Context) /* no return type here! */ { ... }
  versus
  func foo(bar) returnType { ... }

(i.e., return type and receivers are optional in Go, but comby doesn't have a very nice way to specify optional things here, it's doable with regex but I suggest just having separate patterns)

Full example you can copy paste into your terminal (latest version is fine, you don't need the -omega thing)

COMBY_M="$(cat <<"MATCH"                                                                               [21:38:30]
func (... :[type]) :[f.](...) (:[ret]) {:[body]}
MATCH
)"
 COMBY_R="$(cat <<"REWRITE"
:[body]
REWRITE
)"
 COMBY_RULE="$(cat <<"RULE"
where nested, rewrite :[body] {
    :[s.](:[i]) -> edges(:[type].:[f], :[s])
}
RULE
)"
 comby "$COMBY_M"  "$COMBY_R" -rule "$COMBY_RULE" .go -match-newline-at-toplevel -newline-separated -stdout

output

edges(wireFormatConnection.Request, norm.NFC.String)
edges(wireFormatConnection.Request, filepath.ToSlash)
edges(wireFormatConnection.Request, c.Connection.Request)

P.S. the paper is fascinating and the use of comby to extract datalog facts is ingenious!

I thought it's so neat but had a bunch of 💩 reviewers on the paper and stopped bother and now I'm sad. But your comment made me happy.

[0xalpharush]

Thanks for the thorough walkthrough!

I am anticipating the possibility of name collisions causing issues or wanting to resolve the qualified path to its module, and I came across this blog about using LSIF. Is that still supported?

I tried adding /* lisf info :[s].lsif.hover */ to the rewrite rule and ran against kubernetes (Sourcegraph appears to have "precise navigation" enabled for that repo), but it was interpreted as a string literal and not replaced by an API response.

If it is still supported by comby, would I be able to setup an LSIF endpoint for codebases of interest without using Sourcegraph?

[rvantonder]

Is that still supported?

Yep, still in the code.

If it is still supported by comby, would I be able to setup an LSIF endpoint for codebases of interest without using Sourcegraph?

Yes, should work, you'll need to point it at an LSIF server! The code is experimental but it is at the point that it works. Take a look at https://github.com/comby-tools/comby/blob/master/lib/app/configuration/external_semantic.ml

[0xalpharush]

Thanks, I think I figured out why I wasn't getting a response with the debug logging.

comby/lib/app/configuration/external_semantic.ml

Lines 35 to 37 in ccdeb2b

	if debug then Format.printf "Querying type at %d::%d@." line column;
	let context = Lsif.Context.{ repository; lsif_endpoint; revision; formatting = Markdown } in
	Lsif.hover_at context ~filepath:filepath_relative_root ~line ~column

Querying type at 131::37
Big error: (Failure
  "No SSL or TLS support compiled into Conduit. You must install it through opam with one of these commands: `opam install lwt_ssl` or `opam install tls-lwt`")

Adding those packages doesn't fix the error. I can try building from source later. I have been using 1.8.1 via brew

[rvantonder]

Strange. Ah, it may be that lwt_ssl is missing from this opam file, https://github.com/comby-tools/comby/blob/master/comby-semantic.opam. If you build from source it might work (after you have done opam install lwt_ssl) because it's a dep here: https://github.com/comby-tools/comby/blob/master/lib/semantic/dune.

Or I can take a look in a day or two fixing upstream (no promises)