No way to query past Query.Prefix?

[drew-512]

Hi gents!

Currently, Query.Prefix is used to do the initial seek when a Query begins, and iteration only continues while that prefix holds. There needs to be a flag that basically says, hey, don't use Prefix once iteration has begun. Otherwise, there's no way to effectively do a seek and do an arbitrary amount of scanning. This seems like a critical feature to leave behind, yikes.

Hopefully, I'm missing something!?

For example, in go-ds-badger: (line 270+)

it := txn.NewIterator(opt)
it.Seek(prefix)
...
qrb.Process.Go(func(worker goprocess.Process) {
    ...
    for sent := 0; it.ValidForPrefix(prefix); sent++ {
        ...
    }
}
...

could be enhanced to:

it := txn.NewIterator(opt)
it.Seek(prefix)
if q.PrefixOnlySeeks {
    prefix = nil
}
...

Adding this would be easy, but all the impls would need to be updated of course. So maybe it's done as a selectively available feature?

This is a dealbreaker for us, but we love Datastore and want to use and support it! If some at PL agrees to accept our pull requests in a timely fashion, we'll help do the work. I'm sad we're now in this position, and would take less than a few mins for dbs like leveldb, badger, and bolt. What do y'all think?

We kindly need resolution on this relatively soon one way or the other so that we know we need to pass on Datastore.

Thanks,
Drew

[magik6k]

Some datastores may not be able to support this (like flatfs, but it doesn't even support ordered iterating, so it's probably fine).

👍 from me, just make sure that it's optional and not enabled by default (default = zero value)

cc @Stebalien @bigs

[drew-512]

Ok, cool, thanks for the quick response.

How about I whip up some changes, post them here for initial review/discussion and if they look good, we can take it from there.

[drew-512]

Ok, so so here's my proposed change to Query.

query.go ~ line 60

type Query struct {
	Prefix            string   // namespaces the query to results whose keys have Prefix
	IterationPrefix   string   // if set, this overrides Prefix during query iteration (n/a for some impls)

At first I was thinking a bool as I described in my prev post, but the drawback is that client would be forced to step entry by entry in order to close the query (as she checks each Result.Entry.Key to see if iteration has proceeded sufficiently far). That would mean the client couldn't use Query.Rest() since iteration would go on and on.

So If we go with adding the IterationPrefix field (above), the behavior augmentation is clear and well-defined. Impl changes are also trivial and unambiguous.

If y'all like this, give me the green light and I'll start submitting the changes. I'm OCD when it comes to naming and comments, so don't be shy for any name, style, or comment edits -- I get it. :)

[magik6k]

Prefix/IterationPrefix don't really seem intuitive, how about Prefix/StartPrefix:

type Query struct {
  Prefix            string
  StartPrefix     string
[...]
}

func Query[...]
  if q.StartPrefix == "" {
    q.StartPrefix = q.Prefix
  }
  if !strings.HasPrefix(q.StartPrefix, q.Prefix) {
    return errors.New("StartPrefix not within Prefix")
  }
  [...]

  it.Seek(q.StartPrefix)
  for [...] it.ValidForPrefix(q.Prefix)

[drew-512]

Love it -- great suggestion!

[drew-512]

SeekPrefix or StartPrefix? SeekPrefix may more accurately describe the behavior for impls that literally have seek (vs prefix) behavior.

[magik6k]

Yeah, SeekPrefix makes more sense

[bigs]

i'm definitely not against this—i think it could be quite useful. i think we'll need to update datastores that don't natively support this to provide it or at least heavily document its absence.

[drew-512]

How about:

type Query struct {
	Prefix            string   // namespaces the query to results whose keys have Prefix
	SeekPrefix        string   // if set, an alternate to Prefix that initially positions the query (n/a for some impls)
...

[drew-512]

Ok, so I made the discussed changes to leveldb, badger, and bolt -- changes were clean, cost-free, were only a couple lines, and have been tested. Meanwhile, I'm not familiar with S3 to know the obvious path, so I was going to pass on S3 for now. If y'all want, I can have go-ds-S3/s3.go::Query() return an unimp error if Query.SeekPrefix is set -- just let me know.

[magik6k]

Feel free to open the PRs, it's easier to discuss changes there. (https://github.com/whyrusleeping/sql-datastore/ will also need support)

[drew-512]

ok great

[magik6k]

Can you link the PRs here? (makes it easier for others to follow)

[drew-512]

Oops, yeah good idea. :D

4 PRs submitted. Will do SQL once we're past this step if that werks.

The 3 ds impl changes will of course get compile fails until the Query.SeekPrefix change is merged.

ds.Query:
#119

badger:
ipfs/go-ds-badger#47

leveldb:
ipfs/go-ds-leveldb#26

bolt: (I also added support for Query.Prefix)
ipfs/go-ds-bolt#1