idea: augment job-list "filter" with RFC 31 constraint support

[grondo]

Problem: job-list is currently limited in the kinds of general "queries" it can support (e.g. it only supports a single userid, limited set of "filters" etc.).

Without re-implementing job-list as a database (which still may be the correct long term solution), another idea would be to support a version of the Constraint syntax proposed in RFC 31. The benefit of this approach is that the implementation is fairly simple but powerful, and we already have a nice string syntax proposed to build Constraint JSON objects easily (RFC 35), along with a Python parser which could go into flux jobs or flux pgrep.

Some prevalent use cases include:

• list the jobs for > 1 user (user:bob,alice)
• list all jobs that ran or are running on a given node (host:fluke16)
• list all jobs that ran on a bad node since monday morning and failed (host:fluke16 since:"Mon 8am" is:failed)
• list jobs today that failed but were not canceled or timed out (is:failed not (is:canceled or is:timeout))

etc.

The drawback to this approach is more work for the job-list module. It is possible that a user could construct a complex query which would act as a DoS for the service.

[chu11]

Given discussion in #4908. could RFC31 support sorting? (I don't think RFC35 does at the moment, but that could be updated as a result). I'm imagining something like: { "sort-ascending": [ { "properties": [ "t_submit" ] } ] }.

[grondo]

I think it should be a separate option, the way the objects in RFC 31 work currently is they apply like a filter, so the constraint is applied to each resource, job, object, etc. A sort operation doesn't make sense in that context. However, I'm open to ideas on how that could work.

We could instead add a wrapper around an RFC 31 constraint object that applies other operations to the query, e.g. a sort key or keys, the count of values to return, etc.?

[chu11]

curious what folks think, if job-list is updated to use support job constraints, should the old filtering mechanisms (and thus RPC format) be supported for backwards compatibility?

Due to how job-list currently operates and the refactoring that would be needed, it'd be nice to remove backwards compatibility. I imagine user facing tools like flux-jobs can simply convert the current user options into constraints.

[chu11]

after some offline discussion with @grondo, just thought I'd note

Without re-implementing job-list as a database (which still may be the correct long term solution)

I think this is the wrong solution at the moment, sqlite w/ json support is supported pre version 3.38 but requires compilation in, while it is native post 3.38. RHEL8 is at 3.26. sqlite w/ json support would make queries, searches, flexibility, etc. going forward a lot easier. See work in #4336 where we put all job data into a single json blob and store it for the job (as well as jobspec, R, etc.). It'd be limiting to fix on a specific sql table right now.

Not that we would require sqlite as the backend database, but considering other database solutions would also take an effort and probably isn't on the table at the moment. e.g. long term some nosql table which a lot of flexibility, possibly with storage across multiple nodes for scalable access once we start going to millions of jobs, e.g. abstraction layer across multiple database technologies, etc.

So I think "medium term" going with constraints is the right approach and longer out we should revisit the everything in a database approach.

curious what folks think, if job-list is updated to use support job constraints, should the old filtering mechanisms (and thus RPC format) be supported for backwards compatibility?

And also, discussed and getting rid of backwards compatibility in the RPC is probably fine. Especially given #4514

[chu11]

This is me somewhat just thinking out loud.

As I began writing some unit tests for a since constraint, I began to wonder if since cannot work within constraints without some potential performance issues. The inactive list is sorted by t_inactive, biggest t_inactive first. Currently when I iterate through the inactive list, the first t_inactive which is smaller than the since configuration allows me to just break and stop iterating. So we skip unnecessary iteration, yay.

But with something as simple as:

{ \"since\": [ 200.0 ] }
vs
{ \"not\": [ { \"since\": [ 200.0 ] } ] }

it's hard to know when to stop iterating in constraints with the potential of a negation option.

The "however" is that flux-jobs and tools that get job-list pretty much all default since to 0, and the majority of the skipping of iteration is knowing not to search the inactive list. So in practice, this probably just doesn't matter that much. Most people are doing flux jobs -a and iterating through the whole inactive list all of the time anyways. And given default max return counts it probably matter even less.

[grondo]

Haven't looked at this yet, but maybe at first --since should be handled as a special case for the reason you mention. If a user wants to specify the equivalent of not since:200 they should instead be using something like:

$ src/cmd/flux pgrep --dry-run @yesterday..
{"not": [{"t_cleanup": ["<2023-04-27 00:00:00-07:00"]}]}

Attempting to stop iterating based on a constraint object is probably a lost cause. If we're worried about performance here then maybe we should jump straight to a database and instead work on converting RFC 35 query syntax to a database query.

[chu11]

{"not": [{"t_cleanup": ["<2023-04-27 00:00:00-07:00"]}]}

Yeah. My initial implementation / prototype is the just simply convert what is already there. I'm thinking follow up enhancements to add other filtering (most notably hostlist b/c of #4186, but could add many of the things you added in your pgrep PR) or in this case, add filtering per specific timestamp type, so since could be retired. Edit: and support lt, gt, etc.

Attempting to stop iterating based on a constraint object is probably a lost cause.

Yeah, I think i've come to that conclusion.

If we're worried about performance here then maybe we should jump straight to a database and instead work on converting RFC 35 query syntax to a database query.

I'm not too worried at this point given the points I made above. One of those "I'd like to keep on skipping iteration when not necessary", but in reality the vast majority of queries are skipping the inactive list all together. And when the inactive list is iterated, probably 99% of the time it iterates through the whole list until the "max count" is reached anyways. So I don't think it really loses us much.

[grondo]

I'll just note that one of the main use cases here will be queries like "what jobs ran on a given set of hosts during this time period" so queries to the inactive list may be common (when using this feature anyway).