Suggestion - add an indication in `flux jobs` when results are truncated

[kkier]

Current behavior is that e.g. flux jobs -A will return the first(?) 1000 jobs and then exit, without a user necessary realizing that there was more output to be had, since the -c/--count command isn't necessarily advertised unless you read the help. Adding something like 1000 results returned - 1312 results omitted as a last line would at least provide hinting that there's more to be seen.

[chu11]

I was pondering this. Internally within the job-list module, we effectively go through internal lists of all jobs and apply our constraints / filtering rules to see if the job matches. If it matches, we save that off and when we hit the -c/--count cap, we return and we're done.

So in order to get a output result like "1000 results returned - 1312 results omitted", we'd have to iterate through all the rest of the jobs and continue to test/filter. This might not be horrific right now, but with inactive jobs in a DB like with #4336 in the future, that's bad.

So the question is, what could we provide? Would something as simple as Job list cap reached, see --count to increase cap be ok? The minor technical corner case is that we don't know if there are more jobs to be listed. We only know the cap was reached. I assume the message is what's most important, not the "omitted" count.

[grondo]

Agree.

Also, it might be possible to check if there is at least one more matching job and set a truncated flag in the response?

I would just go with a simple error message like:

warning: output truncated at 1000 jobs. Use --filter to reduce matching jobs or --count to increase returned results

or similar.

[kkier]

Good points. Yes, I think checking for N+1 and then messaging the user about it is probably the way to go. Having a simple flux jobs query take some number of minutes to return because it's looping through the set of all jobs since the dawn of creation might have bad effects on my blood pressure.

[chu11]

Also, it might be possible to check if there is at least one more matching job and set a truncated flag in the response?

The issue is we don't know if there's at least one more. We may have to scan all the way to the beginning of time.

[kkier]

Also, it might be possible to check if there is at least one more matching job and set a truncated flag in the response?

The issue is we don't know if there's at least one more. We may have to scan all the way to the beginning of time.

I may be confused:
If we look for 1001 jobs and only return 1000, that's not really taking any longer than if we were looking for 1000, right?

[grondo]

Yeah, that should be a reasonable approach.

An alternate approach would be if job-list gets to count entries before reaching the end of the list of all jobs, it could set the flag which indicates there may be more entries and to adjust the warning accordingly. The vast majority of the time the warning will be meaningful.

In fact, with the above approach, the feature could live in flux-jobs.py alone and we don't need to touch job-list.

However, we should try looking for count+1 instead of count and see if that raises an issue.

[chu11]

If we look for 1001 jobs and only return 1000, that's not really taking any longer than if we were looking for 1000, right?

What I was getting at is that after we iterate through jobs and match 1000 entries, we have no way to know for sure if there is a 1001th match. We could keep on iterating on all the remaining jobs until the very end. So we may end up scanning till the beginning of time anyways.

[chu11]

However, we should try looking for count+1 instead of count and see if that raises an issue.

So my initial attempt was to do this dumbly ... if the returned jobs array length is == args.count, then output a message. But ...

An alternate approach would be if job-list gets to count entries before reaching the end of the list of all jobs, it could set the flag which indicates there may be more entries and to adjust the warning accordingly. The vast majority of the time the warning will be meaningful.

I sort of like this idea. A flag indicating "did we reached the end or not" ....

But then again, if we reached the output count of args.count .... the odds are there would be more anyways. So the change to jobs-list is unnecessary?

[chu11]

If we look for 1001 jobs and only return 1000, that's not really taking any longer than if we were looking for 1000, right?

Oh it just occurred to me that maybe I misinterpreted what you said.

If the "real" output cap is 1000, we could tell job-list to return upwards of 1001 entries. Just taking the risk that could result in more than expected search time.

flux-jobs could output just the first 1000 and now we have a "flag" that indicates we have truncated?

[grondo]

If the "real" output cap is 1000, we could tell job-list to return upwards of 1001 entries.

Similar to this, but don't return the 1001th entry - instead set a truncated flag?

But then again, if we reached the output count of args.count .... the odds are there would be more anyways. So the change to jobs-list is unnecessary?

Yeah, you may have missed this part of my comment:

In fact, with the above approach, the feature could live in flux-jobs.py alone and we don't need to touch job-list.