add simple pagination to the ListSwaps command

[kornpow]

This commit adds the ability to set a max_swaps and an index_offset flag to the ListSwaps command. These new fields are applied AFTER the initial filtering step. The response also now passes additional information about the last_index_offset count and total_filtered_swaps, which can be used together with the --max_swaps and index_offset flags to iterate over your set of swaps in chunks.

$ loop listswaps --loop_out_only --max_swaps 1 --index_offset 3

{
    "swaps":  [
        {
            "amt":  "3000000",
            "id":  "xxx",
            "id_bytes":  "xxx",
            "type":  "LOOP_OUT",
            "state":  "SUCCESS",
            "failure_reason":  "FAILURE_REASON_NONE",
            "initiation_time":  "1743479436945030630",
            "last_update_time":  "1743487638197810121",
            "htlc_address":  "xxx",
            "htlc_address_p2wsh":  "",
            "htlc_address_p2tr":  "xxx",
            "cost_server":  "3043",
            "cost_onchain":  "120",
            "cost_offchain":  "7680",
            "last_hop":  "",
            "outgoing_chan_set":  [],
            "label":  "",
            "asset_info":  null
        }
    ],
    "last_index_offset":  3,
    "total_filtered_swaps":  4
}

Pull Request Checklist

• Update release_notes.md if your PR contains major features, breaking changes or bugfixes

Closes issues

This will close #797

[kornpow]

I'm going to start looking at some tests now, but wanted to open the PR to get some initial feedback on the approach first.

[kornpow]

Added the test which tests the behavior of both the filtering and the pagination.

One thing I am still trying to decide on is when starting startIndex to an out of bounds number. Currently it'll look like this:

swaps: [] 
first_index_offset:5 
last_index_offset:2
total_filtered_swaps:3

Maybe return an error instead of just having weird behavior?

[bhandras]

Concept ACK! Thank you @kornpow, nice addition! Left a few comments.

[kornpow]

Updated the response a bit. Also, now we only unmarshal the swaps we intend to return!

[starius]

swaps: [] 
first_index_offset:5 
last_index_offset:2
total_filtered_swaps:3

I can't find where first_index_offset is set. I think we should add this field, to be consistent with LND. E.g. lncli listinvoices returns both first_index_offset and last_index_offset.

[starius]

Nice improvement! 🎉

My main comment is that the order in the map is not stable and offsets themselves are not stable. I propose to use a timestamp instead of an offset and to use an opaque pagination tokens for future compatibility. Also to order the swaps by the initiation time to make the order predictable.

[kornpow]

swaps: [] 
first_index_offset:5 
last_index_offset:2
total_filtered_swaps:3

I can't find where first_index_offset is set. I think we should add this field, to be consistent with LND. E.g. lncli listinvoices returns both first_index_offset and last_index_offset.

I have returned first_index_offset to the response. I forgot that I copied the LND api response initially. But then I realized that first_index_offset in the response always equals index_offset in request, so I removed it, but it is useful to have so added it back.

[kornpow]

The last test case I am not super satisfied with is:
fetch all swaps set start index out of bounds
In my test case response:
first_index = 5
last_index = 0
Having first be after last seems wrong, so maybe I should throw an error instead?

[starius]

What do you think about switching from indices to timestamps as page identifiers? Timestamps do not depend on filtering, so this would work better in case some swaps appear or disappear in the middle of iteration.

[kornpow]

What do you think about switching from indices to timestamps as page identifiers? Timestamps do not depend on filtering, so this would work better in case some swaps appear or disappear in the middle of iteration.

Similar to the API for listpayments in LND? (This LND api has both filtering on creation_date and the index based pagination that is currently implemented)

kornpow@kornpow:~/Documents/repos/loop listswaps_paginate$ lncli listpayments -h
   --creation_date_start value  timestamp in seconds, if set, filter payments with creation date greater than or equal to it (default: 0)
   --creation_date_end value    timestamp in seconds, if set, filter payments with creation date less than or equal to it (default: 0)

Timestamps do depend on filtering though, its just filtering based on a timestamp?

[starius]

I suggest replacing --index_offset with --initiation-time-after.
The API would return a "last_initiation_time" field in the response, which the user could then pass back as the value for --initiation-time-after. The API would return only swaps initiated after (>) that timestamp.
The page size would still be controlled by the --max_swaps parameter (i.e., the number of swaps to return).

Unlike indices, timestamps remain stable even if some swaps are added or removed from the dataset. Index shifts can lead to duplicates being returned. Using timestamps avoids that issue.

[kornpow]

I did something a little different just to try it out. I can also move to something closer to what you specified @starius if you like.

I kept the paging the same, but added a Nanosecond unix timestamp fields to the filter object. So now we can filter on start/end time.

Since we are ordering based on InitiatiationTime, if we specify start/end time, we can safely page through.

$ loop listswaps \
	--start_time=1729000000000000000
	--end_time=1749012176223580686
	--max_swaps=2
	--index_offset=1

[starius]

start_time could also work, but we need to add a field like next_start_time to the response. So the user could take the value from that field and put into start_time. The value in that field should be max(swap.initiation_time for swap in swaps) + 1.

I think, end_time, index_offset, first_index_offset, and last_index_offset are not really needed. Having two ways to iterate would be confusing.

[kornpow]

Here is the current state. It is getting closer. I appreciate your feedback @starius on the API design. You've definitely helped make it a lot simpler.

Here is an example of paging through.

$ loop listswaps --start_time_ns=0

{
    "swaps":  [
        {
...
            "initiation_time": "1729012176223580685",
...
        },
        {
...
            "initiation_time": "1743457082654026082",
...
        },
        {
...
            "initiation_time": "1743457172366700842",
...
        },
        {
...
            "initiation_time": "1743479436945030630",
...
        }
    ],
    "next_start_time": "0"
}

loop listswaps --start_time_ns=0 --max_swaps=2

{
    "swaps": [
        {
...
            "initiation_time": "1729012176223580685",
...
        },
        {
...
            "initiation_time": "1743457082654026082",
...
        }
    ],
    "next_start_time": "1743457082654026083"

listswaps --start_time_ns=1743457082654026083 --max_swaps=2

{
    "swaps": [
        {
...
            "initiation_time": "1743457172366700842",
...
        },
        {
...
            "initiation_time": "1743479436945030630",
...
        }
    ],
    "next_start_time": "0"
}

[starius]

LGTM, pending comments 🏆

[starius]

LGTM!
Added few comments.

[lightninglabs-deploy]

@bhandras: review reminder

[kornpow]

I'm sorry for the delay, and all the churn. It should now properly pass the lint test.

[bhandras]

LGTM, nice work @kornpow 🎉