Paginate rpc commands that return long lists

[mb300sd]

Mentioned briefly in #4305, now coming up again over on RTL ( Ride-The-Lightning/RTL#737 )

lnd has an index/count option, something similar would be very nice to have on c-lightning.

[whitslack]

Concept NACK.

Pagination against a live data source is inherently racy. If you need to paginate a large JSON array, capture the entire array into a file and use a JSON manipulation tool (I highly recommend jq) to select subranges from it as you desire.

[mb300sd]

Maybe a before x/after y timestamp option would be better then, basically anything to avoid having to return millions of results every time listforwards is called.

[whitslack]

@mb300sd: Hint: you can also just query the database directly. And if you're using PostgreSQL as your backend, you can add a trigger that notifies you of any insertions or updates to the forwarded_payments table as they happen.

[zerofeerouting]

I would really like to emphasize the importance of this. Especially with listforwards we're talking hundreds of thousands, if not millions of entries. Would also be great being able to filter results doing start_date=-2d or end_date=-12h.

[cdecker]

I think @rustyrussell was going to design a generic pagination system that could leverage the DB that is backing most of the queries to limit results, order by field and offset based on the ordering column. Maybe he can share the design in a quick doc and maybe we can get this unblocked a bit :-)

[sangbida]

This issue is being closed as part of issue cleanup.

Reason: About deprecated feature

If you believe this issue should remain open, please comment and we can reopen it.