Add keyset pagination options

[pwlandoll]

This is one possible implementation of keyset-based pagination, as suggested in #815.

The basic flow of offset-based pagination is that the client requests the first page, and the server returns information about the page in headers. The flow for keyset-based pagination differs in that the information for building a request for the next page is not returned in pieces, but in a pre-formatted URL in the Link header. The documentation recommends using that link to make the request, which doesn't seem like it will work cleanly with the existing code here. In particular, the link includes a number of query parameters that differ among endpoints that support keyset-based pagination.

So, the changes here provide a way to directly use all of the query parameters returned in the header when building the next request. ListOptions now includes fields that are shared among all keyset-based paginated endpoints. Other required parameters for paginated endpoints can be added by means of a RequestOptionFunc, for which there is a builder function included.

An alternative would be to have ListOptions enumerate each possible query parameter that could be included in a "next Link" header, and return them in the Response struct in the same way as existing header values. This would mean that the end-user code for each endpoint using this pagination might look different, i.e. there would be no single way to do all keyset-based paginated endpoints. But, it would make the code less opaque, and avoid using a "magic black box function" to add RequestOptionFuncs.

This PR is mostly a suggestion/WIP to get the conversation moving about how this package might support keyset-based paginated endpoints, so please let me know how it looks!

[stanhu]

Minor question: I wonder if this field should be omitted, so that BuildKeysetPaginatedRequestOptionFunc can just take in the NextLink and lazily parse it. That might clean up the interface a little so that clients don't need to think about how to distinguish NextLink and NextLinkParameters.

[pwlandoll]

I think that's a good idea. It seems like the goal of the library right now is more "provide a clean interface without being too opinionated" rather than "provide as little as possible" or "all batteries included." I'll see if I can streamline that a little better.

[stanhu]

Thanks for doing this!

This looks like a pretty good approach to me. I just had one minor point for consideration.

[stanhu]

@svanharmelen Would you mind looking at this? We noticed that on GitLab.com there are clients such as gitlab-ci-pipelines-exporter that are iterating through a large number of CI jobs with offset pagination, when keyset pagination should be used.

[stanhu]

Thanks, this looks good to me!

@svanharmelen Could you review/merge?

[svanharmelen]

I'll have a look at this one tomorrow or Sunday... Looks like something we can get merged 👍🏻

[svanharmelen]

I'll take it... Thanks @pwlandoll 👍🏻

[pwlandoll]

Great, thanks! I was able to pull the latest version and run code like the included example.