Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: use keyset pagination for retrieving project CI jobs #744

Merged
merged 2 commits into from
Nov 29, 2023
Merged

feat: use keyset pagination for retrieving project CI jobs #744

merged 2 commits into from
Nov 29, 2023

Conversation

stanhu
Copy link
Contributor

@stanhu stanhu commented Nov 13, 2023 •
edited
Loading

With xanzy/go-gitlab#1827, go-gitlab now supports keyset pagination, which is much more efficient for iterating through many pages of data. This commit switches the GitLab CI
/api/v4/projects/:id/jobs API to use keyset pagination.

The other pipeline API endpoints need keyset pagination support: https://gitlab.com/gitlab-org/gitlab/-/issues/431632

@dependabot
build(deps): bump github.com/xanzy/go-gitlab from 0.92.3 to 0.94.0
66bcb68 
Bumps [github.com/xanzy/go-gitlab](https://github.com/xanzy/go-gitlab) from 0.92.3 to 0.94.0.
- [Changelog](https://github.com/xanzy/go-gitlab/blob/main/releases_test.go)
- [Commits](xanzy/go-gitlab@v0.92.3...v0.94.0)

---
updated-dependencies:
- dependency-name: github.com/xanzy/go-gitlab
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
@stanhu stanhu marked this pull request as draft November 13, 2023 19:44
@stanhu
feat: use keyset pagination for retrieving project CI jobs
7ac3e79 
With xanzy/go-gitlab#1827, go-gitlab now
supports keyset pagination, which is much more efficient for iterating
through many pages of data.  This commit switches the GitLab CI
`/api/v4/projects/:id/jobs` API to use keyset pagination.

The other pipeline API endpoints need keyset pagination support:
https://gitlab.com/gitlab-org/gitlab/-/issues/431632
@stanhu stanhu force-pushed the sh-use-keyset-pagination-jobs branch from 050ea5c to 7ac3e79 Compare November 13, 2023 20:48
@stanhu stanhu changed the title feat: use keyset pagination for retrieving CI jobs feat: use keyset pagination for retrieving project CI jobs Nov 13, 2023
@stanhu stanhu marked this pull request as ready for review November 13, 2023 20:54
@stanhu
Copy link
Contributor Author

stanhu commented Nov 13, 2023

@mvisonneau Could you review this? We're seeing a high amount of traffic on GitLab.com using offset pagination for GET /api/v4/projects/:id/jobs from this project, and I think this change would help.

@mvisonneau
Copy link
Owner

Thanks a ton @stanhu! Do you know if there are additional calls that could benefit from it?

@mvisonneau mvisonneau self-assigned this Nov 29, 2023
@mvisonneau
Copy link
Owner

also, do you know from which GitLab's version this feature is available?

@coveralls
Copy link

Coverage Status

coverage: 75.187% (-0.04%) from 75.225%
when pulling 7ac3e79 on stanhu:sh-use-keyset-pagination-jobs
into 1a98e3a on mvisonneau:main.

@mvisonneau mvisonneau merged commit c4d9649 into mvisonneau:main Nov 29, 2023
4 checks passed
@stanhu
Copy link
Contributor Author

stanhu commented Nov 29, 2023

also, do you know from which GitLab's version this feature is available?

Good question. According to the documentation, GitLab 15.9 is required. Perhaps we need a way to switch between pagination modes, depending on what version is being used?

Do you know if there are additional calls that could benefit from it?

It looks like this exporter uses:

API description Supports keyset pagination?
GetProjectBranches N
ListEnvironments N
GetProjectPipelines N
ListUserProjects Y
ListGroupProjects N
ListProjects Y
ListTags N

@mvisonneau
Copy link
Owner

Perhaps we need a way to switch between pagination modes, depending on what version is being used?

yeah, perhaps we could dynamically enable it depending on the version we can preliminary fetch (& most likely cache) from the Metadata API, wdyt?

I believe we can do that in a follow up PR introducing the feature as well for ListProjects and ListUserProjects then 🤷

stanhu added a commit to stanhu/gitlab-ci-pipelines-exporter that referenced this pull request Dec 1, 2023
@stanhu
fix: use keyset pagination only when it is supported
b1f6d23 
mvisonneau#744
switched to keyset pagination for the `/api/v4/projects/:id/jobs` API
endpoint, but this feature is only available in GitLab 15.9 and later
(https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs).

This commit retrieves the instance metadata at startup to
determine whether keyset pagination is available. If the metadata
is not available or returns a version < 15.9, offset pagination will
be used.
@stanhu stanhu mentioned this pull request Dec 1, 2023
Merged
stanhu added a commit to stanhu/gitlab-ci-pipelines-exporter that referenced this pull request Dec 1, 2023
@stanhu
fix: use keyset pagination only when it is supported
daba1a1 
mvisonneau#744
switched to keyset pagination for the `/api/v4/projects/:id/jobs` API
endpoint, but this feature is only available in GitLab 15.9 and later
(https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs).

This commit retrieves the instance metadata at startup to
determine whether keyset pagination is available. If the metadata
is not available or returns a version < 15.9, offset pagination will
be used.
stanhu added a commit to stanhu/gitlab-ci-pipelines-exporter that referenced this pull request Dec 1, 2023
@stanhu
fix: use keyset pagination only when it is supported
8e62767 
mvisonneau#744
switched to keyset pagination for the `/api/v4/projects/:id/jobs` API
endpoint, but this feature is only available in GitLab 15.9 and later
(https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs).

This commit retrieves the instance metadata at startup to
determine whether keyset pagination is available. If the metadata
is not available or returns a version < 15.9, offset pagination will
be used.
@stanhu
Copy link
Contributor Author

stanhu commented Dec 1, 2023

yeah, perhaps we could dynamically enable it depending on the version we can preliminary fetch (& most likely cache) from the Metadata API, wdyt?

Sounds good! See #755.

stanhu added a commit to stanhu/gitlab-ci-pipelines-exporter that referenced this pull request Dec 1, 2023
@stanhu
fix: use keyset pagination only when it is supported
430154c 
mvisonneau#744
switched to keyset pagination for the `/api/v4/projects/:id/jobs` API
endpoint, but this feature is only available in GitLab 15.9 and later
(https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs).

This commit retrieves the instance metadata at startup to
determine whether keyset pagination is available. If the metadata
is not available or returns a version < 15.9, offset pagination will
be used.
stanhu added a commit to stanhu/gitlab-ci-pipelines-exporter that referenced this pull request Dec 1, 2023
@stanhu
fix: use keyset pagination only when it is supported
6c0ace3 
mvisonneau#744
switched to keyset pagination for the `/api/v4/projects/:id/jobs` API
endpoint, but this feature is only available in GitLab 15.9 and later
(https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs).

This commit retrieves the instance metadata at startup to
determine whether keyset pagination is available. If the metadata
is not available or returns a version < 15.9, offset pagination will
be used.
stanhu added a commit to stanhu/gitlab-ci-pipelines-exporter that referenced this pull request Dec 1, 2023
@stanhu
fix: use keyset pagination only when it is supported
45dc3ed 
mvisonneau#744
switched to keyset pagination for the `/api/v4/projects/:id/jobs` API
endpoint, but this feature is only available in GitLab 15.9 and later
(https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs).

This commit retrieves the instance metadata at startup to
determine whether keyset pagination is available. If the metadata
is not available or returns a version < 15.9, offset pagination will
be used.
mvisonneau pushed a commit that referenced this pull request Dec 5, 2023
@stanhu
fix: use keyset pagination only when it is supported (#755)
0687efd 
* fix: use keyset pagination only when it is supported

#744
switched to keyset pagination for the `/api/v4/projects/:id/jobs` API
endpoint, but this feature is only available in GitLab 15.9 and later
(https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs).

This commit retrieves the instance metadata at startup to
determine whether keyset pagination is available. If the metadata
is not available or returns a version < 15.9, offset pagination will
be used.

* chore: Use golang.org/x/mod/semver for version parsing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mvisonneau mvisonneau mvisonneau approved these changes

Assignees

@mvisonneau mvisonneau

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@stanhu @mvisonneau @coveralls