#3403 Paginating /api/0/projects by default
Closed: Fixed Opened by bkabrda.

Hi, I have a Pagure instance with more than 44k projects and when someone requests /api/0/projects, the whole server dies because it runs out of memory. I'd like to propose that this API page would be paginated by default and using page=0 would return non-paginated version. Does this make sense? I can send a PR if so.


I wonder if we should make it a configuration key.
Changing this would be a non-backward compatible change, so having an option to enforce it or not could help.

On the other hand, the next release will be a 5.0 and already contains some non-backward compatible changes (changes in default configuration), so maybe it's the right time to do this.

What do you think?

Yeah, this is a backward incompatible change for sure. What I'm wondering is if you'd remove the configuration key eventually. If so, I think just doing the change without the key makes sense for 5.0 if it already contains other incompatible changes.

If your intention would be to keep the key forever, then obviously that's the way to do it.

My opinion is that it'd be weird to have different Pagure instances return different format of API response when based on the same upstream version, so I'd personally go with just breaking the API. Alternatively, you could introduce /api/1 where you'd make all responses that contain list of entities paginated by default.

where you'd make all responses that contain list of entities paginated by default.

Do we have more of these endpoints?

From what I can see, lists of entities are returned by:

  • /{repo}/pull-requests (no pagination now)
  • /groups
  • /{repo}/issues
  • /user/{username}/issues
  • /user/{username}/requests/filed
  • /user/{username}/requests/actionable

I'm not saying that responses by these endpoints would get as big as the /projects one, but I think it's generally good to paginate responses that contain lists of complex entities (while I also think it's ok to not paginate stuff like list of tags or branches).

Well, the /groups one is also to simple to need to be paginated, so disregard that one.

but I think it's generally good to paginate responses that contain lists of complex entities (while I also think it's ok to not paginate stuff like list of tags or branches).

If you have time, let's get this in for 5.0 and break backward compatibility on these endpoints but for the better.

PR for the first part about paginating /api/0/projects per default is here:
https://pagure.io/pagure/pull-request/3494

Here's another PR to paginate /api/0//issues:
https://pagure.io/pagure/pull-request/3504

More pagination in https://pagure.io/pagure/pull-request/3512

And I think the final piece is at #3515

All we'll need is to go over the API docs and ensure they are up to date before we close this ticket :)

docs update is here: https://pagure.io/pagure/pull-request/3521

Metadata Update from @karsten:
- Issue assigned to karsten

everything got merged into master, closing

Metadata Update from @karsten:
- Issue close_status updated to: Fixed

Metadata