Pagination helper for sync scripts

[bastienbeurier]

No description provided.

[jstorm31]

Github's octokit has nice interface for pagination including iterators. That would really improve the DX 📈

for await (const response of octokit.paginate.iterator(
  octokit.rest.issues.listForRepo,
  {
    owner: "octokit",
    repo: "rest.js",
  },
)) {
  // do whatever you want with each response, break out of the loop, etc.
}

[rguldener]

Pretty elegant approach, thanks for sharing!
I like that it lets us define a default pagination strategy for the API. But I wonder of the structure could be further simplified?

If we want to keep it wrapping we could do something like this:

const requestParams = {
   endpoint: '/xyz',
   params: { 'abc': 'def' }
};
for await (const response of nango.paginate(nango.get(requestParams))) {
  // do whatever you want with each response, break out of the loop, etc.
}

But this might make changing the pagination strategy for a specific API request more difficult.

Another approach could be to build it entirely into nango.get, which could also act as an iterator if paginate is set to true:

const requestParams = {
   endpoint: '/xyz',
   params: { 'abc': 'def' },
   paginate: true
};
for await (const response of nango.get(requestParams)) {
  // do whatever you want with each response, break out of the loop, etc.
}

This last one would have the advantage that overwriting the pagination strategy for a specific endpoint is pretty easy, e.g.

const requestParams = {
   endpoint: '/xyz',
   params: { 'abc': 'def' },
   paginate: {
       strategy: nango.paginationStrategy.Cursor,
       nextCursorParamName: 'abc',
       nextCursorResponsePath: 'pages.next'
   }
};
for await (const response of nango.get(requestParams)) {
  // do whatever you want with each response, break out of the loop, etc.
}

I think this would be important because many APIs have different pagination strategies for some of their endpoints.

[jstorm31]

You're right there's a need for different pagination strategies (page, cursor,..), which Github doesn't need. Personaly, I prefer the composition approach, but without a strong opinion.

You could also add pagination params to the paginate function

const requestParams = {
   endpoint: '/xyz',
   params: { 'abc': 'def' }
};
for await (const response of nango.paginate(nango.get(requestParams), {
          strategy: nango.paginationStrategy.Cursor,
           nextCursorParamName: 'abc',
           nextCursorResponsePath: 'pages.next'
})) {
  // do whatever you want with each response, break out of the loop, etc.
}