FQL Paginate - get items in between two items and some offset

Help
,
3

Just like Match, Range returns a Set that you can page through. You can use Paginate with the size option as well as before OR after cursors while using Range same as you would a more simple query.

The basic formula for pagination with cursors is

Paginate(
  SOME_SET,
  { 
    size: SIZE,

    // cannot use both before and after at the same time
    after: NEXT_CURSOR       // only if paging forward
    before:  PREVIOUS_CURSOR // only if paging in reverse
  }
)

What I called SOME_SET can be whatever you’re querying, whether it is

Documents(Collection("inbox"))

// OR

Match(Index("messages"),  "295664821370618375",  "295664230831489541")

// OR

Range(
  Match(Index("messages"),  "295664821370618375",  "295664230831489541"),
  ["2021-05-22T12:18:46.120288Z"],
  ["2021-06-02T14:12:15.101330Z"]
)

// OR whatever!

Example

So if you run this query:

NOTE: the dates shortened for convenience. No magic here, just hoping to mock the data quickly in a relevant way as I go.

Map(
  Paginate(
    Range(
      Match(Index("messages"),  "295664821370618375",  "295664230831489541"),
      ["2021-05-21"],
      ["2021-06-02"]
    ),
    { size: 2 }
  ),
  Lambda(['createdAt', 'ref'], Get(Var('ref')))
)

// RESULT
{
  after: ["2021-05-23", "103"],
  data: [
    {
      ref: Ref(Collection("inbox"), "101"),
      ts: 1621815970440000,
      data: { createdAt: "2021-05-21", /* ... */ }
    },
    {
      ref: Ref(Collection("inbox"), "102"),
      ts: 1621815970440000,
      data: { createdAt: "2021-05-22", /* ... */ }
    }
  ]
}

NOTE: When using Map on a Page, the lambda is applied to the data, but the cursors stick around!!

Then the next query you make can be

Map(
  Paginate(
    // The queried Set has not changed at all!
    Range(
      Match(Index("messages"),  "295664821370618375",  "295664230831489541"),
      ["2021-05-21"],
      ["2021-06-02"]
    ),
    // Only thing we did was add the cursor here
    { size: 2, after: ["2021-05-23", "103"] }
  ),
  Lambda(['createdAt', 'ref'], Get(Var('ref')))
)

// RESULT
{
  before: ["2021-05-23", "103"],  // before cursor shows up.  same as previous after cursor
  after: ["2021-05-25", "105"],   // after cursor moves to next page
  data: [
    {
      ref: Ref(Collection("inbox"), "103"),
      ts: 1621815970440000,
      data: { createdAt: "2021-05-23", /* ... */ }
    },
    {
      ref: Ref(Collection("inbox"), "104"),
      ts: 1621815970440000,
      data: { createdAt: "2021-05-24", /* ... */ }
    }
  ]
}

Other notes

There are some things to watch out for with complex set operations.

There is a Drop function, but it’s applied after Paginate. That means it will fetch all of the data but then literally drop some of it. Since you still have to get the all of the data, you’re still limited to the max size of Paginate (100000)

The JS driver has some pagination helpers make fetching multiple pages easier.