FQL Paginate - get items in between two items and some offset

Help
,
1

I want to use Paginate such that given two points, I want to grab all items starting from that point up to (inclusive of, meaning, it should also get the target point) the target point.

Imagine that there are 10,000 documents in a collection called messages. I have two points, A – the starting point, and B the last point, what I need to do is get all items starting at point A + 1 to point B. I tried using Paginate with before and after but it returned an error.

Paginate(
  Match(
    Index("messages"),
    "295664821370618375",
    "295664230831489541"
  ),
  {
    size: 20,
    before: [
      "2021-05-22T12:18:46.120288Z",
      "299291982379876871"
    ],
    after: [
      "2021-06-02T14:12:15.101330Z",
      "300295688333296129"
    ],
  }
);

Error: [
  {
    "position": [],
    "code": "invalid expression",
    "description": "No form/function found, or invalid argument keys: { paginate, size, before, after }."
  }
]

This is the definition of the index messages:

CreateIndex({
  name: 'messages',
  source: Collection('inbox'),
  terms: [
    { field: ['data', 'senderId'] },
    { field: ['data', 'recipientId'] }
  ],
  values: [
    { field: ['data', 'createdAt'], reverse: true },
    { field: ['ref'] }
  ]
})

EDIT 1:

Additionally, what if I want to say "get all documents from point A + 1 to point B + 5", notice the +5 here, it’s the offset, basically get A + 1 to B and then 5 more items after it.

2

I have found the answer to the first problem: Range | Fauna Documentation

The caveat is that paginate has 100,000 max. So you might need to consider that it may fall beyond that limit.

I’m still figuring out the offset problem.

3

Just like Match, Range returns a Set that you can page through. You can use Paginate with the size option as well as before OR after cursors while using Range same as you would a more simple query.

The basic formula for pagination with cursors is

Paginate(
  SOME_SET,
  { 
    size: SIZE,

    // cannot use both before and after at the same time
    after: NEXT_CURSOR       // only if paging forward
    before:  PREVIOUS_CURSOR // only if paging in reverse
  }
)

What I called SOME_SET can be whatever you’re querying, whether it is

Documents(Collection("inbox"))

// OR

Match(Index("messages"),  "295664821370618375",  "295664230831489541")

// OR

Range(
  Match(Index("messages"),  "295664821370618375",  "295664230831489541"),
  ["2021-05-22T12:18:46.120288Z"],
  ["2021-06-02T14:12:15.101330Z"]
)

// OR whatever!

Example

So if you run this query:

NOTE: the dates shortened for convenience. No magic here, just hoping to mock the data quickly in a relevant way as I go.

Map(
  Paginate(
    Range(
      Match(Index("messages"),  "295664821370618375",  "295664230831489541"),
      ["2021-05-21"],
      ["2021-06-02"]
    ),
    { size: 2 }
  ),
  Lambda(['createdAt', 'ref'], Get(Var('ref')))
)

// RESULT
{
  after: ["2021-05-23", "103"],
  data: [
    {
      ref: Ref(Collection("inbox"), "101"),
      ts: 1621815970440000,
      data: { createdAt: "2021-05-21", /* ... */ }
    },
    {
      ref: Ref(Collection("inbox"), "102"),
      ts: 1621815970440000,
      data: { createdAt: "2021-05-22", /* ... */ }
    }
  ]
}

NOTE: When using Map on a Page, the lambda is applied to the data, but the cursors stick around!!

Then the next query you make can be

Map(
  Paginate(
    // The queried Set has not changed at all!
    Range(
      Match(Index("messages"),  "295664821370618375",  "295664230831489541"),
      ["2021-05-21"],
      ["2021-06-02"]
    ),
    // Only thing we did was add the cursor here
    { size: 2, after: ["2021-05-23", "103"] }
  ),
  Lambda(['createdAt', 'ref'], Get(Var('ref')))
)

// RESULT
{
  before: ["2021-05-23", "103"],  // before cursor shows up.  same as previous after cursor
  after: ["2021-05-25", "105"],   // after cursor moves to next page
  data: [
    {
      ref: Ref(Collection("inbox"), "103"),
      ts: 1621815970440000,
      data: { createdAt: "2021-05-23", /* ... */ }
    },
    {
      ref: Ref(Collection("inbox"), "104"),
      ts: 1621815970440000,
      data: { createdAt: "2021-05-24", /* ... */ }
    }
  ]
}

Other notes

There are some things to watch out for with complex set operations.

There is a Drop function, but it’s applied after Paginate. That means it will fetch all of the data but then literally drop some of it. Since you still have to get the all of the data, you’re still limited to the max size of Paginate (100000)

The JS driver has some pagination helpers make fetching multiple pages easier.