Paginate is not returning all documents

Help
1

Hello everyone !

I’ve used a query :

q.Paginate(
  q.Intersection(
    q.Match(q.Index('exercisesByManualValidation'), true),
    q.Join(
      q.Join(
        q.Match(
          q.Index('lessonsByAuthor'),
          q.Ref(q.Collection('users'), '255160402551243274'),
        ),
        q.Index('skillsBlocksByLesson'),
      ),
      q.Index('exercisesBySkillBlock'),
    ),
  )
)

Which returns 10 results (without after/before, so end of list) :

{
  data: [
    Ref(Collection("exercises"), "264133659798274580"),
    Ref(Collection("exercises"), "264135764300792339"),
    Ref(Collection("exercises"), "264136902673695251"),
    Ref(Collection("exercises"), "264142443312579091"),
    Ref(Collection("exercises"), "264151464950104595"),
    Ref(Collection("exercises"), "264240678558499346"),
    Ref(Collection("exercises"), "271833694434492941"),
    Ref(Collection("exercises"), "287769999151464972"),
    Ref(Collection("exercises"), "287771006711366157"),
    Ref(Collection("exercises"), "287771816320041480")
  ]
}

But I knew there was some results missing, so I used the Count on the same query and I get 11 results. (The missing one). The only way to get it is to use the { size: 11, (or 999999 for more...) },
and now I get all 11 results.

{
  data: [
    Ref(Collection("exercises"), "264133659798274580"),
    Ref(Collection("exercises"), "264135764300792339"),
    Ref(Collection("exercises"), "264136902673695251"),
    Ref(Collection("exercises"), "264142443312579091"),
    Ref(Collection("exercises"), "264151464950104595"),
    Ref(Collection("exercises"), "264240678558499346"),
    Ref(Collection("exercises"), "271833694434492941"),
    Ref(Collection("exercises"), "287769999151464972"),
    Ref(Collection("exercises"), "287771006711366157"),
    Ref(Collection("exercises"), "287771816320041480"),
    Ref(Collection("exercises"), "293778802226496013")
  ]
}

Any idea about this issue ?

2

TL;DR: This is a known issue with the way that Fauna handles arbitrary sets. See: Known issues | Fauna Documentation

A more complete answer:

As a shared resource, Fauna tries to avoid performing full table scans wherever possible; full table scans could block subsequent queries until completed, and could use significantly more resources than actually needed. That’s why Fauna’s indexes place index entries in storage in sorted order. When you need to investigate a range of values covered by an index, the result can be computed quickly and efficiently.

When you combine Join and/or Intersection with indexes, the resulting set is, effectively, unbounded.

In these scenarios, Fauna uses an estimator to determine how many reads to perform for set operations (rather than performing a full table scan). The estimator uses the current Pagination page size, plus a small multiplier, to inform its read prediction. When the estimator’s prediction doesn’t reach far enough into the source set to gather all results, that’s when entries appear to be “missing”.

The simplest workaround is to increase the page size in your Pagination call. Add { size: 1000 } as the last parameter to Paginate to see if that makes a difference. Note that the maximum page size is 100,000. If that’s not large enough to produce the correct result, you might need to adjust your strategy to place intermediate results in a collection, and then operate on those.

We will be working towards a solution for this problem in a future release.

1 Like
3

Clean answer, thanks a lot ! Atm I’m using the size attribute :wink:

4

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.