Confusing cursor structure and overlapping page data when paginating

Help
,
1

This is the code and the results I got in my dashboard’s shell:

q.Let(
  {
    endDate: q.ToDate('2020-12-31'),
    startDate: q.ToDate('2020-01-01'),
    ownerRef: q.Ref(q.Collection('Owner'), "254598257589617153"),
    propertiesSetRef: q.Match(
      q.Index('owner_properties_by_owner'),
      q.Var('ownerRef')
    ),
    startTimeSeconds: q.ToSeconds(q.ToTime(q.Var('startDate'))),
    endTimeSeconds: q.ToSeconds(q.ToTime(q.Var('endDate'))),
    allMonthlyEntriesForOwnerPropertiesSetRanged: q.Join(
      q.Var('propertiesSetRef'),
      q.Lambda(
        'propertyRef',
        q.Range(
          q.Match(
            q.Index('property_entries_by_date_seconds'),
            q.Var('propertyRef')
          ),
          q.Var('startTimeSeconds'),
          q.Var('endTimeSeconds')
        )
      )
    ),
  },
  Paginate(q.Var('allMonthlyEntriesForOwnerPropertiesSetRanged'), {size: 50})
)

{
  after: [
    1588291200,
    Ref(Collection("MonthlyPropertyEntry"), "266869347503309324"),
    Ref(Collection("MonthlyPropertyEntry"), "266869347503309324")
  ],
  data: [
    [1577836800, Ref(Collection("MonthlyPropertyEntry"), "259752406673261075")],
    [1577836800, Ref(Collection("MonthlyPropertyEntry"), "261386505568649747")],
    [1577836800, Ref(Collection("MonthlyPropertyEntry"), "266186908903670273")],
    [1577836800, Ref(Collection("MonthlyPropertyEntry"), "266187437101810177")],
    [1577836800, Ref(Collection("MonthlyPropertyEntry"), "266234099111297546")],
    [1577836800, Ref(Collection("MonthlyPropertyEntry"), "266235052866667017")],
    [1577836800, Ref(Collection("MonthlyPropertyEntry"), "266637456119431689")],
    [1577836800, Ref(Collection("MonthlyPropertyEntry"), "266747902574985740")],
    [1577836800, Ref(Collection("MonthlyPropertyEntry"), "266869421684818445")],
    [1577836800, Ref(Collection("MonthlyPropertyEntry"), "266871226750730764")],
    [1577836800, Ref(Collection("MonthlyPropertyEntry"), "267357720771822092")],
    [1580515200, Ref(Collection("MonthlyPropertyEntry"), "266185750971679233")],
    [1580515200, Ref(Collection("MonthlyPropertyEntry"), "266186988566086154")],
    [1580515200, Ref(Collection("MonthlyPropertyEntry"), "266231730036277769")],
    [1580515200, Ref(Collection("MonthlyPropertyEntry"), "266234235721875978")],
    [1580515200, Ref(Collection("MonthlyPropertyEntry"), "266235118870331913")],
    [1580515200, Ref(Collection("MonthlyPropertyEntry"), "266638586754892289")],
    [1580515200, Ref(Collection("MonthlyPropertyEntry"), "266748080487924232")],
    [1580515200, Ref(Collection("MonthlyPropertyEntry"), "266749325899465229")],
    [1580515200, Ref(Collection("MonthlyPropertyEntry"), "266869599409013260")],
    [1580515200, Ref(Collection("MonthlyPropertyEntry"), "266871246500659725")],
    [1580515200, Ref(Collection("MonthlyPropertyEntry"), "267358253111837196")],
    [1583020800, Ref(Collection("MonthlyPropertyEntry"), "266185900572017161")],
    [1583020800, Ref(Collection("MonthlyPropertyEntry"), "266187085368525321")],
    [1583020800, Ref(Collection("MonthlyPropertyEntry"), "266232936275116553")],
    [1583020800, Ref(Collection("MonthlyPropertyEntry"), "266234755728540169")],
    [1583020800, Ref(Collection("MonthlyPropertyEntry"), "266235132654912009")],
    [1583020800, Ref(Collection("MonthlyPropertyEntry"), "266638925493174785")],
    [1583020800, Ref(Collection("MonthlyPropertyEntry"), "266748451801268748")],
    [1583020800, Ref(Collection("MonthlyPropertyEntry"), "266749641413886465")],
    [1583020800, Ref(Collection("MonthlyPropertyEntry"), "266870603120640525")],
    [1583020800, Ref(Collection("MonthlyPropertyEntry"), "266871266206548488")],
    [1583020800, Ref(Collection("MonthlyPropertyEntry"), "267358976704774669")],
    [1585699200, Ref(Collection("MonthlyPropertyEntry"), "266186818545779201")],
    [1585699200, Ref(Collection("MonthlyPropertyEntry"), "266187160626921994")],
    [1585699200, Ref(Collection("MonthlyPropertyEntry"), "266233165985612297")],
    [1585699200, Ref(Collection("MonthlyPropertyEntry"), "266234831642296841")],
    [1585699200, Ref(Collection("MonthlyPropertyEntry"), "266235159860216321")],
    [1585699200, Ref(Collection("MonthlyPropertyEntry"), "266639087868314122")],
    [1585699200, Ref(Collection("MonthlyPropertyEntry"), "266748746771989000")],
    [1585699200, Ref(Collection("MonthlyPropertyEntry"), "266750097058955777")],
    [1585699200, Ref(Collection("MonthlyPropertyEntry"), "266870736347464200")],
    [1585699200, Ref(Collection("MonthlyPropertyEntry"), "266871305754640908")],
    [1585699200, Ref(Collection("MonthlyPropertyEntry"), "267359203783344648")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266234943418401290")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266639242976821770")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266639951160934913")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266640330785292809")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266640791698407946")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266750499188900355")]
  ]
}

>> Time elapsed: 21ms

q.Let(
  {
    endDate: q.ToDate('2020-12-31'),
    startDate: q.ToDate('2020-01-01'),
    ownerRef: q.Ref(q.Collection('Owner'), "254598257589617153"),
    propertiesSetRef: q.Match(
      q.Index('owner_properties_by_owner'),
      q.Var('ownerRef')
    ),
    startTimeSeconds: q.ToSeconds(q.ToTime(q.Var('startDate'))),
    endTimeSeconds: q.ToSeconds(q.ToTime(q.Var('endDate'))),
    allMonthlyEntriesForOwnerPropertiesSetRanged: q.Join(
      q.Var('propertiesSetRef'),
      q.Lambda(
        'propertyRef',
        q.Range(
          q.Match(
            q.Index('property_entries_by_date_seconds'),
            q.Var('propertyRef')
          ),
          q.Var('startTimeSeconds'),
          q.Var('endTimeSeconds')
        )
      )
    ),
  },
  Paginate(q.Var('allMonthlyEntriesForOwnerPropertiesSetRanged'), {size: 50, after: [
    1588291200,
    Ref(Collection("MonthlyPropertyEntry"), "266869347503309324")
  ]})
)

{
  before: [
    1588291200,
    Ref(Collection("MonthlyPropertyEntry"), "266869347503309324")
  ],
  data: [
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266234943418401290")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266639242976821770")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266639951160934913")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266640330785292809")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266640791698407946")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266750499188900355")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266869347503309324")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266870856943141384")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "266871315675218445")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "267359449213043213")],
    [1588291200, Ref(Collection("MonthlyPropertyEntry"), "283234405925782028")],
    [1590969600, Ref(Collection("MonthlyPropertyEntry"), "267924060145451533")],
    [1590969600, Ref(Collection("MonthlyPropertyEntry"), "270270901336932876")],
    [1590969600, Ref(Collection("MonthlyPropertyEntry"), "281317518724825612")]
  ]
}

Basically what I’m doing is trying to paginate through the data, 50 results at a time. The above code is 2 queries, the first with 50 results, and the second with 14 results.

There are only 58 unique results.

I’m confused because the second query returns a set of overlapped results with the first query, and I don’t understand why.

I don’t understand why the first query’s resulting page has an after cursor with this structure either:

after: [
    1588291200,
    Ref(Collection("MonthlyPropertyEntry"), "266869347503309324"),
    Ref(Collection("MonthlyPropertyEntry"), "266869347503309324")
  ]

Why is the reference duplicated in the cursor? Why isn’t the cursor in the same format as the data in the set? Is it related to using join in my query?

Finally, if I use the above cursor in my second query instead of the one without the duplicated reference, then I do get the correct results (8 items with no overlap).

2

Hi @wallslide ,
I have opened a support ticket for this, but to close the loop here -

Why is the reference duplicated in the cursor? Why isn’t the cursor in the same format as the data in the set?

It appears that the index is created with an explicit ref in values , then a duplicate ref shows up in the after cursor.

Finally, if I use the above cursor in my second query instead of the one without the duplicated reference, then I do get the correct results (8 items with no overlap)

Yes, in this case, you will need to use the same cursor (with duplicate refs) to get accurate results.

I’ve filed an issue so our engineering team can take a closer look.

Thanks!

1 Like
3

Hi @wallslide

We have reviewed this issue, and wanted to let you know that this is expected behavior.

The after/before cursors are composed of the values specified in the index plus the indexed document id(ref).

In this case, it so happens that this index also covers the document ref, hence resulting in the document ref showing up twice in the cursor.

Thanks!

1 Like
4

Thanks for the update! I’m not sure how much of the discord conversation you saw or if you did see it at all. But part of the confusion was that the before and after cursors might have different sized arrays. For instance, the after cursor would look like how I pasted above, and the before cursor would only have two entries (missing the duplicated reference).

Is that expected behavior too? I’m trying to figure out how to deterministically build my own cursor without going through the work of creating a generalized cursor serialization tool.

Thanks!

5

I did see the discord conversation, and it was useful context for me, specially to understand how your indexes are defined.

But part of the confusion was that the before and after cursors might have different sized arrays.

It was my understanding (and I see the same in my testing) that you get different before and after cursors only if you alter the after cursor to remove the extra ref while paginating.

This part of your code -

},
  Paginate(q.Var('allMonthlyEntriesForOwnerPropertiesSetRanged'), {size: 50, after: [
    1588291200,
    Ref(Collection("MonthlyPropertyEntry"), "266869347503309324")
  ]})
)

Here, the after cursor has the second ref removed. If you use the cursor as is

after: [
    1588291200,
    Ref(Collection("MonthlyPropertyEntry"), "266869347503309324"),
    Ref(Collection("MonthlyPropertyEntry"), "266869347503309324")
  ]

then, I expect the before and after cursors to look the same. Is this different from what you are seeing?

2 Likes
6

You are correct! The mismatched cursors only occur if I use an incorrectly formed cursor. Thank you for your help!

2 Likes
7

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.