Paginate doesn't work as expected

Help
,
1

Hello,

There seems to be a problem with the Paginate function.

I have a Collection with 33 Documents.
I’m filtering through this Collection.
For this, I use the functions Paginate> Intersection> [Union, Match, Join].

Paginate Size= 64 - 3 Results

image
image2164×576 115 KB

Paginate Size= 70 - 17 Results

image
image2310×874 362 KB

Paginate Size= 1000 - 33 Results

image
image2060×1518 444 KB

Main Query

image
image2512×912 262 KB

How can Paginate work like that?
As I understand it, it seems that you are taking as many documents as they are worth and filtering them.
But even in such a case, since I have 64 fewer documents, all of them have to come.

Since I was creating queries with dart codes, I could not display all queries properly.
Sorry if it’s not clear.

2

@hasimyerlikaya You probably have lots of events on the documents in this collection. I will have to forensic your data to explain this behavior. I will reach out to you in a DM for user-specificinfo.

1 Like
3

Hi, @hasimyerlikaya I could not locate the query using all the below indexes. I must be doing something wrong with my filtering. Let us coordinate tomorrow morning. You could run your setup with Paginate Size:70 and I will try to find the query within that timeframe. Right now I am filtering the logs for last 24 hours and is quite challenging to find the query we want to troubleshoot.

"PeriodicIncomePaymentDateByWallet"
"PeriodicIncomePaymentRefByRef"
"PeriodicIncomePaymentByPaymentStatusAndWallet"
"PeriodicIncomePaymentAmountByWallet"
"PeriodicIncomePaymentByPaymentMethodAndWallet"
"PeriodicIncomePaymentByPaymentCardAndWallet"
"PeriodicIncomePaymentByPaymentAccountAndWallet"
"PeriodicIncomePaymentByRelevantTypeAndWallet"
"PeriodicIncomePaymentByPersonAndWallet"
"PeriodicIncomePaymentByCorporationAndWallet"
5

Hi, @Jay-Fauna
I’ve simplified the query considerably.
I was able to prepare it as Json.
I think this helps.

 {
  "paginate": {
    "intersection": [
      {
        "join": {
          "range": {
            "match": {
              "index": "PeriodicIncomePaymentDateByWallet"
            },
            "terms": [
              {
                "ref": {
                  "collection": "Wallet"
                },
                "id": "284905087731499525"
              }
            ]
          },
          "from": [
            "2021-01-01"
          ],
          "to": [
            "2021-12-31"
          ]
        },
        "with": {
          "lambda": [
            "date",
            "ref"
          ],
          "expr": {
            "match": {
              "index": "PeriodicIncomePaymentRefByRef"
            },
            "terms": [
              {
                "var": "ref"
              }
            ]
          }
        }
      },
      {
        "union": [
          {
            "match": {
              "index": "PeriodicIncomePaymentByPaymentStatusAndWallet"
            },
            "terms": [
              {
                "ref": {
                  "collection": "Wallet"
                },
                "id": "284905087731499525"
              },
              3
            ]
          }
        ]
      }
    ]
  },
  "size": 70
}
6

Many Thanks. I can reproduce the issue now.

> Paginate(
...   Intersection(
.....    Join(
.......     Range(Match(Index("PeriodicIncomePaymentDateByWallet"),[Ref(Collection("Wallet"),"284905087731499525")]), ["2021-01-01"],["2021-12-31"]),
.......     Lambda(["date", "ref"],Match(Index("PeriodicIncomePaymentRefByRef"),[Var("ref")]))
.......    ),
.....    Union([Match(Index("PeriodicIncomePaymentByPaymentStatusAndWallet"),[Ref(Collection("Wallet"),"284905087731499525"), 3])])
.....    ),
...    {size:64}
...    )
{
  data: [ Ref(Collection("PeriodicIncomePayment"), "293405882402743809") ]
}

> Paginate(
...   Intersection(
.....    Join(
.......     Range(Match(Index("PeriodicIncomePaymentDateByWallet"),[Ref(Collection("Wallet"),"284905087731499525")]), ["2021-01-01"],["2021-12-31"]),
.......     Lambda(["date", "ref"],Match(Index("PeriodicIncomePaymentRefByRef"),[Var("ref")]))
.......    ),
.....    Union([Match(Index("PeriodicIncomePaymentByPaymentStatusAndWallet"),[Ref(Collection("Wallet"),"284905087731499525"), 3])])
.....    ),
...    {size:70}
...    )
{
  data: [
    Ref(Collection("PeriodicIncomePayment"), "293405882402743809"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543991301"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543992325"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543993349"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543994373"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543995397"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543996421"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543997445"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543998469"),
    Ref(Collection("PeriodicIncomePayment"), "293755536226124289"),
    Ref(Collection("PeriodicIncomePayment"), "293755536226125313"),
    Ref(Collection("PeriodicIncomePayment"), "293755536226127361"),
    Ref(Collection("PeriodicIncomePayment"), "293755536226128385")
  ]
}

> Paginate(
...   Intersection(
.....    Join(
.......     Range(Match(Index("PeriodicIncomePaymentDateByWallet"),[Ref(Collection("Wallet"),"284905087731499525")]), ["2021-01-01"],["2021-12-31"]),
.......     Lambda(["date", "ref"],Match(Index("PeriodicIncomePaymentRefByRef"),[Var("ref")]))
.......    ),
.....    Union([Match(Index("PeriodicIncomePaymentByPaymentStatusAndWallet"),[Ref(Collection("Wallet"),"284905087731499525"), 3])])
.....    ),
...    {size:100000}
...    )
{
  data: [
    Ref(Collection("PeriodicIncomePayment"), "293405882402743809"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543991301"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543992325"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543993349"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543994373"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543995397"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543996421"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543997445"),
    Ref(Collection("PeriodicIncomePayment"), "293517732543998469"),
    Ref(Collection("PeriodicIncomePayment"), "293755536226124289"),
    Ref(Collection("PeriodicIncomePayment"), "293755536226125313"),
    Ref(Collection("PeriodicIncomePayment"), "293755536226127361"),
    Ref(Collection("PeriodicIncomePayment"), "293755536226128385"),
    Ref(Collection("PeriodicIncomePayment"), "293755536226129409"),
    Ref(Collection("PeriodicIncomePayment"), "293755536227172865"),
    Ref(Collection("PeriodicIncomePayment"), "293755536227173889"),
    Ref(Collection("PeriodicIncomePayment"), "293755536227174913"),
    Ref(Collection("PeriodicIncomePayment"), "293755536227175937"),
    Ref(Collection("PeriodicIncomePayment"), "293755536227176961"),
    Ref(Collection("PeriodicIncomePayment"), "293755551313035783"),
    Ref(Collection("PeriodicIncomePayment"), "293755551313036807"),
    Ref(Collection("PeriodicIncomePayment"), "293755551313037831"),
    Ref(Collection("PeriodicIncomePayment"), "293755551313038855"),
    Ref(Collection("PeriodicIncomePayment"), "293755551314084359"),
    Ref(Collection("PeriodicIncomePayment"), "293755551316181511"),
    Ref(Collection("PeriodicIncomePayment"), "293755551316182535"),
    Ref(Collection("PeriodicIncomePayment"), "293755551316183559"),
    Ref(Collection("PeriodicIncomePayment"), "293755551316184583"),
    Ref(Collection("PeriodicIncomePayment"), "293755551316185607")
  ]
}
>
1 Like