Limit for Filter

Feature Requests
#1

The idea is to create limit for Filter function.
It would work like: Iterate over data until limit is reached, then break the loop.
For example:

 q.Filter(
    [1, 2, 3,4,5],
     q.Lambda(
       'i',
       q.GT(0,q.Var('i'))),
        3 //limit is here
  )

would return just
[1,2,3]

The problem is that when someone would iterated over 100,000 results he would have filter activated for each item in index.
Limit would reduce that until there’s satisfying number of items returned.

#2

Thanks for raising this feature-request.

You could use the Take function and specify the number of elements to be returned. But, the Lambda Function would have been executed on the entire set.

#3

That’s true for a fixed result such as your example.

q.Filter(
    [1, 2, 3, -1, -3, 4],
     q.Lambda(
       'i',
       q.GT(0,q.Var('i')),
  )
)

Iirc, it’s only partly true for a Paginated result

q.Filter(
    Paginate(Match(Index(..))),
     q.Lambda(
       'i',
       q.GT(0,q.Var('i')),
  )
)

Since in the above case, it will only iterate over the results of the first page. If that page is 100 000 big then you are right.

However, if you calculate it on a set which probably is often the case, you can actually use page size to achieve the same thing. If I’m not mistaking, filter will only be executed that many times until enough values for the first page are calculated. This is quite similar to what you are hoping to achieve.

Paginate(Filter(
   Match(Index(..)),
     q.Lambda(
       'i',
       q.GT(0,q.Var('i')),
  )
), {size: 2})

It could help if you elaborate on the use case.

#4

Thanks for replies.
What you mention works on small sets.

They issue is to have a way to return at least 30 results from 100,000 without knowing where those 30 are.

If one would set size for Paginate to 64 - he might end up with empty set, if none of first 64 results from Index would match filter.

If one will always Filter a big set to make sure he gets 30 results it would be huge waste of resources.

If someone won’t have 30 or more results, Filter will go through all but it would be a trade off.

#5

@ambrt
absolutely, that’s why you use indexes where you can. In this case since it’s a GT you could have used an index if I’m not mistaken. Probably not immediately obvious because you need Greater Than and not Greater Than or Equal but it can still be done. If I didn’t make a mistake, this should be the equivalent of SELECT * FROM table1 WHERE a > 5 AND a < 10

Map(
    Paginate(
        Difference(
            Join(
                Range(Match(Index('table1_by_range_a')), 5, 10),
                Lambda(['value', 'ref'], Match(Index('ref_by_ref'), Var('ref'))
            )),
            Match(Index('table1_by_a'), 5),
            Match(Index('table1_by_a'), 10)
        )
    ),
    Lambda('x', Get(Var('x')))
)