Error 429 - ERROR ThrottlingError: Rate limit exceeded

Help
1

Hi all,

I’m encountering a “rate limit exceeded” error in a Fauna collection where I store parts for purchase orders. The collection currently holds over 40,000 records, and the structure of the records looks like this:

{
  Plnt: "redacted",
  FunctionalLocation: "redacted",
  Department: "redacted",
  EquipmentNumber: "redacted",
  EquipmentDescription: "redacted",
  MaterialNumber: "redacted",
  MaterialDescription: "redacted",
  Bin: "redacted",
  MPN: "redacted",
  ItemCategory: "redacted",
  ManufactSerialNumber: "redacted",
  ModelNumber: "redacted",
  Location: "redacted",
  Price: redacted
}

I have an index set up to query based on department and location and my query would look like:

let set = collection.index(<department>, <location>) { fieldsToProject }
set.pageSize(50)
set.distinct()

If an after cursor is returned with the first response, I run a while loop to continue querying until no after cursor is returned like so:

Set.paginate(<after>, 50)

However, I’ve started hitting the rate limit with my current queries, and I’m on the pay-as-you-go plan, which allows for up to 500 read ops per second.

My questions are:

  1. What are some optimization strategies I could employ to reduce the number of read operations (e.g., adjusting my query structure, pagination approach, or index usage)?
  2. Should I consider breaking this query into smaller batches or limit the frequency of requests?
  3. Is there a more efficient way to structure my collection or index that would help avoid hitting this limit?

Thanks,
Ryan

2

Hi @Ryan_Short. The big thing here is likely that distinct() must read the entire Set, regardless of the page size.

If you need to get the distinct results of 40k documents, you might instead consider paginating your data into your client and then performing the distinct operation on the data locally.

The same would apply to other operations that must read the entire Set to operate, such as .order()

pageSize

As a side note, variables are not mutable, and there are no methods that attempt to mutate objects in place. In your example you execute set.pageSize(50) but do nothing with the result, so the result is not used. You can see in the example on my test DB that the actual page size used is still the default of 16

image
image255×107 6.27 KB

You can, however, shadow variables. That is, you can reuse variable names. You might try

let set = collection.index(<department>, <location>) { fieldsToProject }
let set = set.pageSize(50)

// now return set or do other things with it