Deleting large number of documents in single query results in seemingly erroneous response error even though the query is successful

Help
,
1

I want to be able to empty a collection of all its documents in a single query.

I’m using the following within a Next.js application:

import { query as q } from 'faunadb'
import faunadbClient from '/components/api/faunadb/client'
export default async(req, res) => {
  try {
    await faunadbClient.query(
      q.Map(
        q.Paginate(
          q.Documents(q.Collection(req.query.collection)),
          { size: 99999 }
        ),
        q.Lambda(
          ['ref'],
          q.Delete(q.Var('ref'))
        )
      )
    )
    res.status(200).end()
  } catch (e) {
    // something went wrong
    res.status(500).json({ error: e.message })
  }
}

For my particular use case, the collection may have 10k+ documents. When executing this query on a collection with about 1k documents or less, I get a successful 200 response and the collection has been emptied.

However, when I perform this query when the document number is 10k+, all the documents are successfully deleted but the faunadbClient throws an error: “Unexpected end of JSON input”.

This means to be sure the query was in fact successful, I then need to make another query to check if the collection has zero documents before considering the task as complete.

Any suggestions on what’s going awry? Or, is this a bug in the faunadb Client node module? I’m using version 4.5.2.

2

Hi @alexmaughan and welcome!

It looks like this got taken care of in Discord, so I want to close the loop here.

It’s possible that this is related to the maximum transaction size limit: Transaction exceeded limit: got 24179504, limit 16777216 - #23 by Daniel_Steigerwald

Additionally, deleting each document encurs at least 1 Write Op. Due to the transaction size limit and the cost, it could be beneficial for cost to delete the Collection and recreate it.

Unfortunately, you cannot delete a Collection and immediately create another with the same name. As you suggested in Discord, you can work around this by adding versioning to the Collections’ names. Another caveat to deleting a Collection, is that doing so will also delete the Indexes associated with the Collection. That means you may have to version Index names as well.

If you still prefer to delete the individual Documents, then it may be helpful to use pagination. This would be the recommended best practice to working on 1000’s of Documents anyway, regardless of the transaction size limit. There is little cost benefit (In terms of Read/Write/Compute Operations) to performing a query on 10000 Documents in one query than there is splitting it into separate requests. In fact, big queries on so many documents increase the query execution time, so increase the chance of timeouts and contention with other queries. The balance of performance and page-size also depends on your users’ reliable access to the internet, how much you need to limit network traffic. And of course, it may be critical to complete more work within a single transaction to guarantee some function of your application.

Several drivers, including the JS driver, have pagination helpers that at least make the code easy to split work into smaller chunks.

3

Thanks @ptpaterson. This all makes good sense.

This particular feature is for a super admin user, which may, at most, get used only once a month, so I guess this was partly my own laziness in wanting to write the easiest/quickest solution for myself.

The versioning approach is ultimately way more efficient, a lot more robust against timeouts or other interruptions, and not that much more work to implement.

2 Likes
4

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.