Transaction exceeded limit: got 24179504, limit 16777216

Help
1

What did I wrong?

deleteWeb: Do(
  Delete(Ref(Collection('webs'), selectInput('id'))),
  FMap(
    Paginate(
      Match(
        Index('pages_by_web_sort_by_created_asc'),
        Ref(Collection('webs'), selectInput('id')),
      ),
      { size: AppConfig.maxPagesPerWeb },
    ),
    Lambda(['created', 'ref'], Delete(Var('ref'))),
  ),
  FMap(
    Paginate(
      Match(
        Index('importPageQueue_by_web_sort_by_created_asc'),
        Ref(Collection('webs'), selectInput('id')),
      ),
      { size: AppConfig.maxPagesPerWeb },
    ),
    Lambda(['created', 'ref', 'url'], Delete(Var('ref'))),
  ),
  true,
),
3

I think I know where is the problem. Delete returns deleted data. I tried a simpler example which was fixed when I explicitly used Do(Delete(…), true) to return nothing.

I suppose https://docs.fauna.com/fauna/current/api/fql/functions/delete#returns docs is wrong.

4

There is a 16MB limit on transactions, which might be where 16777216 is coming from. This is hinted here, in Data Manager docs. Maybe somewhere else, but I can’t find it. I dug this up from conversations slack.

Also,

There is definitely something wrong with the Delete docs.

5

As an aside,

You hid your original question by updating it with your answer. What you are saying to readers is that this code block is not working. But later you suggest that this is actually the code that does work. This was very confusing. It would be better to provide the new code in a later post and mark it as the solution. You did explain you figured it out, but why not put the working code there?

Also, what is FMAP? Or selectInput? When I first looked at this question, I read the original post first, and my instincts are to respond, “I cannot say with any certainty what is wrong until you explain what these external functions do”. It does look like your question is answered, and the transaction limit is recorded for everyone, so no need to dwell on this part. But it would be helpful if you can sanitize your code a little more when uploading to questions, because it is most likely other readers will not be familiar with you more bespoke library calls.

6

FMap is just an alias to Map, check this proposal https://github.com/fauna/faunadb-js/issues/270.

// just a helper
const selectInput = (path: ExprArg) => Select(path, Var('input'));

Sorry for the suboptimal issue explanation.

This is the working code. I put true to both FMap (Map alias) and it somehow fixed the limit. It should not, because it’s semantically the same.

deleteWeb: Do(
    Delete(Ref(Collection('webs'), selectInput('id'))),
    FMap(
      Paginate(
        Match(
          Index('pages_by_web_sort_by_created_asc'),
          Ref(Collection('webs'), selectInput('id')),
        ),
        { size: AppConfig.maxPagesPerWeb },
      ),
      // https://forums.fauna.com/t/transaction-exceeded-limit-got-24179504-limit-16777216/227
      Lambda(['created', 'ref'], Do(Delete(Var('ref')), true)),
    ),
    FMap(
      Paginate(
        Match(
          Index('importPageQueue_by_web_sort_by_created_asc'),
          Ref(Collection('webs'), selectInput('id')),
        ),
        { size: AppConfig.maxPagesPerWeb },
      ),
      // https://forums.fauna.com/t/transaction-exceeded-limit-got-24179504-limit-16777216/227
      Lambda(['created', 'ref', 'url'], Do(Delete(Var('ref')), true)),
    ),
    true,
  ),
7

But I don’t have a failing repo to demonstrate it, so feel free to ignore it until I will have one.

8

The 16MB limit on transactions should be described somewhere in docs anyway. Thank you.

9

Yes it should, and I’m sorry for the confusion. This limit was actually pushed in hot to fix an outage which is why the docs are lagging a little, we normally wouldn’t push a breaking update without updating the docs first. :bowing_man:

10

So I hit that error again. I am just deleting a bunch of documents, which are quite large. Just because delete returns deleted data, a limit is exceeded.

How we can delete documents without delete returning data?

11

Do with true does not work anymore.

Map(
  Paginate(
    Documents(Collection('pages')),
    { size: 9999 }
  ),
  Lambda(
    ['ref'],
    Do(Delete(Var('ref')), true),
  )
)
12

@ben It’s impossible to delete a bunch of big documents because the delete function always returns deleted data that exceeds a limit. Is this correct behavior?

13

@Daniel_Steigerwald I’m intrigued to find what your document contains that it got beyond 16MB, can you share it here? even just an example.

14

Serialized HTML and CSS as JSON. One big page can have 800KB. When I have twenty pages, I can’t delete them.

15

Awww geez, sounds more like a disaster, how’s the loading time look like for these?

16

Very fast for one page, the best response I saw was 48ms.

1 Like
17

Would updating the document to clear out its data before deleting it fix the issue?

18

use Let inside the Lambda, put the Delete function in a binding and return anything else.

Use Reduce instead of Map and compress the results all down to a single value.

Reduce(
  Lambda(
    ['acc', 'ref'],
    Let(
      { throwAway: Delete(Var('ref')) },
      true
    )
  ),
  true,
  Paginate(
    Documents(Collection('pages')),
    { size: 9999 }
  )
)
19

Thank you, but it did not help.

deleteWeb: Do(
    Delete(Ref(Collection('webs'), selectInput('id'))),
    Reduce(
      Lambda(
        ['acc', 'cur'],
        Let({ throwAway: Delete(Select(1, Var('cur'))) }, true),
      ),
      true,
      Paginate(
        Match(
          Index('pages_by_web_sort_by_created_asc'),
          Ref(Collection('webs'), selectInput('id')),
        ),
        { size: AppConfig.maxImportPagesPerWeb },
      ),
    ),
    Reduce(
      Lambda(
        ['acc', 'cur'],
        Let({ throwAway: Delete(Select(1, Var('cur'))) }, true),
      ),
      true,
      Paginate(
        Match(
          Index('importPageQueue_by_web_sort_by_created_asc'),
          Ref(Collection('webs'), selectInput('id')),
        ),
        { size: AppConfig.maxImportPagesPerWeb },
      ),
    ),
    true,
  )

@ben Please, fix delete, it’s unusable :sob:

20

I misunderstood that when you said didn’t work, that it was because of the transaction limit error. Ben’s response got it to click for me. Sorry.

21

Hi Daniel - Thanks for your patience on this issue. I’d like to be sure clarify that the limit your hitting isn’t due to the size of number of documents returned, but rather the size of the transaction object which needs to be committed to the log, necessary to enforce the consistency guarantee of the database. In other words, the size of the transaction would exceed the limits whether or not the deleted document are returned or not.

Taking a look at the operation provided above, you’d like to perform here, is it the case that you’d like to drop all the documents in a collection with out deleting the collection itself. Is that correct?