Transaction was aborted due to detection of concurrent modification

[fred]

Hi,

I have a sync nodejs job I need to run every day. I do parallel upserts for each document with the following generic function

const upsertDoc = async (doc, coll, index, terms) => {
  return client.query(
    q.Let(
      {
        m: q.Match(q.Index(index), terms),
      },
      q.If(
        q.IsNonEmpty(q.Var("m")),
        q.Replace(q.Select("ref", q.Get(q.Var("m"))), { data: doc }),
        q.Create(q.Collection(coll), { data: doc })
      )
    )
  )
}

Every upsert is applied to a different doc. I’m not manipulating the same doc.

The problem is I’ve started to get the Transaction was aborted due to detection of concurrent modification error with a parallelism level of 4. I’m really confused because this used to work and I can’t think of anything I’ve changed that might cause this. But obviously, there is something.

I’ve tried creating the client in the upsertDoc function too. That didn’t help, although I don’t know what’s the best practice here.

I’d appreciate any clue that might help.

[Luigi_Servini]

Hi @fred,

does the index just return one document (is a unique index)?

Luigi

[fred]

Hi @Luigi_Servini

One of the indexes is, indeed. And I’ve also noticed while trying to debug this, changing the index to not unique solved the issue.

Although I couldn’t understand why because I wasn’t trying anything that would break the unicity contract .

[Felix]

We are dealing with the same error message in a use case that is very similar, except that we don’t modify documents at all. We basically do:

If(
    Exists(Match("some_unique_index")),
    false,
    Create(...)
)

As soon as there is some concurrency, we get (on JVM):

com.faunadb.client.errors.UnknownException: contended transaction: Transaction was aborted due to detection of concurrent modification.

[fred]

That’s indeed quite similar.

One fact I just realised I missed is, yes, the index I’m using in the upsert code is unique, but the other index I mentioned I’ve changed to unique: false was just another one defined on the collection.

So you might want to test other indexes as well.

[Jay-Fauna]

@fred @Felix we retry 4 times with an exponential delay between each retry to apply the transaction before generating this Contended Exception with error code 409. You could do a retry from client side too.

[Felix]

I see. So if I understand correctly, this basically puts a limit on how much a unique index can scale, write-throughput-wise, am I right? I wonder if there are any workarounds; retrying contended transactions will stop working once the number of concurrent writes to the unique index passes some threshold.

[fred]

Thanks @Jay-Fauna

Though I thought the unit of transaction was the document in Fauna. In this case, I’m not changing a document concurrently. I’m adding or replacing different documents?

[phil]

I think you end up modifying the index itself concurrently. Because it’s a unique index it results in contention.

[pier]

So I just hit this error while doing a small benchmark of our API with Apache Bench.

The query was basically updating a property of a single document.

The benchmark wasn’t particularly stressful, only 1000 requests with 10 concurrency. I’d say aprox 70% of updates were succesful, and 30% failed with the error:

Transaction was aborted due to detection of concurrent modification

From reading this thread it’s not clear what is causing the issue.

Can someone from Fauna elaborate on how to solve this?

Edit:

So I created a new collection with no history:

CreateCollection({
  name: "Test",
  history_days: 0
})

Then a single document to test this out with no indexes:

Create(Collection("Test"), {
  data: {
    count: 0
  }
})

And then repeated the benchmark (1000 queries, max 10 concurrent) running this query:

Let(
  {
    doc: Get(Ref(Collection('Test'), '313660688477192783')),
    count: Select(['data', 'count'], Var('doc'))
  },
  Update(
    Ref(Collection('Test'), '313660688477192783'),
    {
      data: {
        count: Add(Var('count'), 1)
      }
    }
  )
)

The result?

The document was only updated 657 times.

{
  "ref": Ref(Collection("Test"), "313660688477192783"),
  "ts": 1635389695890000,
  "data": {
    "count": 657
  }
}

The rest of the updates resulted in the “Transaction was aborted” error.

Hopefully someone from Fauna will explain what is going on here.

[ptpaterson]

Hey @pier

You said you were running 10 concurrent requests. When Fauna encounters contending transactions it will retry 5 times and then abort with a 409 error. Some of the queries are executing quickly enough to pass by, but in general too many transactions are trying to gain write access to the same document.

Every request in Fauna is a transaction, and read-write transactions are strictly serialized (docs). Here is a forums discussion I’ve thought was helpful to understand that: Concurrency and isolation levels

Your case is a bit more straightforward then the original topic. You are trying to read+write to a single document in many transactions. The database has to do something like this (similar steps from this discussion on aggregation)

1. 10 requests to update the Document come in at roughly the same time.
2. They all eagerly start running their transaction
3. Each proceeds with updating the Document
4. The earliest transaction (as determined by the Calvin algorithm) succeeds in Updating the Document and claims ownership of that document in the transaction log. The 9 later transactions no longer have the latest version of the Document and have to retry.
5. The first transaction completes. 9 requests to update the Document are pending.
6. They all eagerly start running their transaction
7. … and the process repeats until the transactions succeed or retry 5 times.

Note how I worded #4, “The 9 later transactions no longer have the latest version of the Document”. You have to first read the latest version of the document before you write to it. This adds additional constraints on your transaction. You could, for example, not require a strictly-serialized READ, and let your transactions be happy with whatever the value was when the transaction started. That would make for a poor counter, though!

Regarding the original topic, the same kind of contention can happen with Indexes (any Set really) when you try to “read the latest” and then write something.

We are working on optimizing for counter-like operations, but until we can deploy that, straight counters like this are not a good pattern for Fauna. The best way to do this is to make a separate collection to log an event and then execute a separate process to aggregate the events. I already linked to a separate discussion about aggregation, but I’ll repeat it if for nothing else than to highlight the complexity that even a counter can take on. We understand this is a pain and hope we can make changes soon.

[pier]

Thanks for your detailed answer @ptpaterson

I was indeed trying to avoid having to execute a periodic process to solve this.

Hopefully a solution for these use cases will be available soon.