Delete Documents older than (if count > 1)

Help
1

I’m trying to follow best practices and create a separate process that will delete documents, to prevent built up.

To do this, I need to make sure to not just delete old documents (Let’s say older than 48hours).

Each document has a key property inside data, with a UUID. Many documents can have the same UUID.

But I need to make sure that every document belonging to a unique uuid, always has at minimum 1 left in a collection. Otherwise our queries would start to break.

How would I do this? Is this query even possible with Fauna?

2

Not very unique, then, are they! :rofl: :slightly_smiling_face:

Jokes aside, this should be relatively easy to do if you have the documents in the order that you want.

skip one then paginate the rest

If multiple requests are okay, then you can request a page size of 1, get a cursor back, then issue new requests to delete everything after that.

// first request
Paginate(
  Match(Index("things_by_uuid__ts_desc"), "f90b38d9-77d7-47f1-920c-79b2b4bb2c8d"),
  { size: 1 }
)

// result
{
  after: [Ref(Collection("things"), "307286583998939200")],
  data: [Ref(Collection("things"), "307286571965481024")]
}

// next request
Map(
  Paginate(
    Match(Index("things_by_uuid__ts_desc"), "f90b38d9-77d7-47f1-920c-79b2b4bb2c8d"),
    {
      after:  [Ref(Collection("things"), "307286583998939200")]
      size: 100000
    }
  ),
  (ts, ref) => Delete(ref)
)

Note, I did this to make the query simple: size: 100000
But, in practice, there are almost no additional benefits to running such a large query. It would be best practice to paginate over page sizes perhaps the default (64) to maybe 100 entries at a time. This keeps query-time low and still optimizes your cost in terms of Read/Write/Compute ops. Reasons to increase page size might be A) to ensure all work is within a single transaction, but make sure you REALLY do need that, B) network connections are slow or unstable, increasing the cost of network traffic. You may have others, but do consider the actual benefit for your use case.

you need to delete everything in one transaction, AND you can guarantee you can page for all documents in one transaction.

remember, the limit is 100000. This won’t work if you have more. This is also kinda whacky, but should honestly work well if there are only ever a few documents. At the least, I hope it demonstrates the power of FQL as a programming language to do what you need it to do! Basically:

  • Paginate everything
  • save the Ref of the first item
  • map over everything and delete if it’s not the first item
Let(
  {
    things: Paginate(
      Match(Index("things_by_uuid__ts_desc"), "f90b38d9-77d7-47f1-920c-79b2b4bb2c8d"),
      { size: 100000 }
    ),
    first_thing: Select(["data", 0], Var("things"), null)
  },
  Map(
    Var("things"),
    Lambda(["ts", "ref"], 
      If(
        Equals(ref, Var("first_thing")),
        null,
        Delete(Var("ref"))
      )
    )
  )
)
1 Like
3

This seemed like a fun exercise, so I wrote a quick article about it here:

Slightly different approach than @ptpaterson but it looks like it should work too

1 Like