Get documents/events after timestamp - inverse of At function

Help
, ,
1

I want to run a site as cheap and effective as possible. Given that Fauna already stores the history of every document, what I’m trying to achieve is generate a static webpage, save the timestamp of the database at the time of generation inside the static webpage. Then I’d want to hydrate the generated HTML in a front-end JavaScript framework and after that only fetch for the events after the saved timestamp to achieve least amount of reads as possible.

  1. Is this possible?

  2. At:
    a. I found At but it does the reverse, should I do a Difference of At and the index of all documents in the collection to get all the documents after the timestamp?

    b. How many reads would that give me – would it be any better than just getting all the elements?

  3. Should I just index the timestamps somehow?

FEATURE REQUESTS

  1. Could an inverse argument be added to At, to get things after and not before the given ts?

  2. Events
    a. Could the Events function accept afterTimestamp argument to only get me some events?
    b. Are Events indexable so I can do this manually?

Thanks.

2

Hi @jcubed,

Just wondering if you need to get documents created after the saved timestamp.
In that case, you might try to create and index that way

CreateIndex({name:'ref_by_ts',source:Collection(<collection name>),values:[{field:['ts']},{field:['ref']}]})

And then querying it using Range():

Paginate(Range(Match('ref_by_ts'),[<saved timestamp>],[ToMicros(Now())]))

You get back something like this:

{
  data: [
    [1594744613090000, Ref(Collection("test"), "271041967442362886")],
    [1594744629510000, Ref(Collection("test"), "271041984667320837")]
  ]
}

At this point, having the Ref , you can get the full documents created after the saved ts.

Not sure I answered your question. In case, let me know and I can try to help you further.

Luigi Servini

3 Likes
3

This is great, thank you!

4

Very helpful @Luigi_Servini, thank you!

5

I’ll add some extra information since I suspect that this topic will attract people who are interested in changesets. Indeed FaunaDB is temporal so you can build on these features to get efficient timestamps. What Luigi presented is correct and this approach will provide you with all documents that are created or updated after a given timestamp.

Paginate(
  Range(
    Match(Index("ref_by_ts")),
    ToMicros(Time("2020-05-22T19:12:07.121247Z")),
    null
  )
)

This is based on an index on the ts value which represents the last time the document was updated. This will return references but does not tell us exactly what has changed on each of these references. If we need that we can go one step further and map over those references to ask the events

Map(Paginate(
    Range(
      Match(Index("ref_by_ts")),
      ToMicros(Time("2020-05-22T19:12:07.121247Z")),
      null
    )
  ),
  Lambda(['ts', 'ref'], Paginate(Var('ref'), {events: true, after: Time("2020-05-22T19:12:07.121247Z")}))
)

The insight here is that calling Paginate with events: true on a specific reference will also provide you with all the information of what exactly changed in this document. For example when I would just call this directly on a reference:

Paginate(Ref(Collection("accounts"), "268431417906561542"), {
  events: true,
  after: Time("2020-05-22T19:12:07.121247Z")
})

Then the result provides me with all details that changed since that time.

{
  before: {
    ts: 1590174727121247,
    action: "add"
  },
  data: [
    {
      ts: 1592254999010000,
      action: "create",
      document: Ref(Collection("accounts"), "268431417906561542"),
      data: {
        email: "user2@test.com",
        user: Ref(Collection("users"), "268431417898172934"),
        verified: true
      }
    },
    {
      ts: 1597825740110000,
      action: "update",
      document: Ref(Collection("accounts"), "268431417906561542"),
      data: {
        email: "user22@test.com"
      }
    }
  ]
}

Sometimes, you also want to know what is removed since only created + updated + deleted gives you all the information of what changed. The previous index will not provide you with that information since deleted documents will of course not be returned from a regular index.

However, that’s no problem since we can use the temporality there.

Paginate(Documents(Collection("fweets")), {
  events: true,
  after: Time("2020-05-22T19:12:07.121247Z")
})

By setting the events config variable of Paginate to true, Paginate will return events of that selection. Those events will be the deleted and created events (and not the updated)

By combining the two approaches above (the one from Luigi together with the ‘events’) you can get changesets that give you all the information you need and the beautiful part is that it’s efficient. One function uses an index so it only truly gets what was created or was updated, the other function uses the temporal mechanism to efficiently get created/deleted.