TimeAdd only works with hardcoded values

I am using TimeAdd( base, offset, unit ) to calculate when a given document should “expire”.

Each document contains the values for base, offset and unit within itself, and I am using Select to load such values into the TimeAdd function, as such:

TimeAdd(
  Select(["data", "executions", -1], Var("document"), Epoch(0, 'second')),
  Select(["data", "schedule", "amount"], Var("document")),
  Select(["data", "schedule", "unit"], Var("document")),
)

The problem is that the function above always returns null.

I double checked the values and I am sure that Select is returning the values expected:

  • Select(["data", "schedule", "amount"], Var("document")) returns 24
  • Select(["data", "schedule", "unit"], Var("document")) returns "hours"

My debugging showed me that in order to make it work I had to hard code offset and unit.

This works:

TimeAdd(
  Select(["data", "executions", -1], Var("document"), Epoch(0, 'second')),
  24,
  "hours",
)

What is happening? Do I need to always hardcode the parameters of TimeAdd? :exploding_head:

Note: This is happening while I am creating a binding

You haven’t provided a sample document, so I can only demonstrate that TimeAdd does work with document values:

> CreateCollection({ name: "executions" })
{
  ref: Collection("executions"),
  ts: 1666563547730000,
  history_days: 30,
  name: 'executions'
}

> Create(
  Collection("executions"),
  {
    data: {
      executions: [ Now(), Now(), Now()],
      schedule: {
        amount: 24,
        unit: "hours",
      },
    }
  }
)
{
  ref: Ref(Collection("executions"), "346349710971240960"),
  ts: 1666563673860000,
  data: {
    executions: [
      Time("2022-10-23T22:21:13.840Z"),
      Time("2022-10-23T22:21:13.840Z"),
      Time("2022-10-23T22:21:13.840Z")
    ],
    schedule: { amount: 24, unit: 'hours' }
  }
}

> Let(
  {
    document: Get(Ref(Collection("executions"), "346349710971240960")),
  },
  TimeAdd(
    Select(["data", "executions", -1], Var("document"), Epoch(0, 'second')),
    Select(["data", "schedule", "amount"], Var("document")),
    Select(["data", "schedule", "unit"], Var("document")),
  )
)
Time("1970-01-02T00:00:00Z")

One problem I see is that your first Select uses a negative offset. By itself, that would return an error:

> Select(-1, ["A", "B", "C"])
Error: value not found
{
  errors: [
    {
      position: [
        'from'
      ],
      code: 'value not found',
      description: 'Value not found at path [-1].'
    }
  ]
}

But no error occurs because you provided a default value of Unix epoch.

Can you provide a sample document that results in null?

Thanks for the attention @ewan! I am sorry I didn’t provide more data in the first post as I was afraid of making it too complex and scare the community away.

I have just pushed a reproducible of the bug and I hope I managed to make it very straightforward and clear. Please check it out:

No problem, I can flip the array around to have new values prepended instead of appended.

Just out of curiosity though, how can I achieve the effect intended? How to “pop” values (take the last value added)?

Apologies, but providing an application that includes a homebrew introspection framework does make this a lot more complex. I don’t have time to dive into all of the moving pieces.

Since you’re asking about TimeAdd, I think it’s useful to isolate that use case to determine whether the problem is FQL itself or elsewhere. Based on what I can see in your repo, it looks like the FQL-only setup is equivalent to:

> CreateCollection({ name: "CrawlingQuery"})
{
  ref: Collection("CrawlingQuery"),
  ts: 1666654819730000,
  history_days: 30,
  name: 'CrawlingQuery'
}
> Create(Collection("CrawlingQuery"), {
  data: {
    keyword: "A",
    schedule: { amount: 24, unit: "hours" }
  }
})
{
  ref: Ref(Collection("CrawlingQuery"), "346445339134263808"),
  ts: 1666654871970000,
  data: { keyword: 'A', schedule: { amount: 24, unit: 'hours' } }
}
> Create(Collection("CrawlingQuery"), {
  data: {
    keyword: "B",
    schedule: { amount: 24, unit: "days" }
  }
})
{
  ref: Ref(Collection("CrawlingQuery"), "346445363990757888"),
  ts: 1666654895690000,
  data: { keyword: 'B', schedule: { amount: 24, unit: 'days' } }
}
> Create(Collection("CrawlingQuery"), {
  data: {
    keyword: "C",
    schedule: { amount: 24, unit: "minutes" },
    executions: [Now()]
  }
})
{
  ref: Ref(Collection("CrawlingQuery"), "346445483271520768"),
  ts: 1666655009460000,
  data: {
    keyword: 'C',
    schedule: { amount: 24, unit: 'minutes' },
    executions: [ Time("2022-10-24T23:43:29.420Z") ]
  }
}
> CreateIndex({
  name: "by_expiry",
  source: {
    collection: Collection("CrawlingQuery"),
    fields: {
      expiry: Query(
        Lambda(
          "document",
          TimeAdd(
            Select(["data", "executions", 0], Var("document"), Epoch(0, 'second')),
            Select(["data", "schedule", "amount"], Var("document")), 
            Select(["data", "schedule", "unit"], Var("document")),
          )
        )
      ),
    },
  },
  values: [
    { binding: 'expiry' },
    { field: ["ref"] },
  ]
})
{
  ref: Index("by_expiry"),
  ts: 1666655153850000,
  active: true,
  serialized: true,
  name: 'by_expiry',
  source: {
    collection: Collection("CrawlingQuery"),
    fields: {
      expiry: Query(Lambda("document", TimeAdd(Select(["data", "executions", 0], Var("document"), Epoch(0, "second")), Select(["data", "schedule", "amount"], Var("document")), Select(["data", "schedule", "unit"], Var("document")))))
    }
  },
  values: [ { binding: 'expiry' }, { field: [ 'ref' ] } ],
  partitions: 8
}
> CreateFunction({
  name: "nextCrawlingQueries",
  body: Query(
    Lambda(
      ['indexName', 'size', 'afterCursor', 'beforeCursor'],
      Map(
        Paginate(
          Filter(
            Match(Index(Var('indexName'))),
            Lambda(['expiry', '_'], LTE(Var('expiry'), Now()))
          ),
        ),
        Lambda(['_', 'ref'], Get(Var('ref')))
      ),
    ),
  )
})
{
  ref: Function("nextCrawlingQueries"),
  ts: 1666656373770000,
  name: 'nextCrawlingQueries',
  body: Query(Lambda(["indexName", "size", "afterCursor", "beforeCursor"], Map(Paginate(Filter(Match(Index(Var("indexName"))), Lambda(["expiry", "_"], LTE(Var("expiry"), Now())))), Lambda(["_", "ref"], Get(Var("ref"))))))
}

Does that look about right? If so, now we can diagnose.

Since the by_expiry index has no terms defined, this query should demonstrate whether the binding is working as expected:

> Paginate(Match(Index("by_expiry")))
{
  data: [
    [
      Time("1970-01-02T00:00:00Z"),
      Ref(Collection("CrawlingQuery"), "346445339134263808")
    ],
    [
      Time("1970-01-25T00:00:00Z"),
      Ref(Collection("CrawlingQuery"), "346445363990757888")
    ],
    [
      Time("2022-10-25T00:07:29.420Z"),
      Ref(Collection("CrawlingQuery"), "346445483271520768")
    ]
  ]
}

That result certainly makes it look like TimeAdd is behaving as expected. Only the C document has an executions array, so it is the only document that does not have a Unix epoch-based time value in the result.

And when I call the UDF:

> Call(Function("nextCrawlingQueries"), "by_expiry", 50, "after", "before")
{
  data: [
    {
      ref: Ref(Collection("CrawlingQuery"), "346445339134263808"),
      ts: 1666654871970000,
      data: { keyword: 'A', schedule: { amount: 24, unit: 'hours' } }
    },
    {
      ref: Ref(Collection("CrawlingQuery"), "346445363990757888"),
      ts: 1666654895690000,
      data: { keyword: 'B', schedule: { amount: 24, unit: 'days' } }
    }
  ]
}

The C document doesn’t appear in this result because its executions timestamp is 24 minutes in the future, and not less than Now().

All this means is that the FQL portions of your application do appear to be working as expected.

From here, I’m not sure where the problem lies. My expertise is not with GraphQL, or TypeScript. So I’m not sure whether there’s an unexpected interaction with the GraphQL API, a problem with your schema definition, or something in your application that is the problem.

Currently, I see no reason that the non-hardcoded values should return an empty result compared to the hardcoded values, because (as demonstrated), the non-hardcoded queries work well.

Thank you so much @ewan!

I built the reproducible the way I built thinking it would be the most straightforward way. But of course it’s not. We are coming from different places and I see why it makes much more sense to share the shell inputs instead. I am not used to use it, but I will work on improving that! Thanks for the patience :relaxed:

I have run the commands you shared and I can confirm that they work for me. I still don’t know what is wrong in my reproducible, but having your code as reference I will now be able to dive deeper into mine and find any difference that might be causing the issue.

I will come back later on to share my discoveries. Once again, thank you so much for all the attention!

The problem somehow seems to be in the document creation. If I create the document using the code above in the shell, everything works fine.

However if I create the document using the graphql api the index fails do calculate expiry.
The weirdest part is that the created documents look identical in the dashboard and shell. I couldn’t spot any difference on data types.

In the images below, the 3 top documents were created using the shell, while the other ones were created using graphql.

Collection:

Index:

Same results, but in the shell:

> Map(
...   Paginate(Documents(Collection('CrawlingQuery'))),
...   Lambda('ref', Get(Var('ref')))
... )
{
  data: [
    {
      ref: Ref(Collection("CrawlingQuery"), "346481360461890128"),
      ts: 1666689224590000,
      data: { keyword: 'A', schedule: { amount: 24, unit: 'hours' } }
    },
    {
      ref: Ref(Collection("CrawlingQuery"), "346481360688382544"),
      ts: 1666689224810000,
      data: { keyword: 'B', schedule: { amount: 24, unit: 'days' } }
    },
    {
      ref: Ref(Collection("CrawlingQuery"), "346481360915923536"),
      ts: 1666689225020000,
      data: {
        keyword: 'C',
        schedule: { amount: 24, unit: 'minutes' },
        executions: [ Time("2022-10-25T09:13:44.936661Z") ]
      }
    },
    {
      ref: Ref(Collection("CrawlingQuery"), "346481412164026955"),
      ts: 1666689273915000,
      data: { keyword: 'A', schedule: { amount: 24, unit: 'hours' } }
    },
    {
      ref: Ref(Collection("CrawlingQuery"), "346481412480696907"),
      ts: 1666689274190000,
      data: { keyword: 'B', schedule: { amount: 24, unit: 'days' } }
    },
    {
      ref: Ref(Collection("CrawlingQuery"), "346481412821484107"),
      ts: 1666689274523000,
      data: {
        keyword: 'C',
        schedule: { amount: 24, unit: 'minutes' },
        executions: [ Time("2022-10-25T09:14:33.866Z") ]
      }
    }
  ]
}
> Paginate(Match(Index("by_expiry")))
{
  data: [
    [
      Time("1970-01-02T00:00:00Z"),
      Ref(Collection("CrawlingQuery"), "346481360461890128")
    ],
    [
      Time("1970-01-25T00:00:00Z"),
      Ref(Collection("CrawlingQuery"), "346481360688382544")
    ],
    [
      Time("2022-10-25T09:37:44.936661Z"),
      Ref(Collection("CrawlingQuery"), "346481360915923536")
    ],
    [ null, Ref(Collection("CrawlingQuery"), "346481412164026955") ],
    [ null, Ref(Collection("CrawlingQuery"), "346481412480696907") ],
    [ null, Ref(Collection("CrawlingQuery"), "346481412821484107") ]
  ]
}

I will create a new reproducible to capture this finding more clearly.

There we go! The simplest reproducible I could make without brainyduck:

Thanks so much for distilling the reproduction case down to the simplest available!

I can confirm the same problematic result. I can’t currently explain it. I’m asking internally to see what the answer might be.

1 Like

One of our engineers discovered the problem.

TimeAdd works with integers/longs. But your schema declares the amount field as Float!. The documents created via the GraphQL API honor the schema.

Unfortunately, the Dashboard Shell, and fauna-shell, use the JavaScript driver, and JavaScript opportunistically presents floats as ints wherever possible.

Buy it is possible to verify the stored type:

Map(
  Paginate(Documents(Collection("CrawlingQuery"))),
  Lambda(
    "X",
    Let(
      {
        doc: Get(Var("X")),
        amt: Select(["data", "schedule", "amount"], Var("doc")),
      },
      {
        amt: Var("amt"),
        isint: IsInteger(Var("amt")),
      },
    )
  )
)
{
  data: [
    { amt: 24, isint: true },
    { amt: 24, isint: true },
    { amt: 24, isint: true },
    { amt: 24, isint: false },
    { amt: 24, isint: false },
    { amt: 24, isint: false }
  ]
}

There are two ways to solve this:

  1. Change your schema to use amount: Int!
    All amount values would be stored as integers, and would be usable in TimeAdd. That might not suit your use case, though.

  2. Change your index binding to call ToInteger on the amount value. Like so:

    CreateIndex({
      name: "by_expiry-new",
      source: {
        collection: Collection("CrawlingQuery"),
        fields: {
          expiry: Query(
            Lambda(
              "document",
              TimeAdd(
                Select(["data", "executions", 0], Var("document"), Epoch(0, 'second')),
                ToInteger(
                  Select(["data", "schedule", "amount"], Var("document"))
                ),
                Select(["data", "schedule", "unit"], Var("document")),
              )
            )
          ),
        },
      },
      values: [
        { binding: 'expiry' },
        { field: ["ref"] },
      ]
    })
    

    With that change:

    > Paginate(Match(Index("by_expiry-new")))
    {
      data: [
        [
          Time("1970-01-02T00:00:00Z"),
          Ref(Collection("CrawlingQuery"), "346509741536576000")
        ],
        [
          Time("1970-01-02T00:00:00Z"),
          Ref(Collection("CrawlingQuery"), "346509761140752896")
        ],
        [
          Time("1970-01-25T00:00:00Z"),
          Ref(Collection("CrawlingQuery"), "346509741557547520")
        ],
        [
          Time("1970-01-25T00:00:00Z"),
          Ref(Collection("CrawlingQuery"), "346509761141801472")
        ],
        [
          Time("2022-10-25T11:14:22.937Z"),
          Ref(Collection("CrawlingQuery"), "346509761130267136")
        ],
        [
          Time("2022-10-25T17:08:50.916Z"),
          Ref(Collection("CrawlingQuery"), "346509741566984704")
        ]
      ]
    }
    

I am glad you found the problem!

I will be operating with fractional numbers (I just didn’t happen to be using them in the example), but both solutions you presented work with integers. Isn’t there a solution for this problem not based on integers?

According to Fauna’s docs, TimeAdd’s offset is a number:

Numbers can be 64-bit signed two’s complement integers (long ints), or 64-bit double-precision floating point values (doubles);

If TimeAdd works with integers and doubles/longs, and GraphQL with floats, why can’t floats be converted to “64-bit double-precision floating point values (doubles)”? Sounds more reasonable than having them interpreted as integers by TimeAdd.

If TimeAdd works with integers and doubles/longs

TimeAdd works with integers and longs, not doubles. We’ll have to update the documentation to reflect that (DOCS-2256).

why can’t floats be converted to “64-bit double-precision floating point values (doubles)”?

A Float in a GraphQL schema is stored as a Fauna “double”. No conversion is necessary. The problem is that TimeAdd doesn’t accept those.

FYI: We’ve published an update to the TimeAdd and TimeSubtract reference topics to indicate that the offset parameter must be an integer/long, closing DOCS-2256.