How can I create a document reference from Json?

Hi,

I’ve been using Fauna for a long time, and I’m currently updating all queries in my application to migrate to Fauna v10.

In the new version, the format of document references has changed, and I need to create document references from JSON.

I’m using the Dart language, but Fauna does not have a Dart driver. When I serialize my objects to JSON, I cannot generate data in the format Collection(“123”).

The references returned from Fauna come as JSON objects, like this:

{"id": "123", "coll": "Collection"}

Similarly, when I try to create a reference, Fauna does not generate a document reference; instead, it creates an object.

I need help figuring out how to generate document references from JSON.

Hey @hasimyerlikaya! Thanks for the question.

There are two approaches that will work off the top of my head, and the one that’s best for you will depend on the scope of what you’re doing.

The first, and simplest, is to do this directly in an FQL query by grabbing the document you want to ref directly from the collection instead of manually constructing it. This assumes you’re not building a generic Dart driver but rather writing queries for your application. In the sample query you posted, this will look as follows. Note that you can encapsulate this kind of logic in UDF if you’d like, however generally I’d recommend that you avoid using client-side string interpolation to provide the PrivateUser ID because that can expose your application to FQL injection attacks if you’re not careful.

let wallet = Wallet.all().first()
let user = PrivateUser.byId(123)
wallet.update({privateUserRef: user})

The second, and more powerful approach, is what you’d use if you were developing a Dart driver for Fauna or a DSL for your company or application. You use the Tagged variant of Fauna’s wire protocol. With the tagged variant, you provide explicit typing for objects sent to and returned from Fauna.

Here are two ways to send a query to Fauna using tagged types and the sample query above. We describe these here and you can see similar examples where we provide request samples. You can see in both cases, simple JSON serialization doesn’t cut it. In the drivers we’ve written, such as the .NET driver, we implemented custom serializers to accomplish this.

  1. Query as string. Here we’re providing a variable arg that does not exist in the query, and then provide it via the “arguments” key.
{
    "query": "let wallet = Wallet.all().first()\nwallet.update({privateUserRef: arg})",
    "arguments": {
        "arg": {
            "@ref": {
                "id": 123,
                "coll": {
                    "@mod": "PrivateUser" 
                }
            }
        }
    }
}
  1. Query as object. Here we’re providing a more powerful syntax that allows for query composition. This is the format we use in most driver implementations because it allows us to expose composition APIs without risking FQL injection.
{ 
    "query": [
        "let wallet = Wallet.all().first()\nwallet.update({privateUserRef:",
        {
            "@ref": {
                "id": 123,
                "coll": {
                    "@mod": "PrivateUser" 
                }
            }
        },
        "})"
    ]
}

I hope that helps a bit. I’m happy do dive into this a bit more with you if needed.

Hi @pnwpedro,

Thank you very much for your detailed response. I truly appreciate it.

I want to use a simple query format. In fact, everything is working very well except for the Ref type. I am preparing FQL queries in the VSCode extension and sending them as a String.

On the Wire Protocol page, the format shown for Ref is as follows:

I can serialize Ref objects exactly like this without any issues.

Here is my example query:

Wallet.all().first()?.update({
  privateUserRef: {
    "id": "421702934788571200",
    "coll": "PrivateUser"
  }                                    
})

When I execute this query, the operation completes successfully. However, when I check the Fauna Dashboard, the privateUserRef appears as a JSON object rather than a Ref object.

Shouldn’t the privateUserRef field appear like this on the Funa Dashboard?

privateUserRef: PrivateUser(“421702934788571200”)

Could this be a bug?

It’s not a bug, but I can certainly see why it might feel like one. The challenge, and the whole reason for the tagged format, is that we need to tell Fauna that there is a specific type being provided that’s not available in JSON. It’s meant to bridge that gap. When you provide an object in simple format, even though it meets the Ref contract, Fauna sees it as a simple object.

It sounds like what you’re asking for is a kind of duck-type format with simple format queries. If so, it’s totally a reasonable request and would make building relations using simple format relatively easy. I’ll bring this up with the team.

For now though, the best way that I can think of to get around this while using simple format is to encapsulate logic in a UDF. You can do something like setPrivateUser(wallet: ID, user: ID) and pass two IDs. It can get the user from the collection in the implementation and assign it to the wallet. Alternatively, you can explore the tagged format, but sounds like you really want to stick with simple, and I get that.

+1 to everything @pnwpedro is saying.

To dig a bit more into this query:

FQL v10 is interpreting the value of privateUserRef as a plain object. To treat it as a document reference, you’ll want to use an FQL v10 method that returns a document or document reference and set that return as the field value. byId() seems like a good fit for you here:

Wallet.all().first()?.update({
  privateUserRef: PrivateUser.byId("421702934788571200")
})

There are more examples of creating document references in v10 in the docs: Model relationships using document references - Fauna Docs

I really love Fauna and have been using it for a long time. I often recommend Flutter developers to use it as well. Previously, I had made a post on the forum suggesting that Fauna should develop a Dart driver. Flutter currently holds a strong position in mobile application development, and Fauna is a great option. I strongly emphasize that Fauna should focus on Dart and Flutter developers because it has the potential to become a highly successful and widely used database in this field.

I was using an open-source Dart package for Fauna v4. However, the developer stopped maintaining it, and Fauna v4 support will be discontinued in a few months. I had started writing a package for Fauna v10, and I had implemented many queries. However, I later realized that such a package is not really necessary. The new FQL is almost identical to Dart syntax. Preparing queries in the Fauna VSCode extension and sending them as a String is super easy. Even in my own project, I only use the package I developed to create the FaunaClient, allowing me to use it without any special modifications. Having an official driver would be nice, but it is not essential.

Regarding my issue, I can convert a Dart object into JSON. As I demonstrated in the example earlier, I can create Ref types using the “id” and “coll” fields. However, it seems that Fauna does not support creating a Ref object this way. If you add this feature, it would be amazing. With this feature, we could use the new version of Fauna comfortably without requiring a special driver. As you mentioned, I could use a UDF in the queries I write or find the document by its ID and reference it. This could be a possible solution. However, if I convert a Dart object into JSON and then use the “update” or “replace” functions, the references will break. If the team provides a method to create a Ref from a JSON object, that would be fantastic.

Query:

    final walletData = AppSessionManager.activeWallet!.toDataMap();
    final encodedData = walletData.encode();

    final query = '''
       Wallet.all().first()?.replace($encodedData)
     ''';

     final response = await AppFaunaManager.instance.privateDB.queryString(query: query);

encodedData:

{
  "privateUserRef": {
    "id": "421793101252657216",
    "coll": "PrivateUser"
  },
  "accountRef": {
    "id": "421793101311377472",
    "coll": "Account"
  },
  "name": "Birikim",
  "primary": true,
  "currencyCode": "TRY"
}

As you can see, if Funa supported creating references with JSON, we could use serialized objects directly. Now, when I run this query, it will execute successfully, but the document references will be broken.

Thank you for your interest, James. I know that I can reference it this way. Actually, my goal is to use Dart objects converted to JSON directly within queries. I included a detailed example in my previous response. Please take a look there.

As my colleagues have indicated, the Document Reference objects sent back to you are serialized JSON. If you choose to use the “simple” format in your requests, you lose the precision of the types. This is often desired for the sake of human readability, for example. But if you want to preserve and use native Fauna types, you will need to use the, “tagged”, request-response format, deserializing and serializing the data appropriately.

The default format for queries is “simple”. Configure the format to “tagged” to receive results encoded with native type information.

The tagged format was designed specifically for round-trips back and forth between Fauna. For example, if one query response contains the following:

{ "@ref": { "id": 123, "coll": { "@mod": "PrivateUser" } } }

You can hand that right back:

{
    "query": "user",
    "arguments": {
        "user": { "@ref": { "id": 123, "coll": { "@mod": "PrivateUser" } } }
    }
}

If you deserialize native Fauna types represented as JSON into Dart objects, then you will need to serialize them back to JSON with the proper encoding.

If Fauna accepted that plain objects that happen to have a “ref” and “coll” field as documents, we would create ambiguities about whether the author intends to use objects or native References and whether or not we persist plain objects or native References.

After some additional discussion, I can report that while developing FQL v10, we did experiment with allowing a plain object to be coerced into a reference in some situations, but the feedback that we received was ultimately that it was confusing. Given the broader implications and previous feedback, we do not have any intention to introduce any additional feature to enable plain objects to be interpreted as native Fauna References.

UPDATE: corrected my previous assertion that “tagged” was the default format. The default is “simple”.

If we use the “tagged” format, it creates a need for a language-specific driver. For example, Fauna does not have a driver for Dart. I also don’t think you are working on this. Therefore, you are distancing yourself from thousands of developers who are building mobile applications with Flutter today.

The words “id” and “coll” are reserved keywords. I thought that we wouldn’t be able to use them within simple objects because they are listed as reserved in the documentation. It would be enough to consider an object containing only “id” and “coll” as a Ref.

This code is the JSON output that Fauna provides me. It serializes Ref objects exactly like this. However, when I send the same code back to Fauna, it interprets it differently. I think there is an inconsistency here.

{
  "privateUserRef": {
    "id": "421793101252657216",
    "coll": "PrivateUser"
  },
  "accountRef": {
    "id": "421793101311377472",
    "coll": "Account"
  },
  "name": "Birikim",
  "primary": true,
  "currencyCode": "TRY"
}

Actually, we didn’t necessarily need a JSON object to create a Ref. Something like this would have worked as well:

{
  "privateUserRef": "PrivateUser(421793101252657216)",
  "accountRef": "Account(421793101311377472)",
  "name": "Birikim",
  "primary": true,
  "currencyCode": "TRY"
}

If we could send a string like this for Ref fields and Fauna recognized it as a Ref, we wouldn’t have any issues.

Currently, the PrivateUser(123) format that Fauna wants us to use in queries is an undefined data type on the Dart side, making it impossible to serialize for objects.

Currently, 28% of new mobile applications on the App Store are developed with Flutter. This percentage is even higher for the Play Store.

Fauna has access to a huge potential. I don’t know if you are planning to develop a driver for the Dart language, but with a small adjustment, you can eliminate this need.

Flutter developers, like me, can create their queries using the VSCode extension, send them as a string to Fauna, and easily include their serialized data within these queries.

Thank you for your time. I hope the Fauna team considers my suggestion and agrees with me that reaching Flutter developers would add great value to Fauna.

This is incorrect. They are reserved fields only for documents. There is no constraint on other variables or fields for plain objects.

I was mistaken before when I said that “tagged” format was default. My sincere apologies for any confusion that has caused. You should configure your queries to use the tagged format to receive the round-trippable format. It is expected that the simple format loses type information so cannot be sent back to Fauna. See our documentation for more information on available headers: Fauna Core HTTP API - Fauna Docs

And thanks again for highlighting your desire for more support with Fauna and Flutter/Dart! No one is trying to say that wouldn’t be valuable to folks. It clearly would be! We are not arguing with you on that. Please understand that it is not a current priority to provide an officially supported Dart driver. So, in the meantime, we are trying to help you find a solution. If you want to work with the native objects from fauna, then you need to provide a serialization/deserialization layer between JSON and your Dart objects. If you don’t want to do that, then you can use the string-based API. We have provided examples and documentation for doing both. If you have any questions on how to proceed with your chosen path, please feel free to reach out with additional questions to address your specific concerns.

I found a solution that works properly. I separated data fields from Ref fields and created references within the query. Then, I merged them with the data fields and used them where needed.

I’m adding it here in case others might need it in the future.

Many thanks to everyone who helped. :+1:

      final query = '''

        let privateUserId=$privateUserId
        ....       
        let incomeData=$incomeData

        let refs={
          privateUserRef: PrivateUser(privateUserId),
          ...         
        }

        let combinedData=Object.assign(refs, incomeData)

        let income=Income.createData(combinedData)
        
            
        income

        ''';
1 Like

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.