Updating GraphQL schema

Hey,

I’m having problems from time to time when updating a schema. Errors like:

Instance data is not valid.

With, I believe, small changes like adding:

type LoginRes @embedded {
  userToken: String!
  userId: String!
}

And changing:

type Mutation {
  ...
  - loginAUser(data: LoginAUserInput!): String! @resolver(name: "login_auser")
  + loginAUser(data: LoginAUserInput!): LoginRes! @resolver(name: "login_auser")
  ...
}

So, with this I’d just like to ask, what’s the best way to update a schema or how should I go about extending my schema, once it’s in production, without having to restart (create a new DB) from scratch each time?

If overriding the schema is a mandatory part of the workflow, how does this work in production if all the collections, functions are lost? How should I handle this in production?

Also, If creating a new DB is part of the workflow, how do I quickly recreate a DB with all the old data, as to have minimum downtime?

This worries me a bit. Any info is appreciated.

Thanks

Hi, @fillipvt!

The Instance data is not valid. error is being thrown in some identified scenarios. The most common case is when the schema update involves a change in an existing Index. Those errors should be fixed in upcoming releases.

According to your example, it seems that the error should also be thrown if you change the type for a custom Mutation field. I’ve tried reproducing it on my end, but I was not able to do it. Maybe the error is located in some other part of your schema? If you can share more of it, we might be able to track it down!

Regarding the second part of your question, the recommended way to evolve the schema, once in production, is at the moment to use the merge import mode as described here. The override mode was originally meant to use during the development phase. The reason why it deletes all of the database data is to allow a quick iteration on the schema definition while you are still on the early stages of the development process.

Now, we’ve learned from the feedback we’ve received that those two import modes are not enough for covering all of the use cases properly. Because of that, changes on the existing modes and new modes will be released in the future as well.

At the moment, overriding a schema might be necessary for certain cases (e.g., removing elements from your GraphQL schema). So, if you need to override your GraphQL schema, but you don’t want to loose the actual data stored in your database, you can try by manually removing the GraphQL metadata from all of the existing database objects (Database, Collections, Indexes, and Functions).

In order to do so, run the following FQL script from within the database you want to override the schema:

// Remove GraphQL metadata from Collections
Foreach(
  Paginate(Collections()),
    Lambda("ref",
      Update(Var("ref"), {
        "data": {
          "gql": null
        }
      }
    )
  )
);

// Remove GraphQL metadata from Indexes
Foreach(
  Paginate(Indexes()),
    Lambda("ref",
      Update(Var("ref"), {
        "data": {
          "gql": null
        }
      }
    )
  )
);

// Remove GraphQL metadata from Functions
Foreach(
  Paginate(Functions()),
    Lambda("ref",
      Update(Var("ref"), {
        "data": {
          "gql": null
        }
      }
    )
  )
);

NOTE: make sure to adjust the page size parameter as necessary.

Finally, you will need to remove the GraphQL metadata from the database itself. In order to do so, you will need to update the database from its parent database. Make sure to run the following command from the the parent database:

// Remove GraphQL metadata from Database
Update(Database("my-graphl-database"),
  {
    "data": {
      "gql": null
    }
  }
);

In case the database is at your account’s root level (i.e., it not’s contained under any parent database), you will need to run the above command from the shell. Let me know in case you need further instructions in order to do so!

After running all of the queries described above, your database should no longer have any GraphQL metadata associated to it. From then on, you should be able to import a new schema on top of it again as if it were a fresh database.

Hey @lregnier!

Thank you very much for your insights.

After I found the problem when making the post I solved it a bit later by erasing everything and starting from scratch. But, as it happens, I just tried deleting an element from the GraphQL schema and I got the same error: Instance data is not valid

The change I’m trying to do at the moment is deleting one element from a type definition:

type Budget {
    pockets: [Pocket] @relation
    owner: AUser!
    name: String!
    description: String
(-)totalAvailable: Float!
}

But after running all the commands to delete all the GraphQL metadata, as you mentioned, I get an error when trying to import the schema once again:

Instance data is not valid.

Would it be alright to share my schema here? Maybe it is indeed as you say, a problem with my schema, but I’ve been using the same schema when setting up a new DB from scratch, and no error happens there.

Thanks again

This is not an answer, but a commiseration:

I’ve gotten this error in the console for what I thought were very simple changes, only to upload the same schema file via a node script without issue. REALLY don’t know what happened there. I do not have any examples now, but if it happens in the future I will try to share!

3 Likes

@ptpaterson Any chance you can share that script, please? Brand new DB that’s completely empty, and any attempt to upload a schema (even the sample one in the faunaDB tutorial) returns “Instance data is not valid.”

Going slightly batty.

@solace brand new database should not have any issues. Are you sure it is a brand new database ? You could run these on SHELL tab.

Paginate(Databases());
Paginate(Collections());
Paginate(Indexes());
Paginate(Functions());

I had deleted the Database and created one new and had the same issue with both the FaunaDB tutorial sample schema and my own.

Of the several schema changes I made, it was the removal of the indexes that eventually made it load, but I’m not certain that was the problem given the sample schema still failed.

If it is possible to get more detailed errors from loading, that would be really helpful.

But I will keep those diagnostic instructions in mind should the issue arise again.

Thanks

@solace, Here is a fairy straight forward npm package function for uploading a schema. And the really relevant function.

@ptpaterson Ah, just a wrapper. All good. Thanks!

@Jay-Fauna

I have noticed that FaunaDB reports back that one needs to wait at least 60 seconds to perform an operation again. The obvious case is when reusing a name, but I am now wondering if there is throttling, caching, or eventual consistency causing the import errors.

In my circumstance, at the first attempt to override the schema in both the web interface and command line, the “Instance data is not valid” occurred and it is unclear given there were no documents (there had been deleted documents, however) what would have triggered the error.

But once I deleted the collections (in the web interface) then ran the import action (with either the FaunaDB sample scheme or my own) via either the web or command-line interface again, it still returned the same error, and again when I attempted with a new database entirely. But the import was done not instantaneously but within a few seconds of the deletion/creation actions.

The fact that it seems to work at a later time with effectively no significant changes to the schema (or none at all with the FaunaDB sample) suggests that the issue might be upstream.

Can you confirm if caching or eventual reconciliation might be contributing to this? If so, it might need better messaging.

Thanks

1 Like

Hi @solace

We may need more information to correlate the actions(override vs update/merge) you have taken vs errors you are seeing (instance data not valid for all the time, for some time).
Here is some information…

  1. Schema import in override mode deletes all existing schema objects (Collections(Documents too), Indexes, Functions, Roles) in the database, waits for 60 seconds to clear them from cache and then starts creating new objects as per schema file. Technically, everything is brand new here and you should not see any error with data validity. More info Modes
  2. Schema import in update/merge updates the existing collections/indexes and creates non existing ones.
  3. We are immediate consistent and do not have any throttling. Feel free to rule this out.
  4. Possibilities of seeing Instance data is not valid are
    • You are updating i.e. merge a schema file which behind the scenes tries to change the definition of an existing index. More info here
    • The schema file tries to create an Index on an existing huge collection (greater than 128 documents/events) which uses background task to build and you try to access the Index before it is active. More info here

Can you please gather the below when you see this issue again.

  1. Schema file you are overriding/updating.
  2. Screenshot of the error
  3. Dump of all schema objects
    Map(Paginate(Databases()), Lambda("x",Get(Var("x"))))
    Map(Paginate(Collections()), Lambda("x",Get(Var("x"))))
    Map(Paginate(Indexes()), Lambda("x",Get(Var("x"))))
    Map(Paginate(Functions()), Lambda("x",Get(Var("x"))))
    

Hi @Jay-Fauna ,

I was using override both on the web interface as well as command line when I experienced this issue. I had not attempted to use the other modes at that time.

I’ve only just started using FaunaDB so there was either 0 or 1 record at most in the 2-3 collections I had in my schema. After the first few attempts with the instance of the database I was working on, I gave up and deleted it and started again from scratch when I experienced the same issue. It resolved itself eventually without any other changes to the schemas (sample, or custom) and other imports since have succeeded on the first attempt.

But thank you for clarifying those additional details. If I observe the problem again I will provide the diagnostic info you requested. :slight_smile: