This could also be called “Roles, FQLv4 vs FQLv10 vs FSL and how functions are no longer referenced as Function(‘name’) in roles”
“Poor communication and the future of Fauna” is an alternative title.
There are some technical questions - which I will get to, as I am stuck with something, but it is the end result of a long day of getting stuck and solving one thing after another… it really should not be like this.
It is also my hope that anyone in a similar boat to me read this and learn from my mistakes.
Oh, and I find a whole bunch of bugs in the Fauna website along the way (they are marked [Of note / bug])
OK, so where shall we begin?
First - having to have a Professional account in order to keep using the Global hosted databases is rather an odd thing to do. Yes, the data is stored in both the US and the EU, but isn’t that the point of serverless, and especially Fauna? It is a distributed database after all.
My DB is small - only 50MB. The system using it has been designed to be fast, and reduce the amount of storage it uses up. As all software should be. It is in semi production now, but not in use by our users yet, which will cause it to grow, though hopefully not by too much each year.
Received an email from Fauna Jul 2023 - email from Mik Cepulis states and I quote:
Lastly, our existing Classic Region Group is evolving into a new Global Region Group. For customers who wish to be in the US or EU region and not the Global region, we are committed to providing you with additional guidance and tools to ensure a smooth transition.
I read this and thought, well OK. I’m fine where I am. Later, when I realised I wasn’t, I could NOT FIND any documentation about this - Where is this documentation for a “smooth transition”?
Also in the email
If you do not move to one of the new plans before December 30, you will be automatically moved to a similarly priced plan. For example, if you are currently on the Team plan, you will be moved to the Startup plan on that date.
I am using Pay-as-you-go (even though I am mostly within the limits of the Free Tier during development, I decided it was better to support Fauna two years ago, and so signed up for $25 a month.
I am not moving to another plan, so the above snippet from the email seemed besides the point, and didn’t raise any flags.
The price is the same for me, and all seems OK. Or so I thought.
In [October 2023] I get a general email from Fauna
In order to help you with this transition, we analyzed your Fauna usage for the month of September. Given that you currently have databases in the Global Region Group and given your query throughput patterns, we recommend that you upgrade to the new Pro plan in order to maintain the same level of service.
Starting December 1, 2023 access to the Global Region Group will be limited to the Pro plan.
Bombshell. Wait, I have to pay $500 a month, and you think my access pattern is that level? (the Dashboard says I used $0.30 in the last two weeks.
We are not in full production yet, still testing and checking things. This can’t be right.
This was the VERY FIRST TIME that it became clear that I am going to have to move the database.
Also notice the first email mentioned Dec 30, while the second one mentions Dec 1 - in the rush of reading emails and keeping up with day to day tasks on top of getting software written, I had firmly ‘Dec 30’ in my mind…
I was working on another project that was time sensitive, I could not just drop everything to get to work on this.
But I did do some quick googling at the time and took down some notes on how I might move the database.
I am late to get to the actual task of doing this, it is already Dec, and fortunately while I am still on the Pay-as-you-go I appear to have been given a grace period extension to sort this out (which I really appreciate, thank you).
I needed to move the database to the US or EU domains.
So this is the adventure that was unfurled upon me as I have had to fight to sort out.
Let’s start with documentation - again, there is no documentation on how to migrate your database from one server to another.
The answer always seems to boil down to using backup, which requires a Startup account.
On the one hand, it is a ‘startup’ account, on the other, it’s much more expensive than a very fast high availability SQL server.
I’m not going to pay $150 just to move 50MB of data.
There was a post on the forums about this a year or so ago, and a user posted a link to a script that could be used to copy a database. I can’t find the Fauna forum post (sorry) but here is the gist of the script the chap very helpfully and thankfully shared.
I do remember that the response on that post was to upgrade the account, but this script made much more sense.
Through a few trials, I managed to get it to work, and copied the database from the Global region to the US region.
However, it doesn’t handle the roles, and this is where things really started going south.
At first I thought about writing a script to fetch the Roles from one DB and write it to the other, but some commands would not work (using the Fauna Shell terminal installed via Brew on a Mac).
Then it occurred to me that I could use the Fauna Migration tool (see Fauna Schema Migration tool) because I had been using it for managing all the updates to the database schema. Simply copy the role files, rename each role *_2, point the migration tool to the new database, and ‘migrate’ (in other words - write) the roles to the database, and then use the dashboard to remove the _2 from the names. Should reduce the work.
The migration failed with an error “BadRequest: instance already exists”
Totally stumped - there were no roles in the destination DB, and only roles were being migrated (I checked).
Mind you, the error messages from the migration tool are always spurious, and sometimes I have to copy and past the migration script into the Fauna Shell to figure out where the error actually occurred, back trace it by guessing roughly where in my code the problem was (because the script is full of temporary vars that are not present in may actual code, making it difficult to debug) and then fix the problem and try again.
So I found the update script files, and it looked something like this:
CreateRole({
"name": "backend_role",
"privileges": [{
"resource": Function("make_email_verification_token_from_email"),
"actions": {
"call": true
}
}, {
"resource": Function("login_account"),
"actions": {
"call": true
}
}, {
"resource": Function("request_password_reset"),
"actions": {
"call": true
}
}]
})
I pasted this into the Fauna Shell (which is now pointing to the new database in the US region).
We get syntax errors like this:
invalid_query: The query failed 1 validation check
error: Expected end-of-input
at query:1:13
|
1 | privileges: [
| ^
It occurred to me “of course, it is supposed to be on one line…?” so I tried that, and then I see the real error message:
error: Unbound variable
CreateRole
at query:1:1
Mmmmm, let’s check Fauna documentation.
That’s odd, this does not seem to exist anymore - ANYWHERE.
Try searching the Fauna home page - there is no search option.
Most every site worth it’s salt has a search box on the home page to save you time in finding things. Fauna does not.
Scroll to the bottom of the page, either ‘Learn More → Support’, ‘Learn More → Documentation’ or ‘Quick links → Helpdesk’ (which is the same as ‘Learn More → Support’).
Instinct says ‘documentation’.
Searching documentation for CreateRole (paste in CreateRole) it brings up key.create and Multitenancy.
So I tried going to the helpdesk page instead. Searching for CreateRole it is even worse:
No result found for “CreateRole”
Change your query or clear your filters
[Of note / bug]
Clicking “Change your query” does nothing (Mac Safari 16.6 bug?)
What I next did was go back to the docs, manually find Role, and see what it said. I learned of Role.create for the first time.
Oh, of course, I remembered, Fauna changed from v4 to v10 a few ‘months’ (year?) back (I should add that I had no idea that this was going to happen until a few months back - there were no emails to notify me of this development, I found out about it completely by random when I read someone else’s blog post. That’s right, I didn’t find out directly from Fauna about this change.
When I then went looking on Fauna’s site for information about it a few months ago, there was very little information.
To make matters confusing - documentation is not labelled ‘v10’ or ‘v4’ - BUT realise that the bottom left of the documentation has an option Fauna Current and click on that you can will see current or v4.
[Suggestion]
Can I make a recommendation: rename the bottom left hand ‘Fauna’ to ‘Documentation Version’ to make it a bit more noticeable/obvious?
[Of note / bug]
While writing up this report, repeating my steps and copying things down for accuracy, and to try to be as informative in this post as possible - I discovered that the documentation search box does indeed turn up Role.create if you TYPE CreateRo into the box - you can’t paste it in, you have to type it. At first I thought this to be a bug in the way search works on Fauna’s documentation, but it gets worse. If you type ‘CreateRo’ then Role.create appears, but if are a fast typer and get to ‘CreateRol’ before search appears, Role.create won’t appear, and if you finish typing CreateRole well, you get the picture.
So I use the shell, and update the command to create the role using Role.create
Joy? Of course not, more pain (semi sarcasm 
we are used to this right?)
Role.create({"name": "backend_role","privileges": [{"resource": Function("register_with_verification"),"actions": {"call": true }}, {"resource": Function("make_email_verification_token_from_email"),"actions": {"call": true}}, {"resource": Function("login_call_limited"),"actions": {"call": true}}, {"resource": Function("login_account"),"actions": {"call": true}}, {"resource": Function("request_password_reset"),"actions": {"call": true}}]})
error: Anonymous functions cannot be stored. Use a string of FQL instead.
at query:1:12
What?
So this obviously changed as well.
Let’s see what we can find out about functions named in Role privileges.
Not a lot. In fact, I could find NOTHING that helped me (at least, so I thought).
I looked up function, found Function.byName() perhaps that could be used, nope.
Stumped, I thought ‘of course, just use the Dashboard, create a role, and look at what the FQL is’.
I open Dashboard version 10, find roles, and note one of the test roles I created earlier as a test (literally, just Role.create({name:‘work’}) and noticed it was not labelled v4 (so guess it must be v10) it was created using the Fauna Shell in Mac terminal.
So I realise the terminal is now using FQL v10. Alright.
Clicked on a role labelled v4 which had some function settings in it - but the v10 editor would not display it.
So I opened the v4 dashboard, found roles, clicked on it (or so I thought) - bang.
Ooops…
Something went wrong
Please login again.
??? OK, log in again, do the same thing again, well, it turns out the v4 dashboard cannot open v10 Roles (I had accidentally clicked the wrong role, but at the time didn’t realise it).
[Of note / bug]
A Role created using the Fauna Shell (MacOS terminal) or the V10 dashboard cannot be opened on the v4 dashboard - it just crashes.
Back to the v4 dashboard to create a new role, select a function, and enable it to be called. Save, and then open the Role in FQL mode in the Dashboard and this is what I see:
{
ref: Role("work1"),
ts: 1702363776080000,
name: "work1",
privileges: [
{
resource: Ref(Ref("functions"), "build_structure_preview_cache"),
actions: {
call: true
}
}
],
membership: []
}
Great! So the function form has changed, we can use that, right?
[Of note / on purpose ‘bug’]
BTW I note there is a ‘save’ button, but cannot actually edit the role - presume this is because it is V4, and it’s getting left behind. This means you cannot update v4 roles - period. The Fauna Shell is v10, so you are done (is this really right? It seems to be).
Putting that aside, let’s check out
Function("make_email_verification_token_from_email")
Should be this?
Ref(Ref("functions"), "make_email_verification_token_from_email")
Going back to the Fauna Shell, let’s try again with this:
Role.create({"name": "backend_role","privileges": [{"resource": Function("register_with_verification"),"actions": {"call": true }}, {"resource": Ref(Ref("functions"), "make_email_verification_token_from_email"),"actions": {"call": true}}, {"resource": Function("login_call_limited"),"actions": {"call": true}}, {"resource": Function("login_account"),"actions": {"call": true}}, {"resource": Function("request_password_reset"),"actions": {"call": true}}]})
error: Unbound variable
Ref
at query:1:153
So, FQL that was created by the v4 dashboard, cannot be used in the v10 Fauna Shell (on MacOS terminal).
Makes sense. But, the new FQL being used is in fact v10 - because I certainly never used or saw that form of it in v4.
But it doesn’t work in the v10 Fauna Shell.
Very confusing.
So, let’s go to the V10 dashboard, and see what happens when we create a Role there. Obviously this must be the path forwards.
It looks… very different.
role work2 {
}
A little info box above that Role (I) pops up with
“Roles are managed using the Fauna schema editor. Learn more”
OK, let’s check that out. The documentation opens, and I note immediately that this is not FQL. On the left it shows FSL Reference > Role.
Uhh what’s this now?
FSL = Fauna Schema Language and it was announced almost exactly one month ago.
So we have a new language to define the schema. First I had heard of that (I literally found out about it writing this post).
The documentation talks about using it - but I think you should read Fauna’s block post first - which is here:
TLD FSL is to be used for managing the database schema. It can be uploaded from your computer to the DB to push changes to the DB, and it CAN be downloaded from the database - so it will likely be possible to take your existing v10 compliant database, get the current schema, edit it, and then push the changes to the DB. It seems that it is granular in control and looks like a worthwhile improvement to Fauna.
FQL v10 can still work on the schema as well though. I presume a change made using FQL can be downloaded as a new FSL scheme from the DB (though that’s obviously not the intended way of doing things)?
This is very similar to using FQL and JS with the Fauna Migration tool (see Fauna Schema Migration tool)
but the the Fauna Migration tool does not seem to support FQL v10 (otherwise I would not have had issues with the Role Migration at the beginning of this post) so I guess FSL will be the way to go.
It seems that FSL tooling will support roll-back, but have not looked into it yet.
Does it support referencing custom JS functions the way you can with the Fauna Migration tool that can be imported into UDFs to reduce duplicate code (at least in the source), making for more effective programming? I wonder, probably not.
This means I have to nasty amount of code to re-write my schema files to the new format - and as for the JS functions… oh dear.
All this, and I still haven’t solved my original problem - how do we set a function’s actions call allow option in a role?
How do we migrate LOTS of rules from FQL4 to to the new format?
There are v4 to v10 migration documents are here (How to Migrate Applications from Fauna v4 to v10 - Fauna Documentation) but it is very basic, and while it points to the v10 docs, the v10 docs don’t mention how to set a function’s permissions within a role (specifically).
You see, it turns out they actually do, but it’s not what you think.
I thought the word FUNCTION has to be in their somewhere:
Function('fnc_name')
Function(name:'fnc_name')
"[function fnc_name]"
but it’s just "fnc_name" all by its lonesome - how obvious in hindsight - but troubling for the future.
A collection Users and a function Users in the schema will collide - right? Or is there some other option/flag to be used to get around this?
I finally sorted out my role, and used the v10 dashboard and FSL. It looks like this:
role backend_role {
privileges "register_with_verification" { call }
privileges "make_email_verification_token_from_email" { call }
privileges "login_call_limited" { call }
privileges "login_account" { call }
privileges "request_password_reset" { call }
}
Using FQL I tried this - which also worked:
Role.create({"name": "backend_role2","privileges": [{"resource": "register_with_verification", "actions": {"call": true}}, {"resource": "make_email_verification_token_from_email", "actions": {"call": true}}, {"resource": "login_call_limited", "actions": {"call": true}}, {"resource": "login_account", "actions": {"call": true}}, {"resource": "request_password_reset","actions": {"call": true}}]})
What I found interesting is that while the original FQL had "make_email_verification_token_from_email" the Role viewer when opening the role backend_role2 had it as make_email_verification_token_from_email without inverted commas, and the code colouring in the viewer was better, so no need to use “” on names of things it seems.
So you made it this far, it took far too long to get here didn’t it?
If you have been having the same problems I had, I hope this helps you.
I am wondering how many other developers have been caught out by this.
This journey has made me REALLY concerned about the future of Fauna. I’m not talking about in terms of Fauna doing well - it’s a wonderful platform, and it is having growing pains due to lots of changes, but the changes are a bit too painful. I literally have to re-write half a system I spent two years writing because I chose Fauna. I would be finished by now if I had gone with SQL. I had to use a database accessible via HTTPS because Cloudflare didn’t support SQL access (at the time the project started); and Fauna felt like a godsend - because it has relational properties.
While FQL v4 is here, it will be gone eventually, and no doubt Fauna will give notice of this - but given the poor communication over the past year of all these changes, I fear this change could be missed by many developers, and given the work required to update - it could prove deadly to a lot of sites that don’t upgrade to v10 immediately.
My questions / requests are:
-
Where is the documentation on migrating a database?
Why is there not a SINGLE web page, just one, that properly lists all the tasks needed, the common gotchas, and how best to proceed with links to more reading in how to move from ? -
How many sites are going to be in for a painful update when Fauna pull v4 support (because it will happen eventually)
Given the way that information has been communicated up till now, there are going to be plenty of developers that will not be in the know that its going to happen until it suddenly does.
So I urge Team Fauna - please sort out your documentation, please be more upfront about the migration pains, please provide single web page with a full overview of the planned upgrade path so that we know how best to proceed with using Fauna in the future.
Thank you.
but I’ll need your account info to get you situated.
