Receiving Socket Hang Up error often

Help
#1

Hello, I’m getting the following error message quite frequently:

FetchError: request to https://db.fauna.com/ failed, reason: socket hang up

I’m calling Fauna from an EC2 using the Fauna NPM package and have been creating a new Fauna client each time before making a request to Fauna.

There seems to be no pattern to the error. It seems to happen when I try to create a handful of writes within quick succession e.g. maybe 30 writes within 10 seconds of each other, but only occasionally – not every time. I don’t think this has anything to do with the known issue of Lambda thawing, as requests to Fauna are being made from a server.

Does anyone know what’s going on here?

#2

Hi @robertsamarji We are not aware of any issue which could this. Can you please share the code of how you are creating the Fauna Client?

#3

Hi @Jay-Fauna, thank you for replying, and sure. This is the code:

const faunadb = require('faunadb')
const q = faunadb.query
const { Create, Collection } = q

const Fauna = {}
const serverClient = new faunadb.Client({ secret: process.env.FAUNA_KEY_SECRET })

Fauna.createEntry = async (entry) => {
  try {
    await serverClient.query(
      Create(
        Collection('entries'),
        { data: entry }
      )
    )
  } catch (e) {
    console.log('createEntry error', e)
  }
}

Previously, I also created the serverClient variable inside the createEntry function. In both cases I am receiving a socket hang up error on occasion and the odd timeout error.

My environment is:

AWS EC2
App kept up with PM2
Node 8.9.4
faunadb ^3.0.1

#4

Hi @Jay-Fauna
I also have this socket issue
I use it with

  • Node v12.19.0
  • faunadb ^4.0.3

When I run my API with Express (API <-> FaunaDB) on the desktop it happens once in a while, but when another developer runs it on a laptop it happens more often. Most errors we see when run 5,000-10,000 calls to Fauna to read by index.
Maybe these details will help to improve the client code.

#5

Do you mean 5k-10k call in parallel?
Is there any other pattern to it that you noticed, e.g. bigger transactions? Highly dependent transactions? (no problem if not, just trying to get more information).

I currently have a gut-feeling that there is something wrong with node-fetch but it might be something else. :thinking:

#6

Hi @databrecht, I can’t speak for @ygrik, but I see socket hang up and other timeout errors when making calls that are not in parallel with any others - just simply on occasion with no obvious pattern. Mine are write calls and can either be large or even very small (3 - 4 properties) transactions.

#7

Pasting the most common error here:

{
FetchError: request to https://db.fauna.com/ failed, reason: socket hang up
at ClientRequest. (/home/ubuntu/accepty/node_modules/cross-fetch/node_modules/node-fetch/lib/index.js:1461:11)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at TLSSocket.socketOnEnd (_http_client.js:423:9)
at emitNone (events.js:111:20)
at TLSSocket.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1055:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
message: ‘request to https://db.fauna.com/ failed, reason: socket hang up’,
type: ‘system’,
errno: ‘ECONNRESET’,
code: ‘ECONNRESET’
}

#8

Can you please share how the Fauna.createEntry function is used as well? Errors like that are frequently caused by not waiting on Promises.

#9

@wallslide, ah, that may be it, I’m not actually awaiting it as I’m not using the response for anything, just calling it and letting any other code below execute:

const entry = { ...someData } 
Fauna.createEntry(entry)
const continuing = withOtherStuff

Must I await the execution?

#10

Another thing of importance to know is, did your transaction get applied or did it abort / never got presented to Fauna? That last part you probably can’t verified easily but there might be multiple things that go wrong here where some might be purely happening in the node client.

#11

Hey @databrecht, the transactions were never applied.

As a workaround, I since setup a retry mechanism to ensure the transactions were eventually applied. If there was a socket hang up error (or less commonly, “FetchError: network timeout at: https://db.fauna.com/ at Timeout”), the transaction would usually go through on the second try.

#12

We tested a client make requests to API with 10 parallel requests.
We run static content generation for 10 pages in parallel. We send get requests to load content parts (JSON) for each page.
We found an interesting dependency, we remove output logs to console.log in API (Node.JS/Express <-> Fauna) and this reduced the number of socket issues.

#13

That’s fascinating since I also had a similar experience in my tests to try and reproduce this. In my case I was trying to log whether something went out/came in by instrumenting the fetch code to send the request to fiddler. What happened? The error went away completely. What that means… is uncertain to me, but I documented everything in detail. Both your answers will help the driver team get more information since this is definitely a NodeJS issue, in my case the error didn’t happen when I did the exact same thing from the browser.

I have also noticed that sometimes when you draw too much to the log, more socket hangups happen. Almost seems as if some logic could be stealing the resources necessary to keep the socket alive.

#14

Hope you can fix it soon :slight_smile: or find stable workaround to reduce this socket issue

#15

It depends on the environment you are running in. In many lambda/serverless function environments, as soon as you send your response to a request, any further execution is stopped. So if you didn’t wait on the promise before sending your response, then its connection will be terminated before it gets a chance to complete.

#16

Ah yes, I’m aware of this. I’m not awaiting the Fauna call because I’m running it within an app on an EC2 which is being kept up by PM2.

#17

We’re looking into it. Apparently we are not the only ones running into this problem. intermittent network errors ( socket hang up ) · Issue #74 · MichaelDeBoey/gatsby-remark-embedder · GitHub
From what I’ve seen I would be surprised that it’s network related though.

#18

@databrecht we do not use Gatsby, we build SaaS for our own static site generator and content builder. We have socket issues only with Fauna.

#19

I realize that, I’m just saying it seems to be related to node-fetch and that we are not the only company who have issues around node-fetch. I have noticed similar issues when using my library. In that case it seems to be on bigger payloads though. Socket hang up on bigger payloads · Issue #5 · fauna-brecht/fauna-schema-migrate · GitHub which is still different than these issues.