Seth Berrier's picture
Seth Berrier - Wed, 2021/11/10 - 16:59

I am experiencing very strange behavior on one of my new LAMP servers.  This may not be Turnkey specific but I figure I'd start here and see if you all have any ideas!

Here's what I'm observing:

  • When making outgoing web requests, the connection sometimes just times out or stalls indefinitely with no response.
  • Repeating the request immediately afterward usually causes it to succeed.
  • I'm not certain, but the problem looks like it might only happen with TLS connections (HTTPS)
  • This seems to be related to multiple requests going to the same server.
    • Running an 'npm install' is a good example, a few requests in and it will suddenly stall and eventually (after a good 5 minutes) time out
    • If I SIGTERM the process and try again it gets a little further and then times out again
    • As of now, I cannot get any 'npm install' to complete successfully as there are SOOO many timed-out connections (hundreds).
    • 'apt-get update' is another good example, often fails and has to be re-run (I can usually coax this into completing)

I'm kind of at a loss to explain this.  There is a chance it might be related to the larger network and not just my server but I wanted to do my best to explore software solutions before I contact the network administrators.

Any thoughts?
-----------------------

Here's an example showing this behavior.  It stalls at deb.nodesource.com (while running the nodesource repo setup script) and after I cancel it and try again manually, it works fine.  This one is VERY consistent:

+ apt-get update
Hit:1 http://security.debian.org buster/updates InRelease
Hit:2 http://deb.debian.org/debian buster InRelease                                                                
Ign:3 http://archive.turnkeylinux.org/debian buster-security InRelease                                             
Ign:4 http://archive.turnkeylinux.org/debian buster InRelease                                
Hit:5 http://archive.turnkeylinux.org/debian buster-security Release
Hit:6 http://archive.turnkeylinux.org/debian buster Release
0% [Connecting to deb.nodesource.com] *much time passes* ^C

berriers@games ~$ sudo apt-get update
[sudo] password for berriers:
Hit:1 http://security.debian.org buster/updates InRelease
Hit:2 http://deb.debian.org/debian buster InRelease                         
Get:3 https://deb.nodesource.com/node_14.x buster InRelease [4584 B]        
Get:4 https://deb.nodesource.com/node_14.x buster/main amd64 Packages [768 B]
Ign:5 http://archive.turnkeylinux.org/debian buster-security InRelease
Ign:6 http://archive.turnkeylinux.org/debian buster InRelease
Hit:7 http://archive.turnkeylinux.org/debian buster-security Release
Hit:8 http://archive.turnkeylinux.org/debian buster Release
Fetched 5352 B in 1s (3716 B/s)
Reading package lists... Done
Forum: 
Support
Tags: 
lamp
tls
timeout
apt-get
npm
Seth Berrier's picture

Seth Berrier - Wed, 2021/11/10 - 18:05

Here's a log of me trying to run 'npm install' on a complex node.js project (aren't they all).  This one does give back some error info that makes it look to me like it's struggling to resolve the server name in some way, despite having contacted it repeatedly just moments ago:

npm http fetch GET 304 https://registry.npmjs.org/xtend 69ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/fs.realpath 8ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/inflight 10ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/inherits 11ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/once 9ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/path-is-absolute 9ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/xml-parse-from-string 9ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/xml2js 10ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/global 10ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/is-function 11ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/parse-headers 8ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/p-try 8ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/wrappy 9ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/sax 12ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/xmlbuilder 13ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/min-document 11ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/process 11ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/dom-walk 3ms (from cache)
npm WARN registry Unexpected warning for https://registry.npmjs.org/: Miscellaneous Warning EAI_AGAIN: request to https://registry.npmjs.org/react failed, reason: getaddrinfo EAI_AGAIN registry.npmjs.org
npm WARN registry Using stale data from https://registry.npmjs.org/ due to a request error during revalidation.
Jeremy Davis's picture

Jeremy Davis - Wed, 2021/11/10 - 22:22

Unfortunately, it looks like the forums ate your second post.

Regardless, I note from the apt output that it's the deb.nodesource.com URL that is causing the issue. And you also note that it only seems to be a problem with https. My suspicion is that it's related to the recent expiry of the Let's Encrypt root certificate.

Following a little more online research, it seems that nodesource uses Akami CDN and they appear to have been affected. Apparently Akami had some plans to mitigate it but it seems that nodesource is still affected. I reckon that the reason why it's intermittent is because they have round robin DNS and some nodes have been fixed, but others haven't, so it depends whether you hit a broken one or a fixed one.

I suggest that you have a browse through the relevant GitHub issue. Hopefully someone has posted some potential workarounds that you can try.

I suggest that you

Add new comment