networking seems very slow causing timeouts

[phishy]

First off, Love Bref. We successfully migrated with it but we are having some pretty major issues somehow, probably not Bref's fault, but maybe you have an idea. I've been stumped for a week.

Currently using the 7.3 build.

We're in a VPC, used the serverlesss plugin, we're about to make calls to the outside no problem. However it seems that networking is very slow. Our requests take at least 4x longer than local. Successive calls to Mongo call timeouts. It's also on AWS in the same region. Seems all like networking stuff. I've moved it out the VPC, same.

Last night we had what appeared to be a ton of networking errors.

Any ideas?

[mnapoli]

Hey, one idea that comes to mind: are Mongo and Redis running in a different availability zone when the timeout happens?

Depending on your case, you could running everything in one AZ (not redundant though), or have Redis + Mongo run in all AZ (redundant, but probably more expensive).

Our requests take at least 4x longer than local

Just so we get a better idea, what timing are we talking about?

[phishy]

Our API isn't that speedy. But on my local a call to an endpoint in about 300ms. On Lambda not matter how many CPUs, it takes 1.3 seconds. Our Mongo runs in Atlas. I'll have to figure out what AZ its in.

[deleugpn]

Name or Service not known is usually caused by complete lack of internet connection. A slow connection would still give you a timeout, not a complete lack of connectivity.
Most likely you may have issues with ACL, Security Group or Route Tables on your AWS setup

[phishy]

This is prod. It runs. It just freaks out under load. ¯_(ツ)_/¯

[phishy]

Also, I used the serverless VPC plugin, so it's whatever defaults they are. However, as I mentioned, I've also runs this outside a VPC, same results. ¯_(ツ)_/¯

[deleugpn]

If we're talking heavy load, this could be helpful: laravel/framework#35790

[phishy]

Haha not "heavy" load per se, but I can't produce it with wrk. Thanks for this!

[phishy]

The unfortunate part about that article, is that I've never really seen this happen under ECS. So I'm not sure if transient network issues are a Lambda thing. ¯_(ツ)_/¯

[phishy]

Thanks for the tips guys. We moved back to ECS for now, until we can figure out what's going on.