Slow reposonse time #1382

vedmant · 2023-01-27T00:31:39Z

vedmant
Jan 27, 2023

I'm curious why response time might be slower, I tested my app (that is currently deployed on Heroku on the cheapest dyno) on serverless with bref, both apps use the same RDS.

When I do tests I see that serverless in average is twice slower, so requests that take around 400ms on Heroku, take 800ms on serverless. Endpoints that take 200ms on Heroku on Serverless take 400-500ms. This numbers are taken from my browser.

Is there any way to improve the performance, my laravel endpoints are quite database intense, but both deployments use exactly the same DRS, so database connection should not be a bottleneck, then something else is limiting performace on Serverless significantly.

mnapoli · 2023-01-27T16:15:29Z

mnapoli
Jan 27, 2023
Maintainer

Hi! Could you share your serverless.yml? (with sensitive parts redacted) Things we could check: memory size, which runtime you are using, etc.

Also can you confirm that everything (serverless app + RDS) is running in the same AWS region?

And could you detail how you measured those differences? From your machine? Do you account for cold starts? (which metric do you check: if you check the average then the cold stars might be messing up the average, look at p50 instead with a large number of requests)

How do you connect to the database? SSL checks?

0 replies

vedmant · 2023-01-27T16:39:14Z

vedmant
Jan 27, 2023
Author

@mnapoli Yes, sure:

service: my-test-project

provider:
    name: aws
    # The AWS region in which to deploy (us-east-1 is the default)
    region: us-east-1
    # The stage of the application, e.g. dev, production, staging… ('dev' is the default)
    stage: dev
    runtime: provided.al2
    environment:
      APP_KEY: ${env:APP_KEY}
      APP_ENV: ${env:APP_ENV}
      APP_DEBUG: ${env:APP_DEBUG}
      APP_URL: ${env:APP_URL}
      FRONTEND_URL: ${env:FRONTEND_URL}
      CORS_URLS: ${env:CORS_URLS}
      DB_CONNECTION: ${env:DB_CONNECTION}
      DATABASE_URL: ${env:DATABASE_URL}
 
package:
    # Directories to exclude from deployment
    patterns:
        - '!node_modules/**'
        - '!public/storage'
        - '!resources/assets/**'
        - '!storage/**'
        - '!tests/**'

functions:
    # This function runs the Laravel website/API
    web:
        handler: public/index.php
        timeout: 28 # in seconds (API Gateway has a timeout of 29 seconds)
        layers:
            - ${bref:layer.php-81-fpm}
        events:
            - httpApi: '*'
    # This function lets us run artisan commands in Lambda
    artisan:
        handler: artisan
        timeout: 120 # in seconds
        layers:
            - ${bref:layer.php-81} # PHP
            - ${bref:layer.console} # The "console" layer
        events:
          - schedule:
              description: Running the Laravel Scheduler (schedule:run) each minute
              rate: rate(1 minute)
              input:
                cli: schedule:run

plugins:
    # We need to include the Bref plugin
    - ./vendor/bref/bref

Also can you confirm that everything (serverless app + RDS) is running in the same AWS region?

Yes, it's in the same AWS region us-east-1 for DRS and Lambda functions. Actually my Heroku app is on us-east-2 so Heroku supposed to be slower with database, but it's faster in fact.

And could you detail how you measured those differences? From your machine?

Yes, from my machine, just looking for initial document response time. Most of the time is waiting time.
Here you can test it yourself:
Serverless: https://89ttor5z5h.execute-api.us-east-1.amazonaws.com/api/boards/wsf246/effects-of-screen-time-on-children.
Heroku: https://project-credo-backend-staging.herokuapp.com/api/boards/wsf246/effects-of-screen-time-on-children

I see a bit difference on download time due to Heroku gzips data, but wait time is almost twice bigger on Serverless anyway.

Do you account for cold starts?

I run test repeatedly multiple times manually:

How do you connect to the database? SSL checks?

Not sure, just with a DATABASE_URL on a public url, configuration is the same on Heroku and Serverless.

1 reply

mnapoli Jan 27, 2023
Maintainer

Got it, thanks for all the details that's helpful. That all looks good so far.

Just one more question:

      APP_ENV: ${env:APP_ENV}
      APP_DEBUG: ${env:APP_DEBUG}

This will deploy with the env variables you currently have defined. Could you confirm that this isn't deployed in "dev" environment? (e.g. by checking in the AWS console by opening the Lambda function)

It might also be worth checking that Composer dev dependencies are not installed/deployed, but I don't expect that to be the source of the problem here.

Yes, it's in the same AWS region us-east-1 for DRS and Lambda functions. Actually my Heroku app is on us-east-2 so Heroku supposed to be slower with database, but it's faster in fact.

Indeed that's very surprising! Especially since the difference appears to be significant, I find that very confusing.

I tried a few requests from Europe and the difference was not that stark (kind of random at times). Since you are benchmarking from your laptop, do you know if you are closer to us-east-2 or us-east-1? Maybe that geographical difference comes into play 🤔

The ultimate benchmark would be to benchmark from a server running in us-east-1. Or alternatively to look at the response time on the server (for Heroku) and the response time from API Gateway (in CloudWatch metrics). That way network time is completely eliminated from the question.

vedmant · 2023-01-27T23:59:03Z

vedmant
Jan 27, 2023
Author

@mnapoli

This will deploy with the env variables you currently have defined. Could you confirm that this isn't deployed in "dev" environment? (e.g. by checking in the AWS console by opening the Lambda function)

I used
APP_ENV=production
APP_DEBUG=true

I just tried to deploy with APP_DEBUG=false but don't see any difference.

It might also be worth checking that Composer dev dependencies are not installed/deployed, but I don't expect that to be the source of the problem here.

I installed deps with --no-dev.

I tried a few requests from Europe and the difference was not that stark (kind of random at times). Since you are benchmarking from your laptop, do you know if you are closer to us-east-2 or us-east-1? Maybe that geographical difference comes into play

I tried from multiple VPNs, USA, Europe, and it's the same everywhere.

Everywhere Serverless is slower 1.5-2 times.

I also noticed significant delays in my frontend Nuxt app compared to deploying it directly on EC2 instance. Not sure why, but serveless in all cases have some additional latency.

For example this app is deployed to Serverless, takes about 400-500ms to load: https://www.projectcredo.com/about
I added cheapest EC2 and deployed the same app, and load time is 100-120ms: http://100.26.161.88/about
About page doesn't do any API requests, just simple Vue app SSR rendering.
So there is some significant overhead with Serverless that adds 300-400ms to load times.

2 replies

mnapoli Jan 28, 2023
Maintainer

So there is some significant overhead with Serverless that adds 300-400ms to load times.

I am afraid this is specific to your case for some reason. We just don't understand why yet.

I tried from multiple VPNs, USA, Europe, and it's the same everywhere.

I don't think a VPN will help at all to benchmark. Indeed, the requests need to go from the VPN to the server but also from you to the VPN, so your geographic location matters. That's why I think you need to either benchmark from inside the same region (in EC2), or read the server response time to exclude all network overhead.

How much memory and CPU is there on the Heroku app? Do you use any cache there? E.g. APCu? Anything specific in php.ini?

I installed deps with --no-dev.

--optimize-autoloader too?

Since this seems to be a Laravel app, do you run the same commands (e.g. config:cache or other) before deploying?

Have you also tried with endpoints that are not database-intensive?

So far we don't know "what" is slow, so I think it's all premature to come to conclusions until we know what it is (network, DB connection, PHP code, etc.).

vedmant Jan 29, 2023
Author

I don't think a VPN will help at all to benchmark. Indeed, the requests need to go from the VPN to the server but also from you to the VPN, so your geographic location matters. That's why I think you need to either benchmark from inside the same region (in EC2), or read the server response time to exclude all network overhead.

Yeah, but time from me to VPN will not change depending on the host, so that will be a constant value, this can still show difference in VPN - host connection.

Interesting, I tested today and now the difference is less significant:

(base) ➜  ~ for i in {1..20}; do curl -s -o /dev/null -w "%{time_starttransfer}\n" https://project-credo-backend-staging.herokuapp.com/api/boards/wsf246/effects-of-screen-time-on-children ; done
0.494594
0.483662
0.516502
0.502632
0.437033
0.430409
0.522463
0.468106
0.572133
0.472297
0.445863
0.433032
0.463277
0.459315
0.455768
0.464150
0.500207
0.516107
0.486286
0.548850
Average	x =	0.4836343

(base) ➜  ~ for i in {1..20}; do curl -s -o /dev/null -w "%{time_starttransfer}\n" https://89ttor5z5h.execute-api.us-east-1.amazonaws.com/api/boards/wsf246/effects-of-screen-time-on-children ; done
0.671460
0.638781
0.587062
0.561833
0.666870
0.728327
0.542661
0.592858
0.629417
0.662888
0.666163
0.594521
0.570552
0.647614
0.635296
0.663674
0.659241
0.665039
0.570535
0.630183
Average	x =	0.62924875

Here is also an example with VPN (Germany):

(base) ➜  ~ for i in {1..20}; do curl -s -o /dev/null -w "%{time_starttransfer}\n" https://project-credo-backend-staging.herokuapp.com/api/boards/wsf246/effects-of-screen-time-on-children ; done
1.234520
1.219893
1.166210
1.360888
1.413999
1.205935
1.139475
1.193552
1.158750
1.194016
1.258783
1.427209
1.607895
1.144273
1.185186
1.154460
1.210981
1.183926
1.234912
1.174123
Average	x =	1.2434493

(base) ➜  ~ for i in {1..20}; do curl -s -o /dev/null -w "%{time_starttransfer}\n" https://89ttor5z5h.execute-api.us-east-1.amazonaws.com/api/boards/wsf246/effects-of-screen-time-on-children ; done
1.485247
1.287614
1.840225
1.236203
1.279862
1.271111
1.389349
1.302967
1.311875
1.330082
1.299530
1.322314
1.307773
1.331888
1.308550
1.423466
1.305028
1.281585
1.343600
1.297749
Average	x =	1.3478009

So today difference is just about 100ms.

How much memory and CPU is there on the Heroku app? Do you use any cache there? E.g. APCu? Anything specific in php.ini?

Heroku has 512MB and 1 CPU. No, just standard configuration, only postgresql driver added.

--optimize-autoloader too?

Yes, as in example in docs.

Since this seems to be a Laravel app, do you run the same commands (e.g. config:cache or other) before deploying?

No, I didn't do any Laravel optimizations on both.

Have you also tried with endpoints that are not database-intensive?

No, let me try. Here is much less DB intense endpoint:

(base) ➜  ~ for i in {1..20}; do curl -s -o /dev/null -w "%{time_starttransfer}\n" https://project-credo-backend-staging.herokuapp.com/api/profile/Kelas ; done
0.418045
0.287421
0.297474
0.269109
0.266953
0.277265
0.276279
0.279443
0.286998
0.294813
0.287135
0.306868
0.281700
0.265407
0.279482
0.272230
0.269807
0.269491
0.268484
0.269931
Average	x =	0.28621675

for i in {1..20}; do curl -s -o /dev/null -w "%{time_starttransfer}\n" https://89ttor5z5h.execute-api.us-east-1.amazonaws.com/api/profile/Kelas ; done
0.467980
0.373131
0.410371
0.373194
0.372872
0.356009
0.342419
0.360325
0.362589
0.372752
0.384602
0.367005
0.350796
0.358029
0.356809
0.444514
0.350439
0.348581
0.356059
0.366877
Average	x =	0.37376765

And with VPN (Germany):

for i in {1..20}; do curl -s -o /dev/null -w "%{time_starttransfer}\n" https://project-credo-backend-staging.herokuapp.com/api/profile/Kelas ; done
1.176759
1.019473
0.967785
1.002679
0.961354
1.013644
0.979171
0.999446
0.989735
0.970734
0.999109
1.015893
0.977265
1.032165
0.997471
1.154794
0.977612
0.991935
0.977506
0.977287
Average	x =	1.00909085

for i in {1..20}; do curl -s -o /dev/null -w "%{time_starttransfer}\n" https://89ttor5z5h.execute-api.us-east-1.amazonaws.com/api/profile/Kelas ; done
1.231176
1.089008
1.092062
1.084145
1.070999
1.102134
1.076613
1.075499
1.095961
1.066860
1.082991
1.134478
1.058585
1.086798
1.091136
1.117631
1.067221
1.049723
1.072042
1.072689
Average	x =	1.09088755

I see less significant difference, but still Serverless is slower. So could this be a Serverless - Database connections speed issue?

So far we don't know "what" is slow, so I think it's all premature to come to conclusions until we know what it is (network, DB connection, PHP code, etc.).

I did some more test, enabled telescope and what I see is that on Serverless database requests are around 1.5-2x slower, on Heroku I see 60-80ms, on Serverless 120-130ms.
Here are some screenshots:
Serverless:

Heroku:

I'm really curious why RDS works slower with serverless than with Heroku, I actually double checked and it looks like all instances are on the same region (RDS, Lambda and Heroku).

tigitz · 2023-01-28T18:24:38Z

tigitz
Jan 28, 2023

Since it seems like a Laravel app, you could try with the debug bar (https://github.com/barryvdh/laravel-debugbar) enabled and check the profile of your requests.

My money is on I/O bound issues.

If nothing stands out of you could still use APM tools to go deeper like blackfire or new relic.

1 reply

vedmant Jan 29, 2023
Author

Since it seems like a Laravel app, you could try with the debug bar (https://github.com/barryvdh/laravel-debugbar) enabled and check the profile of your requests.

I checked with Telescope, looks like database is slower with Serverless than with Heroku instance (all are in the same region).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Slow reposonse time #1382

{{title}}

Replies: 4 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Slow reposonse time #1382

vedmant Jan 27, 2023

Replies: 4 comments · 4 replies

mnapoli Jan 27, 2023 Maintainer

vedmant Jan 27, 2023 Author

mnapoli Jan 27, 2023 Maintainer

vedmant Jan 27, 2023 Author

mnapoli Jan 28, 2023 Maintainer

vedmant Jan 29, 2023 Author

tigitz Jan 28, 2023

vedmant Jan 29, 2023 Author

vedmant
Jan 27, 2023

Replies: 4 comments 4 replies

mnapoli
Jan 27, 2023
Maintainer

vedmant
Jan 27, 2023
Author

mnapoli Jan 27, 2023
Maintainer

vedmant
Jan 27, 2023
Author

mnapoli Jan 28, 2023
Maintainer

vedmant Jan 29, 2023
Author

tigitz
Jan 28, 2023

vedmant Jan 29, 2023
Author