Slow response and 403

[fponticelli]

I have two issues that are possibly related but I am not 100% sure they are.

1. Every time a call is made to any endpoint, the service returns the expected payload but this error shows in CloudWatch:

I looked up that 403 and I couldn't find anything really helpful (I am aware of issue #385 and my payload is really not big at all). This error also happens on the default 404 generated by Symfony. Again, I get a response on the client and the error is silently ignored.

2. To generate any output, even calls that do not make DB calls, it takes ~8s. I am aware of the warm up and I have it configured but this seems to exceed expectations of a cold start. In Kernel I tried to play with both getCacheDir and getWritableCacheDirectories with no luck.

I also enabled Sentry to see if I could gather more info but there is not a lot to go by there:

Sentry doesn't seem to be aware of the error 1. at all.

A bit of context:

• I inherited this codebase with minimal handover
• Terraform is used to deploy to AWS Lambda
• I upgraded from php-81-fpm to arn:aws:lambda:us-east-1:534081306603:layer:php-83-fpm:31 and getting the same behavior
• This is a Symfony API

I have been stuck on this for days now and any help is greatly appreciated! Thank you!

[fponticelli]

A more readable stack trace:

48f3b46a-fb64-4ec4-b94a-8f50b783b556	Invoke Error	
{
    "errorType": "Exception",
    "errorMessage": "Error 403 while calling the Lambda runtime API: InvalidStateTransition: State transition from RuntimeResponseSentState to InvocationResponse failed for runtime. Error: State transition is not allowed",
    "stack": [
        "#0 /var/task/vendor/bref/bref/src/Runtime/LambdaRuntime.php(190): Bref\\Runtime\\LambdaRuntime->postJson('http://127.0.0....', Array)",
        "#1 /var/task/vendor/bref/bref/src/Runtime/LambdaRuntime.php(96): Bref\\Runtime\\LambdaRuntime->sendResponse('48f3b46a-fb64-4...', Array)",
        "#2 /var/task/vendor/bref/bref/src/FpmRuntime/Main.php(46): Bref\\Runtime\\LambdaRuntime->processNextEvent(Object(Bref\\FpmRuntime\\FpmHandler))",
        "#3 /opt/bref/bootstrap.php(17): Bref\\FpmRuntime\\Main::run()",
        "#4 {main}"
    ]
}

and

PHP Fatal error:  Uncaught Exception: Error 403 while calling the Lambda runtime API: InvalidStateTransition: State transition from RuntimeResponseSentState to InvocationErrorResponse failed for runtime.
Error: State transition is not allowed in /var/task/vendor/bref/bref/src/Runtime/LambdaRuntime.php:345

[mnapoli]

Hi @fponticelli, I'll be honest this is a very weird/exotic error, something very unusual is happening somewhere 🤔

Let's rule out some possibilities (thanks for listing the layer version btw): are you on the latest Bref version? Is the app deploying a custom php.ini? or any custom config for PHP-FPM (this is complex/rare, so unlikely but just to be sure)?

Are you deploying with Docker or zip (standard way)?

Are you doing anything that could trip up PHP-FPM, like using a custom PHP extension, doing stuff async/in parallel/with subprocesses?

Anything specific in serverless.yml? Or that happens on Symfony's startup?

Are you using the Symfony bridge? Latest version?

Some tip if you really want to debug further: you can add random var_dump() statements in the vendor/bref/bref/src codebase in your project (assuming you deploy from your machine so that your changes are deployed) and see what happens in more details (e.g. what's the FPM response, etc.).

[fponticelli]

Thank you for following up.

are you on the latest Bref version?

I believe I am, composer says: "bref/bref": "^2.3" and composer.lock has 2.3.2.

Is the app deploying a custom php.ini?

This is the one.

; See https://github.com/brefphp/bref/blob/master/runtime/layers/fpm/php.ini for defaults.

; Enable extensions
extension=intl
extension=apcu
; extension=openssl

; Not needed as we use Redis internally in Lambda, possible fallback.
; Session configuration
; session.save_handler = redis
; session.save_path = ${REDIS_SESSION_URL}

; Lambda is not compatible with this behaviour.
; See https://github.com/brefphp/bref/commit/a45dcfffb7c575175e67b21cd7bdf415206f9f1f
disable_functions = fastcgi_finish_request ; Bref's default.

; PHP Core
date.timezone = "America/Denver"
default_charset = "UTF-8"
default_mimetype = "text/html"
ldap.max_links = -1

request_order = "GP"
variables_order = "GPCS"
short_open_tag = Off

; Memory limit is set via Symfony/Composer/Lambda, no need to limit here.
memory_limit = 512M

; Database driver config
pdo_mysql.cache_size = 2000

; No cache will be used as Lambda does not have persistent storage.
soap.wsdl_cache_enabled = 0
opcache.enable = 0 ; Does not work locally, Bref uses it.

; Errors config, should be passed as env var
display_errors = ${PHP_DISPLAY_ERRORS}
display_startup_errors = ${PHP_DISPLAY_STARTUP_ERRORS}
log_errors = On
log_errors_max_len = 1024
file_uploads = On
html_errors = On

any custom config for PHP-FPM (this is complex/rare, so unlikely but just to be sure)?

I don't believe so, this are the terraform variables used to compose the lambda service:

backend_api_lambda_handler              = "public/index.php"
backend_api_lambda_runtime              = "provided.al2"
backend_api_lambda_timeout              = 60
backend_api_lambda_memory_size          = 4096
backend_api_lambda_app_env              = "prod"
backend_api_lambda_app_debug            = "true"
backend_api_lambda_app_runtime          = "Runtime\\Bref\\Runtime"
backend_api_lambda_bref_loop_max        = "1"
backend_api_lambda_layers = [
  "arn:aws:lambda:us-east-1:534081306603:layer:php-83-fpm:31",
  "arn:aws:lambda:us-east-1:403367587399:layer:redis-php-83:3"
]

Are you deploying with Docker or zip (standard way)?

Terraform is creating the zip that gets uploaded to S3

Are you doing anything that could trip up PHP-FPM, like using a custom PHP extension, doing stuff async/in parallel/with subprocesses?

I do not believe to be the case but as I mentioned I am still not 100% familiar with this codebase. Anything specific that I should look for?

Anything specific in serverless.yml?

Unfortunately the project is not using serverless but terraform. I stared at those files many times and I couldn't spot anything wrong. From word of mouth this project was previously running on EC2 and later migrated to Lambda/Bref and this error has always been there and ignored.

Are you using the Symfony bridge? Latest version?

Yes to both

    "bref/symfony-bridge": "^1.0",
    "bref/symfony-messenger": "^0.5.4",

Some tip if you really want to debug further: you can add random var_dump() statements in the vendor/bref/bref/src codebase in your project (assuming you deploy from your machine so that your changes are deployed) and see what happens in more details (e.g. what's the FPM response, etc.).

I tried to do that by downloading the lambda zip, modifying its content, zipping and uploading again but unfortunately the lambda crushes entirely with the following error (even if I do not modify it at all):

{"message":"Internal Server Error"}

Unfortunately the dependencies are downloaded from the Github action script so it is tricky to inject my own debugging in the vendor folder.

Thank you a lot for helping!

[fponticelli]

The last point of uploading to lambda manually is now resolved, I was probably forgetting to zip some dotfile. I am adding var_dump liberally, so far nothing new to report.

[fponticelli]

Thanks to the support of @mnapoli we were able to figure out the issue. That was using the runtime/bref library. By removing it and using the default one (Symfony) the error was solved.
There are still performance issues but it is quite unlikely that they are related to Bref itself.