After update from swoole 5.1.3 to swoole 5.1.4 server starts to randomly respond 503 status

[volodymyr-hordiienko]

After update from swoole 5.1.3 to swoole 5.1.4 server starts to randomly respond 503 status and after heavy load starts to respond 503 everytime even there is all good with resources.

Downgrade to swoole 5.1.3 fix the problem.

Projects uses hyperf and runs on hyperf/hyperf:8.3-alpine-v3.19-swoole-v5.1.3, upgrade was on hyperf/hyperf:8.3-alpine-v3.20-swoole-v5.1.4

[hydrapolic]

Hello, do you use curl and coroutines? It may be related to curl, as alpine 3.20 switched to curl 8.10.1 and this was already reported curl/curl#15127. Alpine 3.19 ships an older curl version.

[volodymyr-hordiienko]

@hydrapolic

Thanks a lot for the reply and the direction!

Yeah, actually i used Hyperf\Guzzle client a lot with coroutines, not sure it uses exactly curl under, but its possible. Strange its happening silently, just 503 on all http requests, even requests that not use guzzle. after server restarts it works some time well, but more and more 503 happening untill all server stucks with 503... in the app logs there no issues, amqp consumers works, background tasks also etc, just http request affected.

Well, is it means we need to wait till swoole supports curl 8.10.1 before alpine upgrade?

I know i need to provide more details, but i have no any logs with errors... just do not know what info could be provided here :) could not provide code that 100% reproduce issue, it reproduces only on huge amounts of requests, locally works fine

[NathanFreeman]

This seems to be an issue caused by the last PR related to process restart and resetting some global variables. Please wait a moment, and I will fix it shortly.

[matyhtf]

Has the server been configured with the max_concurrency option ? @volodymyr-hordiienko

[hydrapolic]

@hydrapolic

Thanks a lot for the reply and the direction!

Yeah, actually i used Hyperf\Guzzle client a lot with coroutines, not sure it uses exactly curl under, but its possible. Strange its happening silently, just 503 on all http requests, even requests that not use guzzle. after server restarts it works some time well, but more and more 503 happening untill all server stucks with 503... in the app logs there no issues, amqp consumers works, background tasks also etc, just http request affected.

Well, is it means we need to wait till swoole supports curl 8.10.1 before alpine upgrade?

I know i need to provide more details, but i have no any logs with errors... just do not know what info could be provided here :) could not provide code that 100% reproduce issue, it reproduces only on huge amounts of requests, locally works fine

Actually we discovered this on OpenSwoole (v22.1.2). Our CI pipeline started to fail 1.10.2024 with signal 11 kills, it was the time when newer curl was added to Alpine:
https://gitlab.alpinelinux.org/alpine/aports/-/commit/e17ad295e0810e938f9c4d61c21e44eaf1a59b51
https://gitlab.alpinelinux.org/alpine/aports/-/commit/badfda8e31a0b6e87e78237d7ee4455b35cd4c10

After switching back to Alpine 3.19 and Curl 8.9.1, the segfaults were gone instantly.

[volodymyr-hordiienko]

@matyhtf No, its not, here is my server config

declare(strict_types=1);
/**
 * This file is part of Hyperf.
 *
 * @link     https://www.hyperf.io
 * @document https://hyperf.wiki
 * @contact  [email protected]
 * @license  https://github.com/hyperf/hyperf/blob/master/LICENSE
 */
use Hyperf\Server\Event;
use Hyperf\Server\Server;
use Swoole\Constant;
return [
    'mode' => SWOOLE_PROCESS,
    'servers' => [
        [
            'name' => 'http',
            'type' => Server::SERVER_HTTP,
            'host' => '0.0.0.0',
            'port' => 80,
            'sock_type' => SWOOLE_SOCK_TCP,
            'callbacks' => [
                Event::ON_REQUEST => [Hyperf\HttpServer\Server::class, 'onRequest'],
            ],
            'options' => [
                'enable_request_lifecycle' => true,
            ],
        ],
        [
            'name' => 'socket-io',
            'type' => Server::SERVER_WEBSOCKET,
            'host' => '0.0.0.0',
            'port' => 8001,
            'sock_type' => SWOOLE_SOCK_TCP,
            'callbacks' => [
                Event::ON_HAND_SHAKE => [Hyperf\WebSocketServer\Server::class, 'onHandShake'],
                Event::ON_MESSAGE => [Hyperf\WebSocketServer\Server::class, 'onMessage'],
                Event::ON_CLOSE => [Hyperf\WebSocketServer\Server::class, 'onClose'],
            ],
        ],
    ],
    'settings' => [
        Constant::OPTION_LOG_LEVEL => 4,
        Constant::OPTION_ENABLE_COROUTINE => true,
        Constant::OPTION_WORKER_NUM => swoole_cpu_num(),
        Constant::OPTION_PID_FILE => BASE_PATH . '/runtime/hyperf.pid',
        Constant::OPTION_OPEN_TCP_NODELAY => true,
        Constant::OPTION_MAX_COROUTINE => 100000,
        Constant::OPTION_OPEN_HTTP2_PROTOCOL => true,
        Constant::OPTION_MAX_REQUEST => 100000,
        Constant::OPTION_SOCKET_BUFFER_SIZE => 2 * 1024 * 1024,
        Constant::OPTION_BUFFER_OUTPUT_SIZE => 2 * 1024 * 1024,
        Constant::OPTION_DOCUMENT_ROOT => BASE_PATH . '/storage',
        Constant::OPTION_STATIC_HANDLER_LOCATIONS => ['/public'],
        Constant::OPTION_ENABLE_STATIC_HANDLER => true,
        Constant::OPTION_HTTP_COMPRESSION => true,
        Constant::OPTION_HTTP_COMPRESSION_LEVEL => 4,
    ],
    'callbacks' => [
        Event::ON_WORKER_START => [Hyperf\Framework\Bootstrap\WorkerStartCallback::class, 'onWorkerStart'],
        Event::ON_PIPE_MESSAGE => [Hyperf\Framework\Bootstrap\PipeMessageCallback::class, 'onPipeMessage'],
        Event::ON_WORKER_EXIT => [Hyperf\Framework\Bootstrap\WorkerExitCallback::class, 'onWorkerExit'],
    ],
];