Start Over Error

[ghost]

I am using the current version of the FBCTF. The older version from last didn't seem to have this problem. When the system is under a mild load (12 users) after about 15 minutes of play all users start to experience a "Start Over" error. The problem progressively gets worse until all users / players can not clear the Start Over error. Any advice? I can't recreate it with a single user on my personal network.

[ghost]

would love some comments on this. Nothing conclusive but I believe the issue is aggravated by two items 1) A larger number of Countries assigned challenges. If I stay below 70 or 75 the problems seems to go away 2) Network latency over wireless combined with a large number of challenges / countries.

[v1ncenth]

Hi gjemerick, can you try the latest dev-lite branch version and see if that works?

[punkdis]

I am getting this error too. It is quite troublesome as even with increasing VM resources the problem still occurs. What do you mean latest dev-lite branch version?

[punkdis]

To add I looked into the logs for the HHVM server and discovered I was getting the following errors for "Too many connections" I increased the sql configuration "max_connections = 100" to "max_connections = 1000" and going to retest.

From the Troubleshooting and Performance Guide

Large Events: Increase MySQL max connections
By default, MySQL is set to a connection limit of around 100. It is recommended that you increase the value for larger events. To increase the MySQL connection limit, edit the MySQL configuration file:

sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf
Locate the below line, ensure it is not commented out, and edit the max connections as desired. Note that a single user has approximately two connections at any given time. Save the MySQL configuration file.

max_connections = 100
Restart the MySQL process with the following command:

sudo service mysql restart

Errror log
[Fri Oct 11 02:51:44 2019] [hphp] [1616:7f04f47ff700:736588:000001] [] \nFatal error: Uncaught exception 'AsyncMysqlConnectException' with message 'Error executing AsyncMysql operation: Failed (mysql error: 1040: Too many connections)' in /var/www/fbctf/src/data/controller.php:9\nStack trace:\n#0 /var/www/fbctf/src/data/controller.php(9): HH\Asio\join()\n#1 /var/www/fbctf/src/data/configuration.php(47): DataController->sendData()\n#2 {main}

[Fri Oct 11 02:51:44 2019] [hphp] [1616:7f04e63ff700:741885:000001] [] \nFatal error: Uncaught exception 'AsyncMysqlConnectException' with message 'Error executing AsyncMysql operation: Failed (mysql error: 1040: Too many connections)' in /var/www/fbctf/src/data/controller.php:9\nStack trace:\n#0 /var/www/fbctf/src/data/controller.php(9): HH\Asio\join()\n#1 /var/www/fbctf/src/data/configuration.php(47): DataController->sendData()\n#2 {main}

[Fri Oct 11 02:51:44 2019] [hphp] [1616:7f04ff3ff700:729503:000001] [] \nFatal error: Uncaught exception 'AsyncMysqlConnectException' with message 'Error executing AsyncMysql operation: Failed (mysql error: 1040: Too many connections)' in /var/www/fbctf/src/data/controller.php:9\nStack trace:\n#0 /var/www/fbctf/src/data/controller.php(9): HH\Asio\join()\n#1 /var/www/fbctf/src/data/configuration.php(47): DataController->sendData()\n#2 {main}

[Fri Oct 11 02:51:44 2019] [hphp] [1616:7f04f93ff700:738162:000001] [] \nFatal error: Uncaught exception 'AsyncMysqlConnectException' with message 'Error executing AsyncMysql operation: Failed (mysql error: 1040: Too many connections)' in /var/www/fbctf/src/inc/gameboard/modules/controller.php:9\nStack trace:\n#0 /var/www/fbctf/src/inc/gameboard/modules/controller.php(9): HH\Asio\join()\n#1 /var/www/fbctf/src/inc/gameboard/modules/activity.php(93): ModuleController->sendRender()\n#2 {main}

[Fri Oct 11 02:51:44 2019] [hphp] [1616:7f04f47ff700:736590:000001] [] \nFatal error: Uncaught exception 'AsyncMysqlConnectException' with message 'Error executing AsyncMysql operation: Failed (mysql error: 1040: Too many connections)' in /var/www/fbctf/src/data/controller.php:9\nStack trace:\n#0 /var/www/fbctf/src/data/controller.php(9): HH\Asio\join()\n#1 /var/www/fbctf/src/data/configuration.php(47): DataController->sendData()\n#2 {main}

[Fri Oct 11 02:51:44 2019] [hphp] [1616:7f04ffbff700:740443:000001] [] \nFatal error: Uncaught exception 'AsyncMysqlConnectException' with message 'Error executing AsyncMysql operation: Failed (mysql error: 1040: Too many connections)' in /var/www/fbctf/src/data/controller.php:9\nStack trace:\n#0 /var/www/fbctf/src/data/controller.php(9): HH\Asio\join()\n#1 /var/www/fbctf/src/data/configuration.php(47): DataController->sendData()\n#2 {main}

[Fri Oct 11 02:51:44 2019] [hphp] [1616:7f04e5bff700:737058:000001] [] \nFatal error: Uncaught exception 'AsyncMysqlConnectException' with message 'Error executing AsyncMysql operation: Failed (mysql error: 1040: Too many connections)' in /var/www/fbctf/src/data/controller.php:9\nStack trace:\n#0 /var/www/fbctf/src/data/controller.php(9): HH\Asio\join()\n#1 /var/www/fbctf/src/data/map-data.php(77): DataController->sendData()\n#2 {main}

[punkdis]

Resolved this error with adjusting the max_connections = 1000 and max_user_connections = 1000

[punkdis]

Now I am getting this error

[Tue Oct 29 15:17:01 2019] [hphp] [1692:7f4247fff700:74255:000001] [] \nFatal error: Uncaught exception 'AsyncMysqlConnectException' with message 'Error executing AsyncMysql operation: TimedOut (mysql error: 2013: [7000](MySQL Client) Connect to 192.168.0.4:3306 timed out (took 999.31ms))' in /var/www/fbctf/src/data/controller.php:9\nStack trace:\n#0 /var/www/fbctf/src/data/controller.php(9): HH\Asio\join()\n#1 /var/www/fbctf/src/data/captures.php(28): DataController->sendData()\n#2 {main}

[punkdis]

I have adjusted the following variables and going to retest.

wait_timeout = 28800
connect_timeout = 28800
interactive_timeout = 28800
net_read_timeout = 100

[ghost]

Did your last adjustments resolve the errors? I have tried the last set of configuration changes. I did find that flushing memcache at least temporarily removed the error.