Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement Request: Allow an Exception override for the isConnectionDead() method #2057

Open
arosso opened this issue Mar 29, 2023 · 0 comments

Comments

@arosso
Copy link

arosso commented Mar 29, 2023

Just like the Exception Override Class that can be provided now to tell Hikari whether to evict a connection that is in use or not, we would like to have a similar concept for connections that are idle in the pool.

Currently the isConnectionDead() method is invoked upon checking out the connection, and also in KeepAlive tasks. If the DB server returns errors we want to ignore (just like in the Exception Override Class) we cannot do this currently (testing on Hikari 5.0.1).

This would enable us to run Hikari with the custom AWS JDBC driver and RDS Aurora (MySQL) and cleanly support failover between nodes. Currently Aurora returns special SQLExceptions to initiate failover in the driver, and after the underlying connection is correctly reconnected, Hikari decides to close the connection because of the error. We see something like this in the logs:

13 [2023-03-27 16:58:35 751] [WARNING] HikariPool-1 - Failed to validate connection software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.ConnectionImpl@68878f6d (The active SQL connection has changed due to a connection failure. Please re-configure session state if required.). Possibly consider using a shorter maxLifetime value.
14 [2023-03-27 16:58:35 754] [FINE ] HikariPool-1 - Closing connection software.aws.rds.jdbc.mysql.shading.com.mysql.cj.jdbc.ConnectionImpl@68878f6d: (connection is dead)

This causes problems with Aurora because Hikari tries to reconnect using a cluster endpoint which, due to most likely caching, sometimes reconnects to the wrong node.

Simply adding something like this:

      catch (SQLException se) {
         String sqlState = se.getSQLState();
         if (sqlState.equalsIgnoreCase("08S02") ||
            sqlState.equalsIgnoreCase("08007")) {
            logger.warn("{} - Ignoring connection failure {} ({}).",
                     poolName, connection, se.getMessage());
            return false;
            }
         lastConnectionFailure.set(se);
         logger.warn("{} - Failed to validate connection {} ({}). Possibly consider using a shorter maxLifetime value.",
                     poolName, connection, se.getMessage());
         return true;
      }

in isConnectionDead() solves the issues that we see during failover. We are requesting that there be a configurable option for this or potentially just using the Exception Override class to make the decision.

This PR is similar to our request: #2045 and would solve this issue.

Link to the AWS driver project: https://github.com/awslabs/aws-mysql-jdbc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant