HikariCP try-with-resources Connection leaking

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP



HikariCP try-with-resources Connection leaking



I am working on something where I need to pull data from MariaDB (using HikariCP), and then send it through Redis. Eventually, when I try to pull from the database, the connection will start leaking. This only happens over time, and suddenly.



Here is the full log from when the leak started happening: https://hastebin.com/sekiximehe.makefile



Here is some debug info:


21:04:40 [INFO] 21:04:40.680 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Before cleanup stats (total=6, active=2, idle=4, waiting=0)

21:04:40 [INFO] 21:04:40.680 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - After cleanup stats (total=6, active=2, idle=4, waiting=0)

21:04:40 [INFO] 21:04:40.682 [HikariPool-1 connection adder] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Added connection org.mariadb.jdbc.MariaDbConnection@4b7a5e97

21:04:40 [INFO] 21:04:40.682 [HikariPool-1 connection adder] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - After adding stats (total=7, active=2, idle=5, waiting=0)

21:05:05 [INFO] 21:05:05.323 [HikariPool-1 housekeeper] WARN com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered for org.mariadb.jdbc.MariaDbConnection@52ede989 on thread Thread-272, stack trace follows
java.lang.Exception: Apparent connection leak detected
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:123)
at us.survivewith.bungee.database.FetchPlayerInfo.run(FetchPlayerInfo.java:29)
at java.lang.Thread.run(Thread.java:748)

21:05:10 [INFO] 21:05:10.681 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Before cleanup stats (total=7, active=2, idle=5, waiting=0)

21:05:10 [INFO] 21:05:10.681 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - After cleanup stats (total=7, active=2, idle=5, waiting=0)

21:05:39 [INFO] 21:05:39.352 [HikariPool-1 housekeeper] WARN com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered for org.mariadb.jdbc.MariaDbConnection@3cba7850 on thread Thread-274, stack trace follows
java.lang.Exception: Apparent connection leak detected
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:123)
at us.survivewith.bungee.database.FetchPlayerInfo.run(FetchPlayerInfo.java:29)
at java.lang.Thread.run(Thread.java:748)



Here is the FetchPlayerInfo.run() method:


@Override
public void run()

String select = "SELECT `Rank`,`Playtime` FROM `Players` WHERE PlayerUUID=?;";

// This is line 29. How can this possibly be causing a leak?
try(Connection connection = Database.getHikari().getConnection())

// Get the data by querying the Players table
try(PreparedStatement serverSQL = connection.prepareStatement(select))

serverSQL.setString(1, player);

// Execute statement
try(ResultSet serverRS = serverSQL.executeQuery())

// If a row exists
if(serverRS.next())

String rank = serverRS.getString("Rank");

Jedis jPublisher = Redis.getJedis().getResource();
jPublisher.publish("playerconnections", player + "~" + serverRS.getInt("Playtime") + "~" + rank);

else

Jedis jPublisher = Redis.getJedis().getResource();
jPublisher.publish("playerconnections", player + "~" + 0 + "~DEFAULT");




catch(SQLException e)

//Print out any exception while trying to prepare statement
e.printStackTrace();




This is how I've setup my Database class:


/**
* This class is used to connect to the database
*/
public class Database
{
private static HikariDataSource hikari;

/**
* Connects to the database
*/
public static void connectToDatabase(String address,
String db,
String user,
String password,
int port)

// Setup main Hikari instance
hikari = new HikariDataSource();
hikari.setMaximumPoolSize(20);
hikari.setLeakDetectionThreshold(60 * 1000);
hikari.setDataSourceClassName("org.mariadb.jdbc.MariaDbDataSource");
hikari.addDataSourceProperty("serverName", address);
hikari.addDataSourceProperty("port", port);
hikari.addDataSourceProperty("databaseName", db);
hikari.addDataSourceProperty("user", user);
hikari.addDataSourceProperty("password", password);


/**
* Returns an instance of Hikari.
* This instance is connected to the database that contains all data.
* The stats table is only used in this database every other day
*
* @return The main HikariDataSource
*/
public static HikariDataSource getHikari()

return hikari;



And this is how I am calling the FetchPlayerInfo class:


new Thread(new FetchPlayerInfo(player.getUniqueId().toString())).start();



EDIT:



The problem still persists after using a synchronized getConnection() method from the Database class.





I see what you mean... My only guess is that getConnection() is somehow not thread safe?
– moilejter
Aug 11 at 1:55





@moilejter Would that still be your guess even though this happens over time, and not every time that method is called?
– Trevor
Aug 11 at 2:03





Failing at seemingly random times is one of the clues that it might be a race condition somewhere - you have to hit the timing just right for it to fail... It is easy enough to test - write Database.getConnection() as a synchronized method that just calls getHikari().getConnection() - then use that, and see if the problem goes away...
– moilejter
Aug 11 at 3:18



Database.getConnection()


getHikari().getConnection()





How could it be a race condition though? The FetchPlayerInfo class is the only thing accessing Database.getHikari().getConnection(), and it's only ever being called once at a time. Am I just misunderstanding what a race condition is?
– Trevor
Aug 11 at 4:32





Ah! Well, I thought you were actually kicking off a bunch of threads to run this, not just one...
– moilejter
Aug 11 at 16:11




2 Answers
2



Jedis is also a resource of JedisPool you should close:


/// Jedis implements Closeable. Hence, the jedis instance will be auto-closed after the last statement.
try (Jedis jedis = pool.getResource()) {





Very good catch. Would that be causing the Connection leak though?
– Trevor
Aug 12 at 4:50



What version of HikariCP? It is possible that the leak is not actually a leak. The leak will be reported when the connection is out of the pool for longer than the threshold, he may actually be returned later. Newer versions of HikariCP will log “unleaked” Connections.



EDIT: I am as close to 100% certain as I can be that here is no race condition in HikariCP. This scenario is far to simple, and HikariCP is used by far too many users (millions) for such a fundamental flaw to not have surfaced before.



The only thing that makes sense, looking at the code above and the logs generated, is that one of the calls inside of the outer try-catch is hanging (blocking). I suggest getting a stack dump when the condition occurs, to find if there is a thread blocked inside of FetchPlayerInfo.run().


FetchPlayerInfo.run()





I'm using version 2.7.6. Isn't what I'm experiencing a leak, though?
– Trevor
Aug 11 at 14:31





I wondered about this, too - but the coe doesn't seem to have any kind of loop or wait within the try block where it gets the connection - so I expected it would run for just a little while, then be done - not the hours it would need for the threshold to kick in...
– moilejter
Aug 11 at 16:11





@moilejter I'm not sure where you get the "hours it would need for the threshold to kick in", the configuration in the question sets the threshold to 60 seconds.
– brettw
Aug 13 at 15:55





Ah! You're right - I was thinking of the default for the library, which I think was something like 8 hours (but I could be thinking of something else altogether :-()
– moilejter
Aug 13 at 17:04






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Popular posts from this blog

Firebase Auth - with Email and Password - Check user already registered

Dynamically update html content plain JS

How to determine optimal route across keyboard