How to warm up a quarkus app datasource to start under heavy http load

This is the situation. You have an API built in quarkus and deployed in Kubernetes. As a software engineer used to deal with situations where the application need to scale under this amount of traffic you set up and horizontal pod autoscaler object in Kubernetes just in case a big spike of request comes in and overwhelm the processing capacity of your pods.

All seems to work well, you see the spike of request starting in your monitoring dashboard and the new pods are added to the deployment as expected. But, when looking at the logs, something is not good. There are some hundreds (or even thousands) of messages like this:

Acquisition timeout while waiting for new connection

Something must be wrong, the pods were created in the right moment and started serving incoming http requests but the connections with the database couldn’t be established. First idea you have is to check what’s the number of connections your DB can handle, but when you look this shouldn’t be a problem, your app is only using approximately 25% of the available connections, so…

Let’s go through what is happening step by step. When the pod is created kubernetes goes through all defined health checks and make sure they are up before starting sending traffic. This includes our DB datasource. So if everything is prepared to serve request they can start to come in. But what happens when you receive hundreds of simultaneous request, you will also need a bunch of ready DB connections to fulfill them. Looking a bit more into our Quarkus Data source healthcheck source code DataSourceHealthCheck.java, it can be seen that on startup only 1 connection is created to check that DB can be reached. So our problem seems to be that we start with only ready DB connection to deal with hundreds of HTTP requests, and creating new DB connections is expensive in resources terms, as we usually hear.

There seems to be a property among the datasource configuration properties that can change this. quarkus.datasource.jdbc.initial-size defines the number of initial connections that need to be setup before data source is considered as ready. After this, we can select what is the usual amount of average connections our pods have in situations of heavy loads and set the property equal to this, to make sure all of them are available when http request start to come in. In my case it was:

quarkus.datasource.jdbc.initial-size = 12

And away goes my problem. In case you are worried by what could happen if every pod reserves initially this amount of transactions and could exhaust the DB total number of connections, of course it could happen. To avoid it you need to setup your scaling configuration thoughtfully on your HPA object and also use quarkus.datasource.jdbc.idle-removal-interval to define the time span that the pods will keep the idle connections and make sure they are released after a while of not being used.